Authors: Jean-François Amiot, Byron Chu, Neil Stewart
The concept for the K-12 Firewall as a Service pilot is simple: leverage Cybera's existing cloud resource, the Rapid Access Cloud, to provide a virtualized security solution for schools. These institutions are looking for an alternative to the individual security appliances employed by the majority of K-12 schools in Alberta, as well as the shared hardware firewall solution employed by a small handful of school districts. The solution, dubbed Firewall-as-a-Service, is looking to improve network traffic patterns (e.g. reduce network hairpins) and provide feature-parity with the existing solutions, while reducing the administrative and resource overhead put on schools. The infrastructure and the service itself are deployed and maintained using typical cloud application patterns and DevOps practices.
Our plan was to create a platform for the rapid deployment of fully configured and functioning virtual appliances, with individual deployment-specific values (i.e. networks, policies, access control lists, local users, etc.) injected when the virtual appliance is launched. These values would be stored in a secure version control system. Some form of automated deployment tool would be used to launch each instance of the appliance, as needed. All of these elements would then be tied together with a simple interface to control the lifecycle of each instance.
This plan was developed as a proof-of-concept, using only 'off the shelf' and open tools that we already had at hand. However, once the project was underway, we quickly realized that we were missing a very large component to offer what are essentially virtualized network functions (NFV): we were missing the ability to create virtual networks.
From Groom Lake to Silicon Valley
The Rapid Access Cloud recently added a new pre-production environment called Area51, which includes the technology needed to create the virtual networks to deploy Firewall-as-a-Service: Neutron, OpenStack’s networking project. This, backed with OpenVSwitch, provided the virtual networking to create the Software Defined Networking-like solution we had planned. Using Area51, we spun up an Ubuntu instance with port forwarding enabled between interfaces on two different test networks. We were then able to test the network performance of traffic running through the Rapid Access Cloud and the specific instance.
Our first concern was the base-line throughput performance of the cloud (as we had not tuned the cloud environment, or designed a high-performance virtual appliance image). But the results were encouraging, and we started looking at how performance could be improved, along with exactly what we were going to present to our users.
The next element put in place was the virtual appliance, which would provide the same features existing in the various solutions used by the K-12 school districts. After investigating several products on the market, we settled on Palo Alto Network's VM-Series of firewall. This is a virtual form-factor of their existing hardware products, and utilizes PANOS, the PaloAlto Network Operating System. This product was already familiar to many of our users. PANOS also has a well documented API that permits us to automate deployment of the virtual machines leveraging our existing DevOps tools and practices.
We quickly spun up a few instances with different vCPU configurations (per PaloAlto's specifications), with 2, 4, and 8 vCPUs aligning with required session counts, performance needs, and desired services and features enabled on the firewall. We immediately noticed a significant discrepancy with respect to performance between our initial Ubuntu proof-of-concept and the VM-Series firewalls. Although this was to be expected, the results were disappointing, as it meant a great deal of tuning would be needed before we could accept any production-level traffic.
Back to the drawing board
After consulting with PaloAlto, reading articles on network performance in OpenStack Neutron and OpenVSwitch, and running tests on the network, it was decided to simplify the Neutron configuration in Area51 and remove OpenVSwitch, using only linuxbridge. This improved performance slightly, but not enough to have a functioning service. We went even further and enabled single root I/O virtualization (SRIOV) in Neutron so we could move the burden of virtualizing the network interfaces off the hypervisor and onto the network interface card, which is designed for just this task.
It was this configuration change that gave us exactly what we were looking for. When we first began testing performance, we were topping out at around 400 Mbps of throughput across various tests, but after all the subsequent changes, we saw speeds of over 900 Mbps. Success!
To infinity and beyond
We are continuing to tweak, tune and test our Firewall as a Service technology, and have begun piloting it with a couple of different school districts. We will continue to refine this service and actively pilot it with interested K-12 school districts over the summer and early fall until the pilot ends on October 31, 2017. We are also actively building the other components to the service.
If all goes well, we hope to turn this into a production-level service that can benefit all Alberta K-12 school districts. We strongly believe this project will change the manner in which school districts deploy their network security solutions, which will lead to increased efficiencies for educators in the province!
For more details, please see our webpage and/or contact Cybera's Network Director Jean-François Amiot.