An IPv6 enabled cloud

Infrastructure as a Service (IaaS) provides users with easy access to virtual machines (VM) in a cloud environment. Since the user has complete control over their VM, it makes sense for them to have access to the entire range of TCP and UDP ports. The easiest way to enable this kind of access is to have each virtual machine accessible by a unique IP address.

Large cloud providers such as Amazon EC2 and Rackspace are able to provide this feature because they have access to a very large pool of public IP addresses. Smaller cloud providers, however, don't. This is unfortunate because limited access to public IP addresses becomes a bottleneck that prevents smaller providers from growing.

IPv4

IPv4 was standardized in the late 70's / early 80's by defining the address space with 32-bits. That give us around 4.3 billion IPv4 addresses to work with, which at the time, with just a few nodes interconnected, appeared like a huge number. But today, with more than 7.1 billion people worldwide, that number starts to look small. A few tricks have been used to overcome this, like proxy servers, load balancers, and especially NAT. However, even with these techniques, we are running out of IPv4 addresses.

As per Dec 5 2013, ARIN, the North American Registry for Internet Numbers, only had 1.56 /8 equivalents to assign. When you realize that some companies in the early 80s got a /8 themselves, you know we are not far from depleting the pool. The Asia and Europe registrars have already exhausted their pools, and ARIN is projected to get there in one year. What should we do next?

networking4 smallIPv6

Let's get it right this time. IPv6 has a 128-bit address space, which gives us an exceptionally large number of addresses. For comparison, if all IPv4 addresses would fit the physical dimensions of an iPod, then the IPv6 space is the size of the Earth! Cybera has been IPv6 enabled since 2005, and had full IPv6 deployment on all servers and desktops in 2011. But then we started to move our infrastructure to the cloud…

Creating an IPv6 Enabled Cloud

Since all of Cybera's servers are IPv6 enabled, we required IPv6 support in order for us to take advantage of IaaS ourselves. Below, we will explain how we leveraged our experience with OpenStack and IPv6 and got the two to work together.

Network Configuration

IPv4 has been configured the standard way, where the VM has a private IP and through OpenStack Horizon you can assign a public floating IP to a particular VM. Because we are using the multi-host network mode, each Compute Node does NAT for IPv4 and bridge packets on IPv6, as each VM already has a public globally routable IPv6 address assigned.

This is the configuration for VLAN 30 on our L3 switch:

interface Vlan30
 no ip address
 no ip redirects
 no ip proxy-arp
 ipv6 address 2605:FD00:0:10::1/64
 ipv6 nd other-config-flag
 ipv6 nd router-preference High
 no ipv6 redirects
 ipv6 dhcp server InfraCloudIPv6

Let's go over the IPv6 settings:

 ipv6 address 2605:FD00:0:10::1/64

This is the gateway IPv6 address for the segment. /64 is the default network mask for IPv6. This will automatically make the router send Router Advertisements (RA) packets on VLAN 30. When a server is set to use IPv6 with Stateless Address Auto-Configuration (SLAAC), as soon as it  receives these packets, it will automatically know the subnet range and mask plus the default gateway. We now have IPv6 connectivity, but we are still missing DNS settings for a truly automatic network configuration.

 ipv6 nd other-config-flag

In order for the server to get configurations other than addresses, we need to set the "Other Config" flag. That tells the server that other configurations (in this case, DNS) should be assigned using DHCPv6.

 ipv6 nd router-preference High

There are three router preferences: Low, Medium and High. As we are the authoritative router for this network, we want to make sure we are set to High.

 no ipv6 redirects

IP redirects are bad.

 ipv6 dhcp server InfraCloudIPv6

This assigns DHCPv6 on this interface. The DHCPv6 configuration itself is straightforward:

ipv6 dhcp pool InfraCloudIPv6
 dns-server 2605:FD00::2
 dns-server 2605:FD00::3
 domain-name cybera.ca

Server Configuration

We used Ubuntu 12.04 to create this OpenStack environment. We also used Open vSwitch as the software bridging service. The standard Linux bridge-utils would have worked just fine, but we find Open vSwitch to be a better Linux bridging solution overall.

Each server was configured with a number of interfaces defined in

/etc/network/interfaces:

eth0 is the management NIC. It was connected to a simple management switch.

The rest of the ethX interfaces are grouped into a bonded NIC, bond0. We used LACP 802.3ad for the bonding mechanism.

On top of bond0, we configured three VLANs:

  • vlan20 provides external IPv4 connectivity. Each server had a static public IPv4 address configured on this VLAN.
  • vlan30 is the subnet that all virtual machines communicate on. There is no IP address configured on this VLAN.
  • flatdhcp_br0 is not a VLAN, but is the bridge that all compute nodes use to bridge the virtual machines to vlan30. This interface has a static IPv4 address and it uses MAC-based IPv6 autodiscovery to configure an IPv6 address on itself.
  • Finally, vlan31 is used for OpenStack-to-OpenStack communication. There's a static private IPv4 address assigned on this interface.

Gotchas

Here are some '€œgotchas'€œ that caused us a few hours of troubleshooting:

nova-network will try to add the IPv6 subnet to the bridged interface (more on this later). Once it does this, you will most likely see a dadfailed error on the interface:

$ sudo ip a | grep dadfailed
inet6 2605:fd00:0:10::/64 scope global tentative dadfailed

This is ignorable.

Secondly, if you find that auto-configuration is no longer working on the interface, make sure forwarding is turned off:

$ sudo cat /proc/sys/net/ipv6/conf/flatdhcp_br0/fowarding
0

The ndisc6 package includes a handy utility called rdisc6 that can be used to re-initiate IPv6 autodiscovery:

$ sudo rdisc6 flatdhcp_br0

OpenStack Configuration

For this setup, OpenStack Grizzly was used. Havana was not available at the time, but will be tested soon. We also chose to use nova-network rather than Neutron. This is because we're more familiar with nova-network than Neutron, so during troubleshooting, we could easily distinguish what was an OpenStack issue and what was an IPv6 issue.

Regarding the specifics about nova-network, we used FlatDHCP Manager and a multi-host configuration.

Each compute node has the nova-compute and nova-network services running. The compute node's /etc/nova/nova.conf file has the following network-related options:

network_manager=nova.network.manager.FlatDHCPManager
dhcpbridge_flagfile=/etc/nova/nova.conf
dhcpbridge=/usr/bin/nova-dhcpbridge
force_dhcp_release=True
libvirt_use_virtio_for_bridges=True
public_interface=vlan20
flat_interface=vlan30
flat_network_bridge=flatdhcp_br0
use_ipv6=True

All of the above is pretty standard. The only option that might look unfamiliar is the use_ipv6 option. To see how this option is used within Nova, run the following commands on any server with Nova installed:

$ cd /usr/lib/python2.7/dist-packages/nova
$ grep -R use_ipv6 *

One key area to review is in the network/manager.py file on lines 1636 and 1858. The former is for FlatDHCPManager and the latter is for VlanManager, but they're both doing the same thing, which is setting up a Routing Advertisement daemon. You can see the details of the RA configuration in network/linux_net.py on line 1063.

In order for nova-network to successfully configure the RA, the command radvd must exist. Usually this means having the radvd package installed on the server, but let's take a step back.

The use of a Routing Advertisement daemon is to advertise the availability of an IPv6 subnet. We're already doing this on our upstream router, so we have no need for this. Additionally, since we're running in multi-host network mode, each compute node would therefore have the RA daemon running. For that to even be close to correct, each compute node would have to advertise a separate IPv6 subnet. That's really not practical at all.

To get around this, we can do two things: hack the nova source code and remove the RA parts or create a dummy radvd file and make nova-network think the radvd command ran successfully.

Modifying the OpenStack source code is not recommended unless you really have to. Therefore, we opted for the latter option. We created a file called /usr/sbin/radvd with the following contents:

#!/bin/bash
exit 0

With all of this in place, the nova-manage command can now be used to create a network:

$ sudo nova-manage network create --num_networks 1 --network_size 1024 --fixed_range_v4 10.254.0.0/22 --bridge flatdhcp_br0 --fixed_range_v6 2605:fd00:0:10::/64 --gateway 2605:fd00:0:10::1 --multi_host T nova

Once this command has been issues, make sure you review the entry made in the MySQL nova.networks table:

$ mysql nova
mysql> select * from networks;

Also review the cidr_v6 and gateway_v6 columns '€” you might have to manually update these values.

Now go ahead and launch a virtual machine through OpenStack. Both the dashboard and command line will display the Fixed IPv4 and IPv6 address of the instance:

$ nova show test

+—————————–+———————————————————-+

| Property                    | Value                                                    |

+—————————–+———————————————————-+

| status                      | ACTIVE                                                   |

| updated                     | 2013-12-04T15:47:07Z                                     |

| OS-EXT-STS:task_state       | None                                                     |

| key_name                    | default                                                  |

| image                       | CentOS 6 (b1691574-ec72-485d-a307-6436e25d8a44)          |

| hostId                      | e440696e2d65235c8b5913858b25a167efeb09e385efcf48a503496e |

| OS-EXT-STS:vm_state         | active                                                   |

| nova network                | 10.255.0.102, 2605:fd00:0:10:f816:3eff:fe1e:d0ec         |

| flavor                      | m1.tiny (1)                                              |

| id                          | ee6a1eec-ebf0-4c34-97ac-2cb206cc3f40                     |

| security_groups             | [{u’name’: u’default’}]                                  |

| user_id                     | c84fbc3af66e4ac28c85dc5d961f7388                         |

| name                        | test                                                     |

| created                     | 2013-12-04T15:46:24Z                                     |

| tenant_id                   | a3a7501861b646f89772da798f00591c                         |

| OS-DCF:diskConfig           | MANUAL                                                   |

| metadata                    | {}                                                       |

| accessIPv4                  |                                                          |

| accessIPv6                  |                                                          |

| progress                    | 0                                                        |

| OS-EXT-STS:power_state      | 1                                                        |

| OS-EXT-AZ:availability_zone | nova                                                     |

| config_drive                |                                                          |

+—————————–+———————————————————-+

Conclusion

In our opinion, IPv6 and IaaS are great complements to each other: they enable large amounts of disposable virtual machines to be publicly accessible on the internet without the worries of an IP address conflict or stale DHCP lease. OpenStack's free and open nature is a catalyst for this combination.