Fragmented floating IP pools and multiple AS hack

When an OpenStack Havana cluster is deployed on hardware rented from OVH and Hetzner, IPv4 are rented by the month and are either isolated ( just one IP, not a proper subnet ) or made of a collection of disjoint subnets of various sizes.

91.121.254.238/32
188.165.144.248/30
...

OpenStack does not provide a way to deal with this situation and a hack involving a double nat using a subnet of floating IP is proposed.
A L3 agent runs on an OVH machine and pretends that 10.88.15.0/24 is a subnet of floating IPs, although they are not publicly available. Another L3 agent is setup on a Hetzner machine and uses the 10.88.16.0/24 subnet.
When an instance is created, it may chose a Hetzner private subnet, which is connected to a Hetzner router for which the gateway has been set to a network providing the Hetzner floating IPs. And the same is done for OVH.
A few floating IP are rented from OVH and Hetzner. On the host running the L3 agent dedicated to the OVH AS, a 1 to 1 nat is established between each IP in the 10.88.15.0/24 subnet and the OVH floating IPs. For instance the following /etc/init/nat.conf upstart script associates 10.88.15.3 with the 91.121.254.238 floating IP.

description "OVH nat hack"

start on neutron-l3-agent

script
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  ip addr add 10.88.15.1/24 dev br-ex
  while read private public ; do
    test "$public" || continue
    iptables -t nat -A POSTROUTING -s $private/32 -j SNAT --to-source $public
    iptables -t nat -A PREROUTING -d $public/32 -j DNAT --to-destination $private
  done <<EOF
10.88.15.3      91.121.254.238
EOF
end script

Fragmented floating IP pools and routing

Each floating IP ( also called failover IP ) provided by either OVH or Hetzner is uniquely associated to the MAC of the ethernet interface of a given hardware, using the proprietary web interface provided by OVH and Hetzner. The packets destined to the floating IP are routed to the interface even if the interface does not answer to arp who-has. The subnet to which a given floating IP belong is unknown and it is not possible to figure out if there is a gateway in the same subnet as a floating IP. If an instance is associated with such a floating IP, the outgoing packets are expected to be routed via the same gateway as the host. For instance on an OVH host:

root@bm0015:~# ip addr show dev eth0
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 70:54:d2:1a:9d:76 brd ff:ff:ff:ff:ff:ff
    inet 188.165.223.81/24 brd 188.165.223.255 scope global eth0
root@bm0015:~# ip route
default via 188.165.223.254 dev eth0  metric 100

The IP address of the bm0015.the.re hardware ( it is not a floating IP ) is 188.165.223.81, is in a /24 subnet and 188.165.223.254 is its router. The 91.121.254.238 floating IP is routed to bm0015.the.re and although it is in a completely different subnet, it is expected to use the same gateway, that is 188.165.223.254.

AS segregation

An external network is defined for OVH:

neutron net-create ovh --router:external=True

The L3 agent for OVH is configured to only handle this network so that multiple agents can be run.

# neutron net-list --name ovh
+--------------------------------------+------+----------------------------------------------------+
| id                                   | name | subnets                                            |
+--------------------------------------+------+----------------------------------------------------+
| 033b4851-af21-478e-9a57-e624ff0b1340 | ovh  | 9cb918b6-8737-4416-a5e0-e4bc5a5e6718 10.88.15.0/24 |
+--------------------------------------+------+----------------------------------------------------+
# grep 033b4851 /etc/neutron/l3_agent.ini
gateway_external_network_id = 033b4851-af21-478e-9a57-e624ff0b1340

It is also set to be the default for internal routers

# grep internal /etc/neutron/l3_agent.ini
handle_internal_only_routers = True

The subnet that will act as if it was a public subnet is created within the ovh net:

# neutron subnet-create --name ovh --disable-dhcp \
              --allocation-pool=start=10.88.15.2,end=10.88.15.254 \
              ovh 10.88.15.0/24
Created a new subnet:
+------------------+------------------------------------------------+
| Field            | Value                                          |
+------------------+------------------------------------------------+
| allocation_pools | {"start": "10.88.15.2", "end": "10.88.15.254"} |
| cidr             | 10.88.15.0/24                                  |
| dns_nameservers  |                                                |
| enable_dhcp      | True                                           |
| gateway_ip       | 10.88.15.1                                     |
| host_routes      |                                                |
| id               | 9cb918b6-8737-4416-a5e0-e4bc5a5e6718           |
| ip_version       | 4                                              |
| name             | ovh                                            |
| network_id       | 033b4851-af21-478e-9a57-e624ff0b1340           |
| tenant_id        | 2a2365c4031d47d890bb403db7e92583               |
+------------------+------------------------------------------------+

The –disable-dhcp prevents running a dnsmasq process that is not going to be used anyway. The allocation pool is not served by dnsmasq for floating IPs.
A router is created and the ovh network is set to be its gateway, implicitly meaning the subnet is going to be used when allocating floating IPs.

# neutron router-create ovh
# neutron router-gateway-set ovh ovh
Set gateway for router ovh

A private subnet is created and connected to the router. All instances that intend to get a floating IP from the ovh pool must be connected to this subnet, otherwise there will be no route between them and the floating IP.

# neutron net-create ovh-lan
# neutron subnet-create --name ovh-lan ovh-lan 10.0.15.0/24
# neutron router-interface-add ovh ovh-lan

Double NAT hack

The first nat is established by OpenStack between 10.88.15.0/24 and 10.0.15.0/24. Another nat is maintained outside of OpenStack to map the IPs from the subnet 10.88.15.0/24 to actual public IPs. The map is maintained manually from an upstart script that runs after the neutron L3 agent.
The instances that have no associated public IPs are masqueraded behind 10.88.15.1 ( the gateway_ip picked by default when the ovh subnet was created above ). The line

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

will masquerade them once more, using whatever public IP is the default for eth0. The full script is added in /etc/init/nat.conf and can be run manually with start nat.

description "OVH nat hack"

start on neutron-l3-agent

script
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  ip addr add 10.88.15.1/24 dev br-ex
  while read private public ; do
    test "$public" || continue
    iptables -t nat -A POSTROUTING -s $private/32 -j SNAT --to-source $public
    iptables -t nat -A PREROUTING -d $public/32 -j DNAT --to-destination $private
  done <<EOF
10.88.15.3      91.121.254.238
EOF
end script

Alternatives

After discussing with Edouard Thuleau, Chux Uzoeto, Mathieu Rohon, Pierre-Andre Morey, Christophe Sauthier, Erwan Gallen, Carl Perry, Sylvain Afchain, Emilien Macchi I’ve not been able to find a simpler or less ad-hoc way. It is not possible to attach eth0 to br-ex because OVH will raise alerts if it sees unexpected MAC addresses. It is possible to route a floating IP to br-ex. However it is not possible to subnet-create a single IP ( there needs to be at least one other IP used as a gateway and there is no way to specify a gateway that is no in the same subnet as the IP ). It is also not possible to update the allocation pools using neutron subnet-update because it is a read only attribute. Although it is possible to hack routes and IP directly in the net namespace of the router, the end result is more contorted than relying on a double nat.

This entry was posted in Havana, openstack. Bookmark the permalink.

4 Responses to Fragmented floating IP pools and multiple AS hack

  1. Ryan Lane says:

    An alternative, that I originally tried pushing into nova-network, is to use BGP to announce your floating IPs. In that situation you don’t need subnets at all. My nova-network change was rejected due to quantum (now neutron) existing, but neutron at the time didn’t even have floating IP support, so I just gave up…

    The awesome thing about BGP announced floating IPs is that with a little work on the quantum side, it would be possible to move IPs not just between instances but also between networks, regions, datacenters, etc..

  2. Chux Uzoeto says:

    Loic,
    That is an interesting and amazing hack to work around the issues .. I am still trying to understand it fully in a logical manner. So, the limitation seems not on physical ethernet devices as I worried about, but more on networking address blocks and how they get used ? .. Please, can you elaborate on this a wee bit.

    It seems the best way forward with these multisite setups would be to look towards SDN based solutions like Open Daylight .. SDN is rapidly gaining traction, but as far as I am aware OpenDaylight is not ready for production use right now.

    Again due to the huge allocation blocks of ipv6 (/64) available with these service providers, ipv6 may represent an immediate solution for splitting networking the way neutron expects them to be setup .. We need to look more seriously at ipv6 as a way out.

  3. Loic Dachary says:

    The limitation is indeed about using multiple network address blocks as part of a single floatting IP allocation pool. It is not a common use case.
    I don’t know anything about OpenDaillight :-) As far as I understand, neutron provides a SDN and as soon as it is able to conveniently handle the use case described here the hack won’t be necessary.
    Using IPv6 is definitely a good idea. The problem with OVH and Hetzner is that they only provide a single IPv6 ( i.e. a /64 such as 2001:41d0:2:aa70::/64 ). You would need to run an IPv6 proxy on the machine to further divide it. At the moment only tetaneutral provides a /56 which could be used with OpenStack.

  4. You inspired me another approach, it doesn’t solve that you may need to wire several floating IP ranges, … but may be this can be patched in neutron at some time … http://www.ajo.es/post/74363654373/using-openstack-neutron-sdn-in-ovh-or-hetzner-commodity

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>