ganeti for the gcc compile farm and OpenStreetMap

A ganeti cluster was setup on a GCC Compile Farm machine. A virtual machine was created for the benefit of OpenStreetMap. It will replace the current http://osm.fsffrance.org/ virtual machine with better I/O.

Non intrusive ganeti installation

The machines on the GCC Compile Farm are setup to host user accounts on various architectures. It is sometime convenient to run a virtual machine instead of developing in a user account. For instance one can test a system wide installation without disrupting the other users. The idea was to install a ganeti cluster in the least intrusive way. The installed machine was modified for :

  • NAT the virtual machines:
          up echo 1 > /proc/sys/net/ipv4/ip_forward
          up iptables --table nat --append POSTROUTING --out-interface eth0 -j MASQUERADE
    
  • Use a local DNS.

The installation followed the step of a more ambitious installation and stripped the OSPF part which is not necessary as a first step.

LVM and DRBD

In squeeze all the software required to run ganeti2 are standard. A year ago it was significantly more difficult as it required the compilation of the DRBD from sources, among other things.
The second disk of the machine was used to create a single LVM partition

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1               1      121601   976760001   8e  Linux LVM

that was bound to the all volume group:

root@gcc20:~# vgs
  VG   #PV #LV #SN Attr   VSize   VFree
  all    1   2   0 wz--n- 931.51g 421.51g

The default DRBD configuration was removed so that ganeti can handle it on its own.

mv /etc/drbd.conf /etc/drbd.conf.old
touch /etc/drbd.conf
/etc/init.d/drbd reload

Network

A bridge and a fake lo:1 local interface were created in /etc/network/interfaces:

auto lo:1
iface lo:1 inet static
	address 10.10.0.20
	netmask 255.255.255.255
	up ip route add blackhole 10.0.0.0/8

auto br0
iface br0 inet static
        address 10.10.0.254
        netmask 255.255.255.255
        bridge_ports none
        bridge_stp off
	bridge_maxwait 5
        up ip route add 10.10.0.254 dev br0
        up ip route add 10.10.50.0/24 dev br0

BIND and DHCP

Ganeti hosts and virtual machines names are DNS entries. It would be possible to use a public
DNS to create them. A local DNS configuration is created localy instead, to
reduce the propagation delays, ensure it can always be reached and isolate the maintainance
of the name space into a private network. The top level domain farm was created
and divided into host.farm ( bound to 10.10.49.0/24 ) for all the ganeti hosts
and vm.farm ( bound to 10.10.50.0/24 ) for all the virtual machines.

/etc/bind/named.conf.options
Do not listen or answer on public IPs.

        allow-recursion { 127.0.0.1; 10.0.0.0/8; };
        listen-on { 127.0.0.1; 10.10.0.254; };
        
/etc/bind/named.conf.local
zone "10.10.in-addr.arpa" {
    type master;
    file "/etc/bind/db.10.10";
};

zone "farm" {
        type master;
        file "/etc/bind/db.farm";
};
        
/etc/bind/db.10.10
; -*- mode: zone; -*-
;
; BIND reverse data file for broadcast zone
;
$TTL    604800
@       IN      SOA     localhost. root.localhost. (
        2011031200      ; serial
                         604800         ; Refresh
                          86400         ; Retry
                        2419200         ; Expire
                         604800 )       ; Negative Cache TTL
@       IN      NS      localhost.

;;
;; perl -n -e 'print if(s/(\S+)\s+IN\s+A\s+10\.10\.(\d+)\.(\d+)/\3.\2\tIN\tPTR\t\1.farm./)' < db.farm
;;
1.50    IN      PTR     rodo1.osm.vm.farm.
        
/etc/bind/db.farm
; -*- mode: zone; -*-
;
$TTL 1h ; default Time-to-Live. defines the duration that the record may be cached (24h)

$ORIGIN farm.
@                       IN      SOA             ns hostmaster (
                        2011031200      ; serial
                                1h              ; refresh - time when the slave will try to refresh the zone from the master (8h)
                                30m             ; update retry - time between retries if the slave (secondary) (2h)
                                                ; fails to contact the master when refresh (above) has expired.
                                2w              ; expiry - time slave will continue to supply authoritative data for the zone (1w)
                                                ; after the master died
                                30m)            ; minimum - time a NAME ERROR = NXDOMAIN record is cached (24h)

                IN      NS      ns

ns              IN      A       10.10.0.254

rodo1.osm.vm    IN      A       10.10.50.1
	

A DHCP server is setup for each VM to use.

/etc/default/isc-dhcp-server
INTERFACES="br0"
        
/etc/dhcp/dhcpd.conf
A MAC address is generated with the following oneliner:

MACADDR="52:54:$(dd if=/dev/urandom count=1 2>/dev/null |
  md5sum |
  sed 's/^\(..\)\(..\)\(..\)\(..\).*$/\1:\2:\3:\4/')"; echo $MACADDR
        

and used to define the rodo1.osm.vm.farm entry:

group {
  option domain-name "farm";
  option routers 10.10.0.254;
  option domain-name-servers 10.10.0.254;
  subnet 10.10.0.0 netmask 255.255.0.0 {
    host rodo1.osm.vm.farm {
        hardware ethernet 52:54:5a:8d:77:8e;
        fixed-address rodo1.osm.vm.farm;
        option subnet-mask 255.255.255.255;
    }
  }
}
        

For both DHCP and the DNS, the configuration directories ( /etc/dhcp and /etc/bind respectively ) a git repository was created to keep an history of the changes.

Cluster and instance creation

The hostname of the machine is gcc20 which is bound to 127.0.1.1. This is not acceptable for ganeti2. When creating a cluster, the hostname is used to create the first node of the cluster and make it the master. For this operation to succeed, the hostname was temporarily changed to gcc20.host.farm and it's IP set in /etc/hosts to 10.10.49.20. This was permanently set on eth0 in the /etc/network/interface as follows:

	up ip addr add 10.10.49.20/32 broadcast 10.10.49.255 dev eth0

The corresponding entries were added to the bind configuration:

gnt-cluster.host        IN      A       10.10.49.1
gcc20.host              IN      A       10.10.49.20

The prerquisite to run gnt-cluster init were met:

  • gcc20.host.farm IP is not 127.*
  • gnt-cluster.host.farm IP is not bound to any interface
  • The cluster was created with:

    gnt-cluster init --no-etc-hosts \
                     --enabled-hypervisors=kvm \
                     --nic-parameters link=br0 \
                     --master-netdev br0 \
                     --vg-name all gnt-cluster.host.farm
    

    After downloading the installation CD of Debian GNU/Linux squeeze, the OpenStreetMap instance was created:

    gnt-instance add -d -t plain -s 10G -B memory=512M,vcpus=1 \
      -H kvm:boot_order=cdrom,cdrom_image_path=/srv/ganeti/debian-6.0.0-amd64-CD-1.iso,vnc_bind_address=0.0.0.0 \
      -n gcc20.host.farm -o debootstrap+default --net 0:mac=52:54:5a:8d:77:8e rodo1.osm.vm.farm
    

    The installation process was done using the VNC console

    vncviewer gcc20.fsffrance.org:14050
    

    Because the DHCP server delivers a 10.10.50.1/32 IP and a default route that is not on the same subnet ( 10.10.0.254 ) the DHCP client fails to setup the default route during the installation process. When the network initialization asks if the installation should proceed without a default route, the network was configured manually with the following:

    • IP 10.10.50.1
    • Netmask 10.10.0.0
    • DNS 10.10.0.254
    • Gateway 10.10.0.254

    After the installation completed, the disk was made the primary boot device:

    gnt-instance modify -H boot_order=disk rodo1.osm.vm.farm
    

    and the VM rebooted:

    gnt-instance restart --shutdown-timeout=1 rodo1.osm.vm.farm
    

    OpenStreeMap needs a lot of disk space and an additional 500GB disk was created for the virtual machine.

    gnt-instance modify --disk add:size=500G rodo1.osm.vm.farm
    

    From within the virtual machine it shows as /dev/vdb.

This entry was posted in FSF. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>