Collocated hardware

The OpenStack cluster used to migrate the legacy hardware is configured to allow the collocation of instances and volumes on the same hardware. One OpenStack availability zone groups hardware located in the same rack and uses the same switch as the legacy hardware. This allows for a migration that does not involve changing the IP of the machine. If the OpenStack nodes were located in a different autonomous system, a DNS change would be necessary and require additional preparations.

Maintenance LAN connection

The primary IP address used by the legacy hardware is also used by a number of services provided by the vservers it hosts. Moving this IP address to the OpenStack instance would mean losing access to the legacy hardware, without any hope to fallback, should something unexpected happen. Because both machines involved in the migration are connected to the same switch and use the same VLAN, an additional IP address is manually added to preserve communications:

ns1 : ip addr add 10.222.222.1/24 dev eth0
yopo : ip addr add 10.222.222.2/24 dev eth0

Preparations

In the following, desktop is any machine on which there are enough credentials to either connect to the legacy machine using ssh, run nova commands or EC2 commands targeting the OpenStack cluster, yopo is the OpenStack node, ns1 is the legacy hardware.

desktop: euca-create-volume --zone bm0008 --size 150
+----+-----------+--------------+------+-------------+-------------+
| ID |   Status  | Display Name | Size | Volume Type | Attached to |
+----+-----------+--------------+------+-------------+-------------+
| 3f | available | None         | 150  | None        |             |
+----+-----------+--------------+------+-------------+-------------+
desktop: nova volume-list | grep " $(printf "%d" 0x3f) "
| 63 | available      | None             | 150  | None        |                                      |

The bm0008 is the availability zone matching the OpenStack node known as yopo. Note that euca-create-volume which is an EC2 command reports the volume id as an exadecimal number but nova volume-list shows it as a decimal number. The hexadecimal form is used to name the LV volumes of the LVM backend. A partition table is then created on the 150GB volume and configured to have a single primary partition taking all the space.

yopo: kpartx -av /dev/vg/volume-0000003f
yopo: mkfs.ext3 /dev/mapper/vg-volume--0000003f1
yopo: mount /dev/mapper/vg-volume--0000003f1 /mnt
yopo: rsync -i --exclude=/etc/fstab --exclude=70-persistent-net.rules \
 --exclude=/boot/grub \
 --exclude=/srv/backup \
  --exclude=/var/cache \
  --exclude=/var/lib/backuppc \
  --exclude=/var/tmp \
  --exclude=/proc \
  --exclude=/sys -avHS --delete --numeric-ids 10.222.222.1:/ /mnt/

The partition is formatted with ext3 instead of ext4 to avoid any issues : the installed lenny from ns1 only uses ext3. A copy of the ns1 disk is made and excludes files that will either be replaced or contain data that are not worth replicating.

yopo: echo 'proc /proc proc defaults 0 0' > /mnt/etc/fstab
yopo: echo '/dev/vda1 / ext3 defaults,errors=remount-ro 0 1' >> /mnt/etc/fstab

The fstab is rewritten entirely to take into account the presence of a single partition ( as opposed to seven on ns1 ) and a device name starting with /dev/vd instead of /dev/hd or /dev/sd.

yopo: cp /mnt/boot/vmlinuz-2.6.26-2-vserver-686-bigmem /tmp
yopo: cp /mnt/boot/initrd.img-2.6.26-2-vserver-686-bigmem /tmp
yopo: umount
yopo: kpartx -dv /dev/vg/volume-0000003f
yopo: sed -i -e 's:kopt=.*:kopt=root=/dev/vda1' \
 -e 's/default=.*/default=0/' \
 -e 's/groot=.*/groot=(hd0,0)/' /boot/grub/menu.lst
yopo: echo '(hd0) /dev/vda' > /mnt/boot/grub/device.map
yopo: kvm -m 1024 -drive file=/dev/mapper/vg-volume--0000003f,if=virtio,index=0 \
  -boot c -initrd /tmp/initrd.img-2.6.26-2-vserver-686-bigmem\
   -kernel /tmp/vmlinuz-2.6.26-2-vserver-686-bigmem -append 'root=/dev/vda1' \
  -net nic -net user -nographic -curses -monitor unix:/tmp/file.mon,server,nowait
curses: grub-install /dev/vda
curses: update-grub
curses: halt

Grub is installed on the disk by using kvm to actually boot the instance, using a curses based console instead of a VGA console. The grub menu is edited to update the menu.lst and the device.map to reflect the changes with the disk and the partition table. The kernel and initrd are copied out of the file system imported from ns1 to be given as arguments to kvm to allow it to boot under conditions that are close to the one existing on the legacy hardware. Once the machine is successfully booted, grub-install and update-grub are called to allow kvm to boot without an external kernel. It can be verified with:

yopo: kvm -m 1024 -drive file=/dev/mapper/vg-volume--0000003f,if=virtio,index=0 \
  -boot c -net nic -net user -nographic \
  -curses -monitor unix:/tmp/file.mon,server,nowait

Routing the public IP

The legacy installation for ns1 does not obtain its IP address from DHCP and may contain a number of occurrence of this IP in various configuration files. The OpenStack node is configured to add a route dedicated to this IP by adding the following to /etc/rc.local.

brctl addbr br2004
ip link set br2004 up
ip r add 88.191.240.4/32 dev br2004

The br2004 bridge is dedicated to the tenant used to run the OpenStack instance, as shown by 2004 :

desktop: keystone tenant-list | grep ' april '
| 7c918c873280465da3785f5699d48316 | april           | True    |
desktop: nova-manage network list | grep 7c918c873280465da3785f5699d48316
5 10.145.4.0/24 None 10.145.4.3 None None 2004 7c918c873280465da3785f5699d48316 20941588-2c35-40b3-9ecb-af87cadae446

The bridge can be created before OpenStack runs so that the public IP can be routed to it. The existing router will be used by OpenStack.

Migrating

The rsync command shown is run to update copy, without stoping any service.

yopo: ssh 10.222.222.1 ip addr del 88.191.250.4/27 dev eth0
yopo: ssh 10.222.222.1 /etc/init.d/util-vserver stop

The rsync command is run again after stopping all vservers on ns1 and removing the IP from the interface.

yopo: umount /mnt
yopo: kpartx -dv /dev/vg/volume-0000003f
desktop: ssh controller.vm.april-int nova boot \
 --image 'CirrOS 0.3' \
 --block_device_mapping vda=63::0:0 \
 --flavor e.1-cpu.0GB-disk.8GB-ram \
 --key_name loic --availability_zone=bm0008 ns1 --poll

The partition is unmounted and the instance booted from the volume. It should recover as if a power failure happened.

yopo: ip r add 88.191.240.4/32 dev br2004

After the public IP is routed to the bridge br2004 to which the newly created instance is connected, the services should be up and communicated properly.

Setup the arp proxy

The interface of the OpenStack node that is used for floating IPs must be configured as an arp proxy.

echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp
echo 1 > /proc/sys/net/ipv4/conf/br2004/proxy_arp

These lines are appended to /etc/rc.local so that they are run at boot time. The switch to which both machines were connected has an arp cache. It needs to be cleared so that it notices that packets must be sent to another MAC.

 ip addr add 88.191.250.4/32 dev eth0
arping -U 88.191.240.4 -I eth0
ip addr del 88.191.250.4/32 dev eth0

dpkg -i ceph-deploy_0.0.1-1_all.deb

A ssh key is generated without a password and copied over to the root .ssh/authorized_keys file of each host on which ceph-deploy will act:

# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
ca:1f:c3:ce:8d:7e:27:54:71:3b:d7:31:32:14:ba:68 root@bm0014.the.re
The key's randomart image is:
+--[ RSA 2048]----+
|            .o.  |
|            oo.o |
|           . oo.+|
|          . o o o|
|        SE o   o |
|     . o. .      |
|      o +.       |
|       + =o .    |
|       .*..o     |
+-----------------+
# for i in 12 14 15
do
 ssh bm00$i.the.re cat \>\> .ssh/authorized_keys < .ssh/id_rsa.pub
done

Each host is installed with Ubuntu raring and has a spare, unused, disk at /dev/sdb. The ceph packages are installed with:

ceph-deploy  install bm0012.the.re bm0014.the.re bm0015.the.re

The short version of each FQDN is added to /etc/hosts on each host, because ceph-deploy will assume that it exists:

for host in bm0012.the.re bm0014.the.re bm0015.the.re
do
 getent hosts bm0012.the.re bm0014.the.re bm0015.the.re | \
   sed -e 's/\.the\.re//' | ssh $host cat \>\> /etc/hosts
done

The ceph cluster configuration is created with:

# ceph-deploy new bm0012.the.re bm0014.the.re bm0015.the.re

and the corresponding mon are deployed with

ceph-deploy mon create bm0012.the.re bm0014.the.re bm0015.the.re

Even after the command returns, it takes a few seconds for the keys to be generated on each host: the ceph-mon process shows when it is complete. Before creating the osd, the keys are obtained from a mon with:

ceph-deploy gatherkeys bm0012.the.re

The osds are then created with:

ceph-deploy osd create bm0012.the.re:/dev/sdb  bm0014.the.re:/dev/sdb  bm0015.the.re:/dev/sdb

After a few seconds the cluster stabilizes, as shown with

# ceph -s
   health HEALTH_OK
   monmap e1: 3 mons at {bm0012=188.165:6789/0,bm0014=188.165:6789/0,bm0015=188.165:6789/0}, election epoch 24, quorum 0,1,2 bm0012,bm0014,bm0015
   osdmap e14: 3 osds: 3 up, 3 in
    pgmap v106: 192 pgs: 192 active+clean; 0 bytes data, 118 MB used, 5583 GB / 5583 GB avail
   mdsmap e1: 0/0/1 up

A 10GB RBD is created, mounted and destroyed with:

# rbd create --size 10240 test1
# rbd map test1 --pool rbd
# mkfs.ext4 /dev/rbd/rbd/test1
# mount /dev/rbd/rbd/test1 /mnt
# df -h /mnt
Filesystem      Size  Used Avail Use% Mounted on
/dev/rbd1       9.8G   23M  9.2G   1% /mnt
# umount /mnt
# rbd unmap /dev/rbd/rbd/test1
# rbd rm test1
Removing image: 100% complete...done.

Ubuntu raring package

A series of patches fix minor build and deploy problems for the ceph-deploy package:

• the debian packages need python-setuptools as a build dependency
• Add python-pushy to the list of packages required to run ceph-deploy when installed on debian
• The list of path added by ceph-deploy does not cover all the deployment scenarios. In particular, when installed from a package it will end up in /usr/lib/python2.7/dist-packages/ceph_deploy . The error message is removed : the from will fail if it does not find the module.
• add missing python-setuptools runtime dependency to debian/control

Reseting the installation

To restart from scratch ( i.e. discarding all data and all installation parameters ), uninstall the software with

ceph-deploy uninstall bm0012.the.re bm0014.the.re bm0015.the.re

and purge any leftovers with

for host in bm0012.the.re bm0014.the.re bm0015.the.re
do
 ssh $host apt-get remove --purge ceph ceph-common ceph-mds
done

Remove the configuration files and data files with

for host in bm0012.the.re bm0014.the.re bm0015.the.re
do
 ssh $host rm -fr /etc/ceph /var/lib/ceph
done

Reset the disk with

for host in bm0012.the.re bm0014.the.re bm0015.the.re
do
 ssh $host <<EOF
umount /dev/sdb1
dd if=/dev/zero of=/dev/sdb bs=1024k count=100
sgdisk -g --clear /dev/sdb
EOF
done

restoring an instance from backup

The host is marked as unavailable

nova-manage service disable --host=bm0002.the.re --service=nova-compute
nova-manage service disable --host=bm0002.the.re --service=nova-network
nova-manage service disable --host=bm0002.the.re --service=nova-volume

and shows as such when listed

# nova-manage service list --host=bm0002.the.re
Binary           Host    Zone Status     State Updated_At
nova-compute     bm0002.the.re  bm0002  disabled   XXX   2013-05-11 09:18:25
nova-network     bm0002.the.re  bm0002  disabled   XXX   2013-05-11 09:18:30
nova-volume      bm0002.the.re  bm0002  disabled   XXX   2013-05-11 09:18:33

It can be removed completely later by modifying the mysql database directly. The april-ci instance was running on bm0002.the.re:

# nova list --name april-ci
+--------------------------------------+----------+---------+--------------------------------------+
|                  ID                  |   Name   |  Status |               Networks               |
+--------------------------------------+----------+---------+--------------------------------------+
| 4e8a8126-b27d-4c9e-abeb-4dc574c54254 | april-ci | SHUTOFF | novanetwork=10.145.9.5, 176.31.18.26 |
+--------------------------------------+----------+---------+--------------------------------------+

It is artificially moved to a host that is enabled:

mysql -e "update instances set host = 'bm0001.the.re', availability_zone = 'bm0001' where hostname = 'april-ci'" nova

and deleted

nova delete april-ci

Assuming the content of failed host was backed up entirely ( i.e. rsync / ), the april-ci disk is located using the id shown above as the output of nova list

# grep 4dc574c54254 /var/lib/nova/instances/*/*.xml
/var/lib/nova/instances/instance-000001de/libvirt.xml:    4e8a8126-b27d-4c9e-abeb-4dc574c54254

and the corresponding disk is turned into a minimal file system

chroot /backup/bm0002.the.re
mount -t proc none /proc
qemu-nbd --port 20000 /var/lib/nova/instances/instance-000001de/disk &
nbd-client localhost 20000 /dev/nbd0
pv /dev/nbd0 > april-ci.april-ci.img
fsck -fy $(pwd)/april-ci.april-ci.img
resize2fs -M april-ci.april-ci.img
exit

and uploaded to glance, using the same kernel and initrd, as shown with nova image-show original-image-of-april-ci

glance add name="april-ci-2013-05-11" disk_format=ami container_format=ami \
 kernel_id=2e714ea3-45e5-4bb8-ab5d-92bfff64ad28 \
 ramdisk_id=6458acca-24ef-4568-bb2b-e52322a5a11c < /backup/bm0002.the.re/april-ci.april-ci.img

it is then rebooted using the same flavor

nova boot --image 'april-ci-2013-05-11' \
  --flavor e.1-cpu.10GB-disk.1GB-ram \
  --key_name loic --availability_zone=bm0001 --poll april-ci

recovering from a partially damaged logical volume

A 30GB volume contains bad blocks toward the end ( after 26GB ) but it was not full. A fsck is run on a copy of the disk to check how much the recovery process would lose. It turns out to be less than a hundred files in a non-critical area. A new disk of the same size is allocated on another machine with

# euca-create-volume --zone bm0001 --size 30
VOLUME  vol-0000005b    30      bm0001  creating        2013-05-11T11:22:19.889Z

and the content of the damaged volume are copied over, until it fails with an I/O error.

ssh -A root@bm0001.the.re
ssh bm0002.the.re pv /dev/nova-volumes/volume-00000143 | \
 pv > /dev/nova-volumes/volume-0000005b

and it is repaired

fsck -fy /dev/nova-volumes/volume-0000005b

The volume residing on the failed host is removed directly from the database

mysql -e "update volumes set deleted = 1 where id = 30" nova

recovering from a partially damanged instance disk

An instance disk has a few failed blocks and may be recovered if the others are copied over. Because rsync is more resilient to I/O errors than dd or pv, it is used to recover as much as possible with:

# ssh -A root@bm0002.the.re
# rsync --inplace --progress /var/lib/nova/instances/instance-00000089/disk root@bm0001.the.re:/backup/bm0002.the.re/var/lib/nova/instances/instance-00000089/disk
  1843396608 100%    8.41MB/s    0:03:28 (xfer#1, to-check=0/1)
rsync: read errors mapping "/mnt/var/lib/nova/instances/instance-00000089/disk": Input/output error (5)
WARNING: disk failed verification -- update retained (will try again).
disk
  1843396608 100%   37.37MB/s    0:00:47 (xfer#2, to-check=0/1)
rsync: read errors mapping "/var/lib/nova/instances/instance-00000089/disk": Input/output error (5)
ERROR: disk failed verification -- update retained.
sent 1843836447 bytes  received 858892 bytes  7000741.32 bytes/sec
total size is 1843396608  speedup is 1.00
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1070) [sender=3.0.9]

It is then turned into a file using nbd as shown above and checked for errors:

# fsck -fy $(pwd)/openstack.jenkins.img
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
/openstack.jenkins.img: recovering journal
Clearing orphaned inode 117551 (uid=0, gid=0, mode=0100644, size=0)
Clearing orphaned inode 9764 (uid=0, gid=0, mode=0100644, size=1393052)
Clearing orphaned inode 9765 (uid=0, gid=0, mode=0100644, size=302040)
Clearing orphaned inode 7050 (uid=105, gid=109, mode=0100644, size=0)
Clearing orphaned inode 8841 (uid=0, gid=0, mode=0100644, size=81800)
Clearing orphaned inode 10235 (uid=0, gid=0, mode=0100644, size=253328)
Clearing orphaned inode 10240 (uid=0, gid=0, mode=0100644, size=180624)
Clearing orphaned inode 8840 (uid=0, gid=0, mode=0100644, size=874608)
Clearing orphaned inode 6469 (uid=0, gid=0, mode=0100755, size=1245180)
Clearing orphaned inode 10739 (uid=0, gid=0, mode=0100644, size=18192)
Clearing orphaned inode 10927 (uid=0, gid=0, mode=0100644, size=19908)
Clearing orphaned inode 10754 (uid=0, gid=0, mode=0100644, size=100820)
Clearing orphaned inode 10738 (uid=0, gid=0, mode=0100644, size=11468)
Clearing orphaned inode 10926 (uid=0, gid=0, mode=0100644, size=31568)
Clearing orphaned inode 10956 (uid=0, gid=0, mode=0100644, size=18780)
Clearing orphaned inode 10958 (uid=0, gid=0, mode=0100644, size=22312)
Clearing orphaned inode 10723 (uid=0, gid=0, mode=0100644, size=13976)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Free blocks count wrong (2299561, counted=2283092).
Fix? yes
Free inodes count wrong (538192, counted=534536).
Fix? yes
/openstack.jenkins.img: ***** FILE SYSTEM WAS MODIFIED *****
/openstack.jenkins.img: 52984/587520 files (0.3% non-contiguous), 338348/2621440 blocks

If the lossage is better than recovering from yesterday's backup, the instance is rebooting using this copy.

fqdn2fqdn = {
    'foo.com': 'foo.me',
    'bar.com': 'bar.me',
}

It will map foo.com to foo.me as follows:

$ sudo python dns_spoof.py 8.8.8.8 &
$ ping -c 1 foo.me
PING foo.me (91.185.200.115) 56(84) bytes of data.
64 bytes from 91.185.200.115: icmp_req=1 ttl=47 time=42.2 ms
--- foo.me ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 42.268/42.268/42.268/0.000 ms
$ ping -c 1 foo.com
PING foo.com (91.185.200.115) 56(84) bytes of data.
64 bytes from 91.185.200.115: icmp_req=1 ttl=47 time=42.2 ms
--- foo.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 42.290/42.290/42.290/0.000 ms

Update May 10, 2013: an easier solution is to configure your BIND resolvers to lie using Response Policy Zones (RPZ). Thanks to S. Bortzmeyer for pointing in the right direction.

OpenStack based test environment

In an integration environment where tests scripts boot instances using

nova boot foo

the associated domain name will be foo.novalocal by default but the associated private IP is not known in advance. When testing the integration of a puppet manifests designed to deploy this machine to serve foo.com, the manifest will typically contain:

node 'foo.com', 'foo.novalocal' {
...
}

so that it deploys on the production machine ( foo.com ) and the test machine ( foo.novalocal ). If the integration test checks foo.com availability, the corresponding nagios command will try to access foo.com and not foo.novalocal.
Although it would be possible to avoid using FQDN such as foo.com, there are a number of circumstances where the software will make it inconvenient or the operator will just forget about this rule.

spoofing DNS requests

To transparently ensure that all FQDN used in puppet manifests resolve to private IPs matching a *.novalocal name, the manifests are parsed to create a table mapping each node name to the .novalocal FQDN found in the same node stanza. This dictionary can then be used by a daemon to spoof each DNS request to foo.com into a request to foo.me.

spoofing DNS server

The DNS server and client library provided with twisted is used to implement a spoofing DNS reverse proxy.
It reads a python file mapping FQDN to their spoofed equivalent.
When an incoming DNS request is received, the FQDN is substituted if it is found in the spoof map.
The spoofed name is saved in the spoofed variable and included in the closure added to the deferred created to forward the query to the authoritative DNS.
When the answer is received, the spoofed name is restored so that the original requester does not notice the difference. The name is also restored in case of an error otherwise the client is not notified of the error because it contains a reference to a FQDN that is not know to the original client.

testing the DNS server

The standalone python script is isolated to allow for inclusion by the test script. It is run as follows:

$ trial test.py
test
  DnsSpoofTestCase
    testAddressRecord_one ...                                    [OK]
    testAddressRecord_two ...                                    [OK]
    testSpoofedRecord_fail ...                                   [OK]
    testSpoofedRecord_one ...                                    [OK]
    testSpoofedRecord_two ...                                    [OK]
---------------------------------------------------------------------
Ran 5 tests in 0.017s
PASSED (successes=5)

Before each test is run, a DNS server is run to provide information for a pre-defined set of FQDN. Another DNS server is run based on the DNSSpoofFactory class, which is configured to forward to the first DNS.
The first DNS is tested to check that it resolves one.my-domain.com and two.my-domain.com. The spoofing DNS is tested to check that is resolves one.my-domain.com to the same IP as two.my-domain.com because the spoof map asks for it:

fqdn2fqdn = { 'two.my-domain.com': 'one.my-domain.com' }

The error handler is checked to raise a DNSNameError if trying to resolve a hostname that is unknown to the authoritative DNS.

other DNS servers

bind9 could be used to create zones in which some hostnames (foo.com for instance) are CNAME to the corresponding private hostname (foo.novalocal). The puppet module creating these CNAME could even be merged into the puppet module bind9 view. But the bind9 configuration to achieve is either difficult or does not exist.
The unbound, dnsmasq and ettercap DNS servers do not support CNAME : they all require that the IP of a hostname is known in advance, which is not the case when launching an instance in OpenStack.

nova suspend target1

The instance is suspended to disk (as if it was a laptop) and the corresponding KVM process is killed. While the instance is suspended, nova-network can be stopped.

/etc/init.d/nova-network stop

The source of the problem was a typo in the public interface leading to an incorrect VLAN interface

13: vlan100@eth2:  mtu 1500 qdisc noqueue state DOWN mode DEFAULT
    link/ether fa:16:3e:54:5b:57 brd ff:ff:ff:ff:ff:ff

it can be fixed in the /etc/nova/nova.conf configuration file at the line:

public_interface = eth3

The incorrect VLAN interface is manually deleted and nova-network can be restarted. The instance is then resumed with

nova resume target1

and nova-network will automatically re-create the VLAN interface.

fixing nova-network configuration and restarting the service

When the public IP of the bare metal host is not configured properly in /etc/nova/nova.conf

my_ip = 192.168.20.10

nova-network will create an incorrect SNAT iptables rule

0 0 SNAT all  --  any tun0 10.20.0.0/16 anywhere to:192.168.20.10

When the my_ip line is fixed, /etc/init.d/nova-network can be safely be restarted, even when instances are running on the bare metal. It will not disrupt their connections and the iptables rule will be updated as expected.

modifying the interfaces and restarting the service

Some problems cannot be fixed by simply modifying the /etc/nova/nova.conf file and the VLAN interface must be deleted manually. When the public interface is wrongly configured:

public_interface = eth2

and an instance has been created on the bare metal, a VLAN interface and a bridge are created:

# ip link vlan100
13: vlan100@eth2:  mtu 1500 qdisc noqueue state DOWN mode DEFAULT
    link/ether fa:16:3e:54:5b:57 brd ff:ff:ff:ff:ff:ff
# brctl show br100
bridge name     bridge id               STP enabled     interfaces
br100           8000.fa163e6e08de       no              vlan100
                                                        vnet0
                                                        vnet1
                                                        vnet2
                                                        vnet3

If the configuration file is fixed to use eth3 instead of eth2

public_interface = eth3

restarting nova-network will not change the interface to which vlan100 is attached. Assuming the instances bound to the bridge are as follows:

 nova list
+--------------------------------------+------------+--------+---------------------+
| ID                                   | Name       | Status | Networks            |
+--------------------------------------+------------+--------+---------------------+
| 5e263310-a578-4653-bb48-697cca589297 | target1    | ACTIVE | private_0=10.20.0.6 |
| b108007d-d7e4-4289-83a8-a72280541eb2 | target2    | ACTIVE | private_0=10.20.0.7 |
| 8b97f260-f888-49a2-8f80-065ea49ea3b6 | target3    | ACTIVE | private_0=10.20.0.8 |
| 585ad852-2d34-45a5-8036-607fa0087511 | teuthology | ACTIVE | private_0=10.20.0.5 |
+--------------------------------------+------------+--------+---------------------+

they can be temporarily suspended with:

# nova suspend target1
# nova suspend target2
# nova suspend target3
# nova suspend teuthology
# nova list
+--------------------------------------+------------+-----------+---------------------+
| ID                                   | Name       | Status    | Networks            |
+--------------------------------------+------------+-----------+---------------------+
| 5e263310-a578-4653-bb48-697cca589297 | target1    | SUSPENDED | private_0=10.20.0.6 |
| b108007d-d7e4-4289-83a8-a72280541eb2 | target2    | SUSPENDED | private_0=10.20.0.7 |
| 8b97f260-f888-49a2-8f80-065ea49ea3b6 | target3    | SUSPENDED | private_0=10.20.0.8 |
| 585ad852-2d34-45a5-8036-607fa0087511 | teuthology | SUSPENDED | private_0=10.20.0.5 |
+--------------------------------------+------------+-----------+---------------------+

It will kill the KVM process running the instance and save its state to disk, the equivalent of a laptop suspend to disk. The bridge shows they are no longer attachd to it:

# brctl show br100
bridge name     bridge id               STP enabled     interfaces
br100           8000.fa163e6e08de       no              vlan100

The bridge and the VLAN interface are manually deleted

# ip link set br100 down
# brctl delbr br100
# ip link delete vlan100

When nova-network is stopped, the dnsmasq process persists and it will not notice when a new bridge is created. It must be killed so that nova-network restarts it after the re-creating the bridge.

pkill dnsmasq

If dnsmasq is not killed, the instance will resume properly but will loose its IP after trying to renew the DHCP lease. The instances can be resumed after starting nova-network

# /etc/init.d/nova-network start
# nova resume target1
# nova resume target2
# nova resume target3
# nova resume teuthology

The VLAN interface is created as a side effect of starting the first instance:

# ip link vlan100
13: vlan100@eth3:  mtu 1500 qdisc noqueue state DOWN mode DEFAULT
    link/ether fa:16:3e:54:5b:57 brd ff:ff:ff:ff:ff:ff

and the instances will not notice the difference.

                                             +---------+
                                             | +-----+ |
    list              ptr                    | |     | |
 +----------+       +-----+                  | |     | |
 | append_  >------->     >-------------------->     | |
 |  buffer  |       +-----+                  | |     | |
 +----------+                        ptr     | |     | |
 |   _len   |      list            +-----+   | |     | |
 +----------+    +------+     ,--->+     >----->     | |
 | _buffers >---->      >-----     +-----+   | +-----+ |
 +----------+    +----^-+     \      ptr     |   raw   |
 |  last_p  |        /         `-->+-----+   | +-----+ |
 +--------+-+       /              +     >----->     | |
          |       ,-          ,--->+-----+   | |     | |
          |      /        ,---               | |     | |
          |     /     ,---                   | |     | |
        +-v--+-^--+--^+-------+              | |     | |
        | bl | ls | p | p_off >--------------->|     | |
        +----+----+-----+-----+              | +-----+ |
        |               | off >------------->|   raw   |
        +---------------+-----+              |         |
              iterator                       +---------+

The actual data is stored in buffer::raw opaque objects. They are accessed through a buffer::ptr. A buffer::list is a sequential list of buffer::ptr which can be used as if it was a contiguous data area although it can be spread over many buffer::raw containers, as represented by the rectangle enclosing the two buffer::raw objects in the above drawing. The buffer::list::iterator can be used to walk each character of the buffer::list as follows:

  bufferlist bl;
  bl.append("ABC", 3);
  {
    bufferlist::iterator i(&bl);
    ++i;
    EXPECT_EQ('B', *i);
  }

documentation and unit tests

The ultimate documentation for buffer.cc and buffer.h are the unit tests that demonstrate how it actually works. This document is a short guide designed to provide an overview and does not attempt to cover everything.

buffer::ptr and buffer::raw

The buffer::raw is where the data is actually stored. It is allocated with malloc, new or reusing a pointer provided by the caller. A variant of the malloc constructor provides an area that is aligned on CEPH_PAGE_SIZE. The address of the allocated memory will be a multiple of CEPH_PAGE_SIZE, which must be a power of two and a multiple of sizeof(void *).

 +-----------+                +-----+
 |           |                |     |
 |  offset   +----------------+     |
 |           |                |     |
 |  length   +----            |     |
 |           |    \-------    |     |
 +-----------+            \---+     |
 |   ptr     |                +-----+
 +-----------+                | raw |
                              +-----+

The buffer::raw area can only be accessed through the buffer::ptr. It adresses the buffer::raw bytes in the range [offset,offset+length[. The buffer::ptr methods are very flexible and mostly designed to be used to implement buffer::lists. The constructors can allocate a buffer::raw area with ptr(unsigned l) or be assigned an existing buffer::raw with ptr(raw *r), as in buffer::list::rebuild().
Bytes can be copied in or copied out within the [offset,offset+length[ range. If the underlying buffer::raw extends beyond offset+length (as reported by unused_tail_length()), bytes can be appended.

    bufferptr ptr(2);
    ptr.set_length(0);
    ptr.append('A');
    EXPECT_EQ((unsigned)1, ptr.length());
    EXPECT_EQ('A', ptr[0]);
    ptr.append(“B”, (unsigned)1);
    EXPECT_EQ((unsigned)2, ptr.length());
    EXPECT_EQ(‘B’, ptr[1]);

buffer::list and buffer::list::iterator

A buffer::list is a list of buffer::ptr, as shown below.

                                             +---------+
                                             | +-----+ |
    list                                     | |     | |
 +----------+                                | |     | |
 | append_  |                                | |     | |
 |  buffer  |                                | |     | |
 +----------+                        ptr     | |     | |
 |   _len   |      list            +-----+   | |     | |
 +----------+    +------+     ,--->+     >----->     | |
 | _buffers >---->      >-----     +-----+   | +-----+ |
 +----------+    +----^-+     \      ptr     |   raw   |
 |  last_p  |                  `-->+-----+   | +-----+ |
 +----------+                      |     >----->     | |
                                   +-----+   | |     | |
                                             | |     | |
                                             | |     | |
                                             | |     | |
                                             | |     | |
                                             | +-----+ |
                                             |   raw   |
                                             |         |
                                             +---------+

The operator[] abstracts it.

  bufferlist bl;
  bl.append('A');
  bufferlist other;
  other.append('B');
  bl.append(other);
  EXPECT_EQ((unsigned)2, bl.buffers().size());
  EXPECT_EQ('B', bl[1]);

The buffer::list::iterator class provides some of the usual iterator behavior and is used in the buffer::list::contents_equal(ceph::buffer::list& other) method.

                                             +---------+
                                             | +-----+ |
    list                                     | |     | |
 +----------+                                | |     | |
 | append_  |                                | |     | |
 |  buffer  |                                | |     | |
 +----------+                        ptr     | |     | |
 |   _len   |      list            +-----+   | |     | |
 +----------+    +------+     ,--->+     >----->     | |
 | _buffers >---->      >-----     +-----+   | +-----+ |
 +----------+    +----^-+     \      ptr     |   raw   |
 |  last_p  |        /         `-->+-----+   | +-----+ |
 +--------+-+       /              +     >----->     | |
          |       ,-          ,--->+-----+   | |     | |
          |      /        ,---               | |     | |
          |     /     ,---                   | |     | |
        +-v--+-^--+--^+-------+              | |     | |
        | bl | ls | p | p_off >--------------->|     | |
        +----+----+-----+-----+              | +-----+ |
        |               | off >------------->|   raw   |
        +---------------+-----+              |         |
              iterator                       +---------+

The bl data member points to the buffer::list. The ls data member is used to avoid dereferencing a pointer and is equivalent to bl->_buffers. The p data member is a std::list<ptr>::iterator used to walk the list of buffer::ptr. The p_off data member is the offset at which the iterator currently is, within the buffer::raw pointed by p. The off data member is the offset of the iterator, as if there was only one buffer::raw.
Although buffer::list::iterator exposes copy in and out methods, they are designed to be used as supporting methods for the corresponding copy in and out methods from the buffer::list class.

{
    bufferlist bl;
    bufferlist dest;
    const char *expected = "ABC";
    bl.append(expected);
    bl.copy(1, 2, dest);
    EXPECT_EQ(0, ::memcmp(expected + 1, dest.c_str(), 2));
}
{
    bufferlist bl;
    bl.append("XXX");
    bl.copy_in(1, 2, "AB");
    EXPECT_EQ(0, ::memcmp("XAB", bl.c_str(), 3));
}

The internal representation of the buffer::list can be rebuilt to use a single buffer::raw.

  {
    bufferlist bl;
    const std::string str(CEPH_PAGE_SIZE, 'X');
    bl.append(str.c_str(), str.size());
    bl.append(str.c_str(), str.size());
    EXPECT_EQ((unsigned)2, bl.buffers().size());
    bl.rebuild();
    EXPECT_EQ((unsigned)1, bl.buffers().size());
  }

It can also be rebuilt to only use buffer::raw that are aligned on CEPH_PAGE_SIZE.

  {
    bufferlist bl;
    {
      bufferptr ptr(CEPH_PAGE_SIZE + 1);
      ptr.set_offset(1);
      ptr.set_length(CEPH_PAGE_SIZE);
      bl.append(ptr);
    }
    EXPECT_EQ((unsigned)1, bl.buffers().size());
    EXPECT_FALSE(bl.is_page_aligned());
    bl.rebuild_page_aligned();
    EXPECT_TRUE(bl.is_page_aligned());
    EXPECT_EQ((unsigned)1, bl.buffers().size());
  }

The content of the buffer::list can be read from a file or written to a file, either with a file descriptor or a path.

{
  bufferlist bl;
  bl.append("ABC");
  EXPECT_EQ(0, bl.write_file("testfile"));
}
{
  std::string error;
  bufferlist bl;
  ::system("echo ABC > testfile");
  EXPECT_EQ(0, bl.read_file("testfile", &error));
  EXPECT_EQ((unsigned)4, bl.length());
  std::string actual(bl.c_str(), bl.length());
  EXPECT_EQ("ABC\n", actual);
}

• user.key : first 2048 characters
• user.key@1 : next 2048 characters
• user.key@2 : last 1024 characters

The proposed unit tests may be used as a documentation describing in detail how it is implemented from the caller point of view.

coverage

The unit tests cover most (> 93%) of the chained extended attributes implementation. The following functions are tested:

• int chain_getxattr(const char *fn, const char *name, void *val, size_t size);
• int chain_fgetxattr(int fd, const char *name, void *val, size_t size);
• int chain_setxattr(const char *fn, const char *name, const void *val, size_t size);
• int chain_fsetxattr(int fd, const char *name, const void *val, size_t size);
• int chain_listxattr(const char *fn, char *names, size_t len);
• int chain_flistxattr(int fd, char *names, size_t len);
• int chain_removexattr(const char *fn, const char *name);
• int chain_fremovexattr(int fd, const char *name);

untested lines

The function translate_raw_name substitutes @@ into @. When the trailing
character is a @, it breaks. However, such an occurrence cannot be created by chain_setxattr because it always create pairs of @. Instead of silently breaking the loop, the function should probably return on error so that the caller can ignore it.

The function chain_fgetxattr_len may return on error if fgetxattr returns on error. However, it is only called after another attr function returned success and the tests cannot create the conditions under which it would fail.

The function chain_fsetxattr contains untested code being discussed on the mailing list

dependency to the underlying filesystem

If the file system in which the tests are run does not support extended attributes, the tests are not run. The detection uses the same logic as the one implemented in FileStore::_detect_fs. The test should be run as part of teuthology to check how it behaves when used with the supported file systems.

minimizing the noise

The output of the tests are silenced to reduce the output when testing assertions ( except for the dout_emergency function which cannot be controlled).