Welcome to the Free Software contributions diary of Loïc Dachary. Although the posts look like blog entries, they really are technical reports about the work done during the day. They are meant to be used as a reference by co-developers and managers.

Fixing OpenVSwitch and GRE asymetric performances

OpenStack Havana is configured to use OpenVSwitch 1.10.2 as packaged for Ubuntu precise, with a linux-3.11 kernel.The cluster is connected to a 100Mb/s link. When sending data from an instance to the internet (using iperf), it shows ~90Mb/s. When receiving data from the internet to the instance, it is down to ~1Mb/s. After capturing the packets on the interface used by the default route on the hypervisor running the neutron router with

tcpdump -i eth0 host 91.224.149.132 -w /tmp/bad.cap

wireshark /tmp/bad.cap shows a lot of retransmissions.

A similar problem was reported back in October 2013 and hints that it may be a kernel problem. Upgrading the kernel of the hypervisor running the neutron router to linux-3.13 indeed fixes the problem. The compute nodes running the instances do not need their kernel updated, they can keep using the linux-3.11 kernel with the 1.10.2 version of the OpenVSwitch datapath kernel module. The OpenVSwitch kernel part is in the linux-3.13 tree and the openvswitch-datapath-dkms is not used any longer. It will fail to compile against the linux-3.13 headers but the error can be ignored and the package uninstalled.

Posted in Havana, openstack | 8 Comments

Two minor pitfalls when upgrading Havana stable

When upgrading an OpenStack compute or l3 agent node from 1:2013.2 to 1:2013.2.3 on Ubuntu precise 12.04.4:

  • The nova-compute version 1:2013.2 is expected to fail with
    /var/log/nova/nova-compute.log
    IncompatibleObjectVersion: Version 1.9 of Instance is not supported
    

    when interfaced with a 1:2013.2.3. It will not disrupt the running instances but will prevent operations on them until the upgrade is complete.

  • neutron-l3-agent will fail with:

    /var/log/neutron/metadata-agent.log
    AttributeError: 'HTTPClient' object has no attribute 'auth_tenant_id'
    

    because the python-neutronclient package must also be upgraded . It only happens if upgrading with apt-get install neutron-common but will be fine if upgrading with apt-get dist-upgrade.

Posted in Havana, openstack | Leave a comment

Reseting an instance {power,vm,task}_state in Havana

Sometime, after an hypervisor crash or nova-compute error, an OpenStack instance can be left in a state that cannot be conveniently fixed with nova reset-state.

$ nova list
+--------------------------------------+----------------+---------+...
| ID                                   | Name           | Status  |
+--------------------------------------+----------------+---------+...
| ca9496e9-0bd2-4734-9cf9-eb4e264628f7 | www            | SHUTOFF |
+--------------------------------------+----------------+---------+...
...  -------------+-------------+----------------------------------+
      Task State  | Power State | Networks                         |
...  -------------+-------------+----------------------------------+
      powering-on | Shutdown    | fsf-lan=10.0.3.18, 93.20.168.177 |
...  -------------+-------------+----------------------------------+

Setting the fields for the instance directly in the database will allow operations on the instance (nova start or nova volume-detach for instance):

$ mysql -e "update instances set task_state = NULL, \
           vm_state = 'stopped', \
           power_state = 4 \
           where deleted = 0 and hostname = 'www' and \
           uuid = 'ca9496e9-0bd2-4734-9cf9-eb4e264628f7'" nova

Using the uuid is necessary to avoid modifying an unrelated instance with the same name. This should be done only after verifying that the instance does not exist on the hypervisor with:

$ ps -fauwwwx | grep ca9496e9-0bd2-4734-9cf9-eb4e264628f7
Posted in Havana, openstack | Leave a comment

Sharing hard drives with Ceph

A group of users give hard drives to the system administrator of the Ceph cluster. In exchange, each of them get credentials to access a dedicated pool of a given size from the Ceph cluster.
Continue reading

Posted in ceph | 4 Comments

Working on a patch series with gerrit

A small patch series (1, 2, 3) is worked on in the wip-key-inject-unless topic branch.

When running git review –yes, each commit is dispatched to the review designated by the Change-Id: of the message.
Continue reading

Posted in gerrit, openstack | Leave a comment

HOWTO migrate an AMI from Essex to a bootable volume on Havana

A snapshot of an Essex OpenStack instance contains an AMI ext3 file system. It is rsync’ed to a partitioned volume in the Havana cluster. After installing grub from chroot, a new instance can be booted from the volume.
Continue reading

Posted in Essex, Havana, openstack | 1 Comment

Backporting with magit

Ceph uses GF-Complete stable branch (v1). Modifications required by Ceph were proposed to the master branch. While waiting for the corresponding pull request to be reviewed, a temporary branch (v1-ceph) was created in the Ceph name space for immediate use. The changes have been accepted in master, they are to be backported to v1
Continue reading

Posted in emacs, git, magit | Leave a comment

Merging with magit and ediff

The jerasure v2 branch is merged into the v2-ceph branch.


Continue reading

Posted in Uncategorized | Leave a comment

Celebrate Firefly and Icehouse

If you’re in Atlanta Sunday 11th, may 2014 evening, for the OpenStack summit or any other reason, join us to celebrate the OpenStack Icehouse release and the Ceph Firefly release. There will be both OpenStack and Ceph developers present and no presentations planned : it’s 100% informal :-)

Registration is here : Celebrate Firefly and Icehouse meetup page.

Posted in ceph, openstack | Leave a comment

SSE optimization for erasure code in Ceph

The jerasure library is the default erasure code plugin of Ceph. The gf-complete companion library supports SSE optimizations at compile time, when the compiler provides them (-msse4.2 etc.). The jerasure (and gf-complete with it) plugin is compiled multiple times with various levels of SSE features:

  • jerasure_sse4 uses SSE4.2, SSE4.1, SSSE3, SSE3, SSE2, SSE
  • jerasure_sse3 uses SSSE3, SSE3, SSE2, SSE
  • jerasure_generic uses no SSE instructions

When an OSD loads the jerasure plugin, the CPU features are probed and the appropriate plugin is selected depending on their availability.
The gf-complete source code is cleanly divided into functions that take advantage of specific SSE features. It should be easy to use the ifunc attribute to semi-manually select each function individually, at runtime and without performance penalty (because the choice is made the first time the function is called and recorded for later calls). With such a fine grain selection, there would be no need to compile three plugins because each function would be compiled with exactly the set of flag it needs.

Posted in ceph | 2 Comments