Welcome to the Free Software contributions diary of Loïc Dachary. Although the posts look like blog entries, they really are technical reports about the work done during the day. They are meant to be used as a reference by co-developers and managers. Erasure Code Patents StreamScale.

configuring ansible for teuthology

As of July 8th, 2015, teuthology (the Ceph integration test software) switched from using Chef to using Ansible. To keep it working, two files must be created. The /etc/ansible/hosts/group_vars/all.yml file with:

modify_fstab: false

The modify_fstab is necessary for OpenStack provisioned instances but it won’t hurt if it’s always there (the only drawback being that mount options are not persisted in /etc/fstab, but they are set as they should). The /etc/ansible/hosts/mylab file must then be populated with


where ovh224000.teuthology etc. are the fqdns of all machines that will be used as teuthology targets. The Ansible playbooks will expect to find all targets under the [testnodes] section. The output of a teuthology job should show that the Ansible playbook is being used with something like:

teuthology.run_tasks:Running task ansible.cephlab...
INFO:teuthology.task.ansible.out:PLAY [all] *****
TASK: [ansible-managed | Create the sudo group.] ******************************

Continue reading

Posted in ceph | Leave a comment

Public OpenStack providers usable within the hour

The OpenStack marketplace provides a list of OpenStack public clouds, a few of which enable the user to launch an instance at most one hour after registration.

Enter Cloud Suite has a 2GB RAM, 2 CPU, 40GB Disk instance for 0.06 euros / hour (~40 euros per month) and there is no plan to provide a flavor with only 1 CPU instead of 2 CPU. The nova, cinder and neutron API are available.

HP Helion Public Cloud has a 2GB RAM, 2 CPU, 10GB Disk instance for 0.05 euros / hour (0.06 USD / hour) (~40 euros per month).

OVH has a 2GB RAM, 1 CPU, 10GB Disk instance for 0.008 euros / hour (~3 euros per month). The nova API is available, not cinder nor neutron.

Rackspace has a 2GB RAM, 1 CPU, 10GB DIsk instance for ~40 euros per month (plus ~50 euros / month service fee, regardless of the number of instances). The nova and cinder API are available, not neutron.

DataCentred has 2GB RAM, 1CPU, 40GB Disk instance for ~40 euros per month. The nova, cinder and neutron API are available (but the router quota are set to zero by default). There are 2GB RAM, 1CPU, 40GB Disk AARCH64 instances for ~80 euros per month.

Cloudwatt has no 2GB RAM instance but a 3.75GB RAM, 1CPU, 50GB Disk instance for ~35 euros per month which makes it less expensive than all but OVH. The nova, cinder and neutron API are available.

Posted in openstack | 3 Comments

Setting a custom name server on an OpenStack instance

In an OpenStack tenant that is not allowed to create a network with neutron net-create, the name server can be set via cloudinit. The resolv-conf module although documented in the examples is not always available. It can be worked around with

 - echo nameserver | tee /etc/resolvconf/resolv.conf.d/head
 - resolvconf -u

for Ubuntu or

 - echo nameserver | tee /etc/resolv.conf
 - sed -ie 's/PEERDNS="yes"/PEERDNS="no"/' /etc/sysconfig/network-scripts/ifcfg-eth0

for CentOS.

Posted in openstack | 1 Comment

OpenStack instance name based on its IP address

A DNS has a set of pre-defined names such as:


If nova fixed-ip-reserve is denied by the OpenStack policy and neutron net-create is not available to create a network with the subnet that is exclusive to the OpenStack tenant, the naming of the instance must be done after openstack server create completes.
A cloudinit user-data file is created with:

 - url= \
  ( curl --silent $url/hostname | sed -e 's/\..*//' ; \
    printf "%03d" $(curl --silent $url/local-ipv4 | \
       sed -e 's/.*\.\(.*\)/\1/') \
  ) | \
  tee /etc/hostname
- hostname $(cat /etc/hostname)
preserve_hostname: true

Where $url/hostname retrieves the prefix of the hostname (multiple instances can have the same name, two simultaneous instance creation won’t race), $url/local-ipv4 gets the IPv4 address, keeps the last digits (sed -e ‘s/.*\.\(.*\)/\1/’)) and pad them with zeros if necessary (printf “%03d”). The hostname is stored in /etc/hostname and displayed in the /var/log/cloud-init.log logs (tee /etc/hostname) for debugging. This is done early in the cloudinit sequence (bootcmd) and the default cloudinit setting of the hostname is disabled (preserve_hostname: true) so that it does not override the custom name set with hostname $(cat /etc/hostname).
The instance is created with

$ openstack server create \
  --image 'ubuntu-trusty-14.04'
  --key-name loic \
  --flavor m1.small \
  --user-data user-data.txt \
  -f json \
  --wait \
... {"Field": "addresses", "Value": "fsf-lan="} ...
... {"Field": "id", "Value": "cd1a8a0f-83f9-4266-bd61-f3e2f583d59d"} ...

Whe user-data.txt contains the above cloudinit lines. The IPv4 address returned by openstack server create ( can then be used to rename the instance with

$ openstack server set --name the-re019 cd1a8a0f-83f9-4266-bd61-f3e2f583d59d

where cd1a8a0f-83f9-4266-bd61-f3e2f583d59d is the unique id of the instance which is preferred to the the-re prefix that could race with another identical openstack server create command.
To verify that the instance name matches the IPv4 address that is pre-set in the DNS:

$ ssh ubuntu@the-re019 hostname
Warning: Permanently added '' (ECDSA) to the list of known hosts.

Thanks to Josh Durgin for suggesting this solution.

Posted in openstack | 1 Comment

Delete the last port of an OpenStack router

When trying to delete an OpenStack subnet and the associated router, the command neutron router-delete complains because of the port allocated for the gateway and the port of the gateway cannot be removed with neutron port-delete because it is owned by the router. The solution is to clear the owner of the port with something like:

neutron port-update --device-owner clear 7f9685cb-794d-4847

and then delete the router. This is on Icehouse as provided by Enter Cloud Suite.

Posted in openstack | 2 Comments

OpenStack script to pre-allocate fixed IPs

The create-ports.py script allocates ports and indirectly gets fixed IPs from the DHCP server. The ports are named openstack000, openstack001 etc. and they are displayed in a format suitable for dnsmasq:

$ python create-ports.py --count 2 --net fsf-lan |  \
   sudo tee /etc/dnsmasq.d/openstack

If fsf-lan is a network shared with other tenants, it makes sure the IP are reserved, although they are not yet bound to an instance.

$ neutron port-list
| id                                   | name         | ...
| 1d1a05b1-383d-49ef-ae75-5ddcb5c714db | openstack001 |....

An new instance can then be given a known IP with:

$ openstack server create --image ubuntu-trusty-14.04 \
  --flavor 1cpu-1G \
  --key-name teuthology \
  --nic net-id=d936f445-5d68-485a-94f2-b852fd6b7d0c,v4-fixed-ip= \
  --wait openstack001

In the case of teuthology it is useful because the DNS can be configured once and for all while instances are dynamically created using IPs from the DNS instead of relying on allocation from the OpenStack DHCP server.
Continue reading

Posted in openstack | Leave a comment

create / delete an OpenStack instance with python-openstackclient

The python-openstackclient library has an example that provides the basic structure for a new command (the auth_url problem workaround may be needed). To create a virtual machine with 1GB RAM, 1CPU, ubuntu-14.04, using the teuthology keypair on the fsf-lan network, the matching flavor, image, keypair and network objects can be found with:

    for flavor in client_manager.compute.flavors.list():
        if flavor.ram == 1024 and flavor.vcpus == 1:
    for network in client_manager.compute.networks.list():
        if network.label == 'fsf-lan':
    for image in client_manager.compute.images.list():
        if 'ubuntu' in image.name and '14.04' in image.name:
    for keypair in client_manager.compute.keypairs.list():
        if keypair.name == 'teuthology':

The test instance can then be created

   server = client_manager.compute.servers.create('test',
                        image, flavor,
                        nics=[{'net-id': network.id}])

but it won’t be immediately active and the wait_for_status can be used to block until it is:

from openstackclient.common import utils

Deleting the instance is simpler:

    utils.wait_for_delete(client_manager.compute.servers.get, server.id)

See create-delete.py for a standalone script including the above lines that can be run as:

$ python create-server.py --help
usage: create-server.py [-h] [--os-compute-api-version ]
$ python create-server.py
FLAVOR: {'name': u'm1.small', ...
NETWORK: {'cidr_v6': None, 'dns2': None, 'dns1': None, 'netmask': None, 'label': u'fsf-lan',...
IMAGE: {'status': u'ACTIVE', 'updated': u'2014-05-19T11:43:00Z', 'name': u'ubuntu-trusty-14.04',...
KEYPAIR: {'public_key': u'ssh-rsa AAAAB3...

Continue reading

Posted in openstack | 2 Comments

Teuthology docker targets hack (5/5)

The teuthology container hack is improved to run teuthology-suite. For instance:

./virtualenv/bin/teuthology-suite \
  --distro ubuntu \
  --suite-dir $HOME/software/ceph/ceph-qa-suite \
  --config-file docker-integration/teuthology.yaml \
  --machine-type container \
  --owner loic@dachary.org \
  --filter 'rados:basic/{clusters/fixed-2.yaml fs/btrfs.yaml \
     msgr-failures/few.yaml tasks/rados_cls_all.yaml}' \
  --suite rados/basic --ceph ANY \

schedules a single job out of the rados suite and the results can be collected in the teuthology-worker archive directory:

$ tail -5 /tmp/a/loic-2015-06-06_16:06:57-rados:\
    tasks/rados_cls_all.yaml}', duration: 1017.5819008350372, \
  flavor: basic, owner: loic@dachary.org,
  success: true}
2015-06-06T16:24:38.634 WARNING:teuthology.report:No result_server \
  in config; not reporting results
2015-06-06T16:24:38.634 INFO:teuthology.run:pass

Continue reading

Posted in ceph, docker | Leave a comment

Ceph Jerasure and ISA plugins benchmarks

In Ceph, a pool can be configured to use erasure coding instead of replication to save space. When used with Intel processors, the default Jerasure plugin that computes erasure code can be replaced by the ISA plugin for better write performances. Here is how they compare on a Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz.

Encoding and decoding all used 4KB objects which is the default stripe width. Two variants of the jerasure plugins were used: Generic (jerasure_generic) and SIMD (erasure_sse4) which is used when running on an Intel processor with SIMD instructions.
This benchmark was run after compiling from sources using

$ ( cd src ; make ceph_erasure_code_benchmark )
$ TOTAL_SIZE=$((4 * 1024 * 1024 * 1024)) \
CEPH_ERASURE_CODE_BENCHMARK=src/ceph_erasure_code_benchmark \
  qa/workunits/erasure-code/bench.sh fplot | \
  tee qa/workunits/erasure-code/bench.js

and displayed with

firefox qa/workunits/erasure-code/bench.html
Posted in ceph, jerasure | 2 Comments

Improving Ceph python scripts tests

The Ceph command line and ceph-disk helper are python scripts for which there are integration tests (ceph-disk.sh and test.sh). It would be useful to add unit tests and pep8 checks.
It can be done by creating a python module instead of an isolated file (see for instance ceph-detect-init) with a tox.ini file including pep8, python2 and python3 test environments.
Since Ceph relies on autotools, the setup.py can be used with -local targets. For instance:

        python setup.py build
	python setup.py clean
	python setup.py install --root=$(DESTDIR) --install-layout=deb

Note the double : meaning it appends to an existing rule instead of overriding it. The –root=$(DESTDIR) will install the module files in the appropriate directory when building packages.
tox uses pip to fetch dependencies required to run tests from PyPI, but tests sometime run without network access. The depedencies can be collected by wheel with something like:

pip wheel -r requirements.txt

It will create a wheelhouse directory which can later be used with

pip install --no-index --use-wheel --find-links=wheelhouse \
  -r requirements.txt

Continue reading

Posted in ceph | 1 Comment