Welcome to the Free Software contributions diary of Loïc Dachary. Although the posts look like blog entries, they really are technical reports about the work done during the day. They are meant to be used as a reference by co-developers and managers. Erasure Code Patents StreamScale.

restoring an OpenStack ssh public key

When a ssh private key is obtained from OpenStack via

openstack keypair create foobar > foobar.pem

the matching public key is stored in the OpenStack tenant. If it is later deleted with

openstack keypair delete foobar

it can be restored with

ssh-keygen -y  -f foobar.pem > foobar.pub
openstack keypair create --public-key foobar.pub foobar
Posted in openstack | 2 Comments

oneliner to deploy teuthology on OpenStack

Note: this is obsoleted by Ceph integration tests made simple with OpenStack

The teuthology can be installed as a dedicated OpenStack instance on OVH using the OpenStack backend with:

nova boot \
   --image 'Ubuntu 14.04' \
   --flavor 'vps-ssd-1' \
   --key-name loic \
   --user-data <(curl --silent \
     https://raw.githubusercontent.com/dachary/teuthology/wip-6502-openstack/openstack-user-data.txt | \
     sed -e "s|OPENRC|$(env | grep OS_ | tr '\n' ' ')|") teuthology

Assuming the IP assigned to the instance is, the following will display the progress of the integration tests that are run immediately after the instance is created:

ssh ubuntu@ tail -n 2000000 -f /tmp/init.out

If all goes well, it will complete with:

========================= 8 passed in 1845.59 seconds =============
___________________________________ summary _________________________
  openstack-integration: commands succeeded
  congratulations :)

And the pulpito dashboard will display the remains of the integration tests at like so:

Posted in ceph, openstack | Leave a comment

Running your own Ceph integration tests with OpenStack

Note: this is obsoleted by Ceph integration tests made simple with OpenStack

The Ceph lab has hundreds of machines continuously running integration and upgrade tests. For instance, when a pull request modifies the Ceph core, it goes through a run of the rados suite before being merged into master. The Ceph lab has between 100 to 3000 jobs in its queue at all times and it is convenient to be able to run integration tests on an independent infrastructure to:

  • run a failed job and verify a patch fixes it
  • run a full suite prior to submitting a complex modification
  • verify the upgrade path from a given Ceph version to another
  • etc.

If an OpenStack account is not available (a tenant in the OpenStack parlance), it is possible to rent one (it takes a few minutes). For instance, OVH provides an horizon dashboard showing how many instances are being used to run integration tests:

The OpenStack usage is billed monthly and the accumulated costs are displayed on the customer dashboard:

Continue reading

Posted in ceph, openstack | 1 Comment

configuring ansible for teuthology

As of July 8th, 2015, teuthology (the Ceph integration test software) switched from using Chef to using Ansible. To keep it working, two files must be created. The /etc/ansible/hosts/group_vars/all.yml file with:

modify_fstab: false

The modify_fstab is necessary for OpenStack provisioned instances but it won’t hurt if it’s always there (the only drawback being that mount options are not persisted in /etc/fstab, but they are set as they should). The /etc/ansible/hosts/mylab file must then be populated with


where ovh224000.teuthology etc. are the fqdns of all machines that will be used as teuthology targets. The Ansible playbooks will expect to find all targets under the [testnodes] section. The output of a teuthology job should show that the Ansible playbook is being used with something like:

teuthology.run_tasks:Running task ansible.cephlab...
INFO:teuthology.task.ansible.out:PLAY [all] *****
TASK: [ansible-managed | Create the sudo group.] ******************************

Continue reading

Posted in ceph | Leave a comment

Public OpenStack providers useable within the hour

The OpenStack marketplace provides a list of OpenStack public clouds, a few of which enable the user to launch an instance at most one hour after registration.

Enter Cloud Suite has a 2GB RAM, 2 CPU, 40GB Disk instance for 0.06 euros / hour (~40 euros per month) and there is no plan to provide a flavor with only 1 CPU instead of 2 CPU. The nova, cinder and neutron API are available.

HP Helion Public Cloud has a 2GB RAM, 2 CPU, 10GB Disk instance for 0.05 euros / hour (0.06 USD / hour) (~40 euros per month).

OVH has a 2GB RAM, 1 CPU, 10GB Disk instance for 0.008 euros / hour (~3 euros per month). The nova API is available, not cinder nor neutron.

Rackspace has a 2GB RAM, 1 CPU, 10GB DIsk instance for ~40 euros per month (plus ~50 euros / month service fee, regardless of the number of instances). The nova and cinder API are available, not neutron.

DataCentred has 2GB RAM, 1CPU, 40GB Disk instance for ~40 euros per month. The nova, cinder and neutron API are available (but the router quota are set to zero by default). There are 2GB RAM, 1CPU, 40GB Disk AARCH64 instances for ~80 euros per month.

Cloudwatt has no 2GB RAM instance but a 3.75GB RAM, 1CPU, 50GB Disk instance for ~35 euros per month which makes it less expensive than all but OVH. The nova, cinder and neutron API are available.

Posted in openstack | 3 Comments

Setting a custom name server on an OpenStack instance

In an OpenStack tenant that is not allowed to create a network with neutron net-create, the name server can be set via cloudinit. The resolv-conf module although documented in the examples is not always available. It can be worked around with

 - echo nameserver | tee /etc/resolvconf/resolv.conf.d/head
 - resolvconf -u

for Ubuntu or

 - echo nameserver | tee /etc/resolv.conf
 - sed -ie 's/PEERDNS="yes"/PEERDNS="no"/' /etc/sysconfig/network-scripts/ifcfg-eth0

for CentOS.

Posted in openstack | 1 Comment

OpenStack instance name based on its IP address

A DNS has a set of pre-defined names such as:


If nova fixed-ip-reserve is denied by the OpenStack policy and neutron net-create is not available to create a network with the subnet that is exclusive to the OpenStack tenant, the naming of the instance must be done after openstack server create completes.
A cloudinit user-data file is created with:

 - url= \
  ( curl --silent $url/hostname | sed -e 's/\..*//' ; \
    printf "%03d" $(curl --silent $url/local-ipv4 | \
       sed -e 's/.*\.\(.*\)/\1/') \
  ) | \
  tee /etc/hostname
- hostname $(cat /etc/hostname)
preserve_hostname: true

Where $url/hostname retrieves the prefix of the hostname (multiple instances can have the same name, two simultaneous instance creation won’t race), $url/local-ipv4 gets the IPv4 address, keeps the last digits (sed -e ‘s/.*\.\(.*\)/\1/’)) and pad them with zeros if necessary (printf “%03d”). The hostname is stored in /etc/hostname and displayed in the /var/log/cloud-init.log logs (tee /etc/hostname) for debugging. This is done early in the cloudinit sequence (bootcmd) and the default cloudinit setting of the hostname is disabled (preserve_hostname: true) so that it does not override the custom name set with hostname $(cat /etc/hostname).
The instance is created with

$ openstack server create \
  --image 'ubuntu-trusty-14.04'
  --key-name loic \
  --flavor m1.small \
  --user-data user-data.txt \
  -f json \
  --wait \
... {"Field": "addresses", "Value": "fsf-lan="} ...
... {"Field": "id", "Value": "cd1a8a0f-83f9-4266-bd61-f3e2f583d59d"} ...

Whe user-data.txt contains the above cloudinit lines. The IPv4 address returned by openstack server create ( can then be used to rename the instance with

$ openstack server set --name the-re019 cd1a8a0f-83f9-4266-bd61-f3e2f583d59d

where cd1a8a0f-83f9-4266-bd61-f3e2f583d59d is the unique id of the instance which is preferred to the the-re prefix that could race with another identical openstack server create command.
To verify that the instance name matches the IPv4 address that is pre-set in the DNS:

$ ssh ubuntu@the-re019 hostname
Warning: Permanently added '' (ECDSA) to the list of known hosts.

Thanks to Josh Durgin for suggesting this solution.

Posted in openstack | 1 Comment

Delete the last port of an OpenStack router

When trying to delete an OpenStack subnet and the associated router, the command neutron router-delete complains because of the port allocated for the gateway and the port of the gateway cannot be removed with neutron port-delete because it is owned by the router. The solution is to clear the owner of the port with something like:

neutron port-update --device-owner clear 7f9685cb-794d-4847

and then delete the router. This is on Icehouse as provided by Enter Cloud Suite.

Posted in openstack | 2 Comments

OpenStack script to pre-allocate fixed IPs

The create-ports.py script allocates ports and indirectly gets fixed IPs from the DHCP server. The ports are named openstack000, openstack001 etc. and they are displayed in a format suitable for dnsmasq:

$ python create-ports.py --count 2 --net fsf-lan |  \
   sudo tee /etc/dnsmasq.d/openstack

If fsf-lan is a network shared with other tenants, it makes sure the IP are reserved, although they are not yet bound to an instance.

$ neutron port-list
| id                                   | name         | ...
| 1d1a05b1-383d-49ef-ae75-5ddcb5c714db | openstack001 |....

An new instance can then be given a known IP with:

$ openstack server create --image ubuntu-trusty-14.04 \
  --flavor 1cpu-1G \
  --key-name teuthology \
  --nic net-id=d936f445-5d68-485a-94f2-b852fd6b7d0c,v4-fixed-ip= \
  --wait openstack001

In the case of teuthology it is useful because the DNS can be configured once and for all while instances are dynamically created using IPs from the DNS instead of relying on allocation from the OpenStack DHCP server.
Continue reading

Posted in openstack | Leave a comment

create / delete an OpenStack instance with python-openstackclient

The python-openstackclient library has an example that provides the basic structure for a new command (the auth_url problem workaround may be needed). To create a virtual machine with 1GB RAM, 1CPU, ubuntu-14.04, using the teuthology keypair on the fsf-lan network, the matching flavor, image, keypair and network objects can be found with:

    for flavor in client_manager.compute.flavors.list():
        if flavor.ram == 1024 and flavor.vcpus == 1:
    for network in client_manager.compute.networks.list():
        if network.label == 'fsf-lan':
    for image in client_manager.compute.images.list():
        if 'ubuntu' in image.name and '14.04' in image.name:
    for keypair in client_manager.compute.keypairs.list():
        if keypair.name == 'teuthology':

The test instance can then be created

   server = client_manager.compute.servers.create('test',
                        image, flavor,
                        nics=[{'net-id': network.id}])

but it won’t be immediately active and the wait_for_status can be used to block until it is:

from openstackclient.common import utils

Deleting the instance is simpler:

    utils.wait_for_delete(client_manager.compute.servers.get, server.id)

See create-delete.py for a standalone script including the above lines that can be run as:

$ python create-server.py --help
usage: create-server.py [-h] [--os-compute-api-version ]
$ python create-server.py
FLAVOR: {'name': u'm1.small', ...
NETWORK: {'cidr_v6': None, 'dns2': None, 'dns1': None, 'netmask': None, 'label': u'fsf-lan',...
IMAGE: {'status': u'ACTIVE', 'updated': u'2014-05-19T11:43:00Z', 'name': u'ubuntu-trusty-14.04',...
KEYPAIR: {'public_key': u'ssh-rsa AAAAB3...

Continue reading

Posted in openstack | 2 Comments