Category Archives: Havana

What cinder volume is missing an RBD object ?

Although it is extremely unlikely to loose an object stored in Ceph, it is not impossible. When it happens to a Cinder volume based on RBD, knowing which has an object missing will help with disaster recovery.

Posted in Havana, ceph, openstack | Leave a comment

enable secondary network interface and ignore the default route

When two network interfaces are associated to an OpenStack instance, the Ubuntu precise guest will only configure the first one. Assuming the second can be configured via DHCP, it can be added with: cat > /etc/network/interfaces.d/eth1.cfg <<EOF auto eth1 iface … Continue reading

Posted in Havana, openstack | Leave a comment

Recovering from a cinder RBD host failure

OpenStack Havana Cinder volumes associated with a RBD Ceph pool are bound to a host. cinder service-list –host bm0014.the.re@rbd-ovh +—————+———————–+——+———+——-+ | Binary | Host | Zone | Status | State | +—————+———————–+——+———+——-+ | cinder-volume | bm0014.the.re@rbd-ovh | ovh | enabled … Continue reading

Posted in Havana, ceph, openstack | 3 Comments

Non profit OpenStack & Ceph cluster distributed over five datacenters

A few non profit organizations (April, FSF France, tetaneutral.net…) and volunteers constantly research how to get compute, storage and bandwidth that are: 100% Free Software Content neutral Low maintenance Reliable Cheap The latest setup, in use since ocbober 2013, is … Continue reading

Posted in Havana, ceph, openstack | 5 Comments

Fixing OpenVSwitch and GRE asymetric performances

OpenStack Havana is configured to use OpenVSwitch 1.10.2 as packaged for Ubuntu precise, with a linux-3.11 kernel.The cluster is connected to a 100Mb/s link. When sending data from an instance to the internet (using iperf), it shows ~90Mb/s. When receiving … Continue reading

Posted in Havana, openstack | 8 Comments

Two minor pitfalls when upgrading Havana stable

When upgrading an OpenStack compute or l3 agent node from 1:2013.2 to 1:2013.2.3 on Ubuntu precise 12.04.4: The nova-compute version 1:2013.2 is expected to fail with /var/log/nova/nova-compute.log IncompatibleObjectVersion: Version 1.9 of Instance is not supported when interfaced with a 1:2013.2.3. … Continue reading

Posted in Havana, openstack | Leave a comment

Reseting an instance {power,vm,task}_state in Havana

Sometime, after an hypervisor crash or nova-compute error, an OpenStack instance can be left in a state that cannot be conveniently fixed with nova reset-state. $ nova list +————————————–+—————-+———+… | ID | Name | Status | +————————————–+—————-+———+… | ca9496e9-0bd2-4734-9cf9-eb4e264628f7 | … Continue reading

Posted in Havana, openstack | Leave a comment

HOWTO migrate an AMI from Essex to a bootable volume on Havana

A snapshot of an Essex OpenStack instance contains an AMI ext3 file system. It is rsync’ed to a partitioned volume in the Havana cluster. After installing grub from chroot, a new instance can be booted from the volume.

Posted in Essex, Havana, openstack | 1 Comment

wget on an OpenStack instance hangs ? Try lowering the MTU

Why would OpenStack instances fail to wget a URL and work perfectly on others ? For instance: $ wget -O – ‘https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/autobuild.asc’ Connecting to ceph.com (ceph.com)|208.113.241.137|:443… connected. HTTP request sent, awaiting response… 200 OK Length: unspecified [text/plain] Saving to: `STDOUT’ … Continue reading

Posted in Havana, openstack, tinc | 1 Comment

Mixing Ceph and LVM volumes in OpenStack

Ceph pools are defined to collocate volumes and instances in OpenStack Havana. For volumes that do not need the resilience provided by Ceph, a LVM cinder backend is defined in /etc/cinder/cinder.conf: [lvm] volume_group=cinder-volumes volume_driver=cinder.volume.drivers.lvm.LVMISCSIDriver volume_backend_name=LVM and appended to the list … Continue reading

Posted in Havana, ceph, openstack | Leave a comment