Welcome to the Free Software contributions diary of Loïc Dachary. Although the posts look like blog entries, they really are technical reports about the work done during the day. They are meant to be used as a reference by co-developers and managers.

HOWTO migrate an AMI from Essex to a bootable volume on Havana

A snapshot of an Essex OpenStack instance contains an AMI ext3 file system. It is rsync’ed to a partitioned volume in the Havana cluster. After installing grub from chroot, a new instance can be booted from the volume.
Continue reading

Posted in Essex, Havana, openstack | 1 Comment

Backporting with magit

Ceph uses GF-Complete stable branch (v1). Modifications required by Ceph were proposed to the master branch. While waiting for the corresponding pull request to be reviewed, a temporary branch (v1-ceph) was created in the Ceph name space for immediate use. The changes have been accepted in master, they are to be backported to v1
Continue reading

Posted in emacs, git, magit | Leave a comment

Merging with magit and ediff

The jerasure v2 branch is merged into the v2-ceph branch.

Continue reading

Posted in Uncategorized | Leave a comment

Celebrate Firefly and Icehouse

If you’re in Atlanta Sunday 11th, may 2014 evening, for the OpenStack summit or any other reason, join us to celebrate the OpenStack Icehouse release and the Ceph Firefly release. There will be both OpenStack and Ceph developers present and no presentations planned : it’s 100% informal :-)

Registration is here : Celebrate Firefly and Icehouse meetup page.

Posted in ceph, openstack | Leave a comment

SSE optimization for erasure code in Ceph

The jerasure library is the default erasure code plugin of Ceph. The gf-complete companion library supports SSE optimizations at compile time, when the compiler provides them (-msse4.2 etc.). The jerasure (and gf-complete with it) plugin is compiled multiple times with various levels of SSE features:

  • jerasure_sse4 uses SSE4.2, SSE4.1, SSSE3, SSE3, SSE2, SSE
  • jerasure_sse3 uses SSSE3, SSE3, SSE2, SSE
  • jerasure_generic uses no SSE instructions

When an OSD loads the jerasure plugin, the CPU features are probed and the appropriate plugin is selected depending on their availability.
The gf-complete source code is cleanly divided into functions that take advantage of specific SSE features. It should be easy to use the ifunc attribute to semi-manually select each function individually, at runtime and without performance penalty (because the choice is made the first time the function is called and recorded for later calls). With such a fine grain selection, there would be no need to compile three plugins because each function would be compiled with exactly the set of flag it needs.

Posted in ceph | 2 Comments

Testing CPU features with Qemu

The Ceph erasure code plugin must run on Intel CPU that have no SSE4.2 support. A Qemu is run without SSE4.2 support:

qemu-system-x86_64 -machine accel=kvm:tcg -m 2048 \
  -drive file=server.img -boot c \
  -display sdl \
  -net nic -net user,hostfwd=tcp::2222-:22 \
  -fsdev local,security_model=passthrough,id=fsdev0,path=~/ceph \
  -device virtio-9p-pci,id=fs0,fsdev=fsdev0,mount_tag=hostshare

The qemu CPU has no SSE4.2 although the native CPU has it:

$ grep sse4.2 /proc/cpuinfo | wc -l
$ ssh -p 2222 loic@ grep sse4.2 /proc/cpuinfo | wc -l

The local development directory is a Plan 9 folder shared over Virtio mounted inside the VM:

sudo mount -t 9p -o trans=virtio,version=9p2000.L hostshare /home/loic/ceph

and the functional test is run to assert that encoding and decoding an object:

$ cd /home/loic/ceph/src
$ ./unittest_erasure_code_jerasure
[----------] Global test environment tear-down
[==========] 16 tests from 8 test cases ran. (30 ms total)
[  PASSED  ] 16 tests.
Posted in Uncategorized | Leave a comment

Your first exabyte in a Ceph cluster

$ rbd create --size $((1024 * 1024 * 1024 * 1024)) tiny
$ rbd info tiny
rbd image 'tiny':
	size 1024 PB in 274877906944 objects
	order 22 (4096 kB objects)
	block_name_prefix: rb.0.1009.6b8b4567
	format: 1

Note: rbd rm tiny will take a long time.

Posted in ceph | Leave a comment

The footprints of 192 Ceph developers

Gource is run on the Ceph git repository for each of the 192 developers who contributed to its development over the past six years. Their footprint is the last image of a video clip created from all the commits they authored.

video clip

Sage Weil

video clip

Yehuda Sadeh

video clip

Greg Farnum

video clip

Samuel Just

video clip

Colin P. McCabe

video clip

Danny Al-Gaaf

video clip

Josh Durgin

video clip

John Wilkins

video clip

Loic Dachary

video clip

Dan Mick

Continue reading

Posted in ceph, git, gource | Leave a comment

working with git submodules in Ceph

The gf-complete and jerasure libraries implement the erasure code functions used in Ceph. They were copied in Ceph in 2013 because there were no reference repositories at the time. The copy was removed from the Ceph repository and replaced by git submodules to decouple the release cycles.
Continue reading

Posted in ceph, git | Leave a comment

BIOS and console access via VNC

The AMT of an ASRock Q87M motherboard is configured to enable remote power control (power cycle) and display of the BIOS and the console. It is a cheap alternative to iLO or IPMI that can be used with Free Software. AMT is a feature of vPro that was available in 2011 with some Sandy Bridge chipsets. It is included in many of the more recent Haswell chipsets.

The following is a screenshot of vinagre connected to the AMT VNC server displaying the BIOS of the ASRock Q87M motherboard.

Continue reading

Posted in AMT, VNC, vPro | Leave a comment