Welcome to the Free Software contributions diary of Loïc Dachary. Although the posts look like blog entries, they really are technical reports about the work done during the day. They are meant to be used as a reference by co-developers and managers.
OpenStack is installed with puppet and configured to use a Ceph cluster installed with ceph-deploy. The hardware is composed of about 25 heterogeneous HP machines that are over three years old. Five of them have been racked in a basement that not as dry as it should be. The 100Mb/s Internet connection uses a 30m category 5 cable going through two holes in the walls before reaching the rack.
The cluster will be connected to other similar clusters to reduce the risk of loosing data.
Junk hardware, in a basement
Adding, moving and removing disks in a Ceph cluster can easily be automated and require no manual intervention. New disks are formatted when the configuration tool ( Puppet, Chef etc. ) notices they are unknown according to ceph-disk. When disks are removed for whatever reason, Ceph recovers the information it contained, as expected. If disks are hot plugged from one machine to another, udev rules will automatically allow them to rejoin the cluster and update the crush map accordingly.
The micro-osd bash script is 91 lines long and creates a Ceph cluster with a single OSD and a single MON in less that five seconds
$ time bash micro-osd.sh single-osd
starting osd.0 at :/0 osd_data single-osd/osd single-osd/osd.journal
# id weight type name up/down reweight
-1 1 root default
0 1 osd.0 up 1
export CEPH_ARGS='--conf single-osd/ceph.conf'
ceph osd tree
It is meant to be used for integration tests, for example in the context of the openstack-installer puppet manifest to deploy OpenStack. Ceph is configured to run from a directory, on a single host, without cephx, in a non privileged environment and uses about 100MB of disk space.
$ du -sh single-osd/
openstack-installer is a data oriented replacement of puppet-openstack. The following HOWTO runs some basic tests on vagrant virtual machines that are preserved for introspection with:
# vagrant status
# vagrant ssh control_basevm
vagrant@control-server:~$ ps -ax | grep keystone
15020 ? Ss 0:01 /usr/bin/python /usr/bin/keystone-all
The control_basevm runs the horizon dashboard:
The UMR de Génétique Végétale is a state funded french research facility located in a historical monument: a large farm surrounded by fields harvested for experiments. Johann Joets was assigned a most unusual mission : setup a state of the art datacenter in a former pigsty. His colleague Olivier Langella does the system administration for PAPPSO and researched an extendable storage solution able to transparently sustain the loss of any hardware component. A simple Ceph setup was chosen and is in use since 2012.
The roof above the pigsty turned into machine room
R515 Ceph node and switch on top
This is a dialog where Samuel Just, Ceph core developer answers Loïc Dachary’s questions to explain the details of PGBackend (i.e. placement group backend). It is a key componenent of the implementation of erasure coded pools.
When reviewing a large changeset with magit, it can be difficult to separate meaningfull changes from purely syntactic substitutions. Using ediff to navigate the patch highlights the words changed between two lines instead of just showing a line removed and another added.
In the above screenshot the oid change to oid.hobj is a syntactic change where the new block after oid.generation != ghobject_t::NO_GEN deserves more attention.
Posted in emacs, git, magit
Running git_stats against Ceph master today produced the following output.
The stats before 2010 were trimmed because the report is too slow to navigate.
sudo apt-get install ruby1.9.1-dev ruby-mkrf
sudo gem install git_stats
git clone email@example.com:ceph/ceph.git
git_stats generate --from=07005fa1e501973846d666ed073ce64b45144d39
This is a dialog (also published on ceph.com) where Joao Luis, Ceph core developer with an intimate knowledge of the monitor, answers Loïc Dachary’s questions who discovers this area of the Ceph code base. Many answers are covered in Joao’s article Ceph’s New Monitor Changes (March 2013), with a different angle.
July 1st 2013, Heinlein setup a Ceph cuttlefish ( now upgraded to version 0.61.8 ) cluster using the desktop of seven employees willing to host a ceph node and share part of their disk. The nodes are partly connected with 1Gb/s links and some only have 100Mb/s. The cluster supports a 4TB ceph file system
ceph-office$ df -h .
Filesystem Size Used Avail Use% Mounted on
x.x.x.x,y.y.y.y,z.z.z.z:/ 4,0T 2,0T 2,1T 49% /mnt/ceph-office
which is used as a temporary space to exchange files. On a typical day at least one desktop is switched off and on again. The cluster has been self healing since its installation, with the only exception of a placement group being stuck and fixed with a manual pg repair .