Welcome to the Free Software contributions diary of Loïc Dachary. Although the posts look like blog entries, they really are technical reports about the work done during the day. They are meant to be used as a reference by co-developers and managers. Erasure Code Patents StreamScale.

Ceph Jerasure and ISA plugins benchmarks

In Ceph, a pool can be configured to use erasure coding instead of replication to save space. When used with Intel processors, the default Jerasure plugin that computes erasure code can be replaced by the ISA plugin for better write performances. Here is how they compare on a Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz.

Encoding and decoding all used 4KB objects which is the default stripe width. Two variants of the jerasure plugins were used: Generic (jerasure_generic) and SIMD (erasure_sse4) which is used when running on an Intel processor with SIMD instructions.
This benchmark was run after compiling from sources using

$ ( cd src ; make ceph_erasure_code_benchmark )
$ TOTAL_SIZE=$((4 * 1024 * 1024 * 1024)) \
CEPH_ERASURE_CODE_BENCHMARK=src/ceph_erasure_code_benchmark \
PLUGIN_DIRECTORY=src/.libs \
  qa/workunits/erasure-code/bench.sh fplot | \
  tee qa/workunits/erasure-code/bench.js

and displayed with

firefox qa/workunits/erasure-code/bench.html
Posted in ceph, jerasure | Leave a comment

Improving Ceph python scripts tests

The Ceph command line and ceph-disk helper are python scripts for which there are integration tests (ceph-disk.sh and test.sh). It would be useful to add unit tests and pep8 checks.
It can be done by creating a python module instead of an isolated file (see for instance ceph-detect-init) with a tox.ini file including pep8, python2 and python3 test environments.
Since Ceph relies on autotools, the setup.py can be used with -local targets. For instance:

all-local::
        python setup.py build
clean-local::
	python setup.py clean
install-data-local::
	python setup.py install --root=$(DESTDIR) --install-layout=deb

Note the double : meaning it appends to an existing rule instead of overriding it. The –root=$(DESTDIR) will install the module files in the appropriate directory when building packages.
tox uses pip to fetch dependencies required to run tests from PyPI, but tests sometime run without network access. The depedencies can be collected by wheel with something like:

pip wheel -r requirements.txt

It will create a wheelhouse directory which can later be used with

pip install --no-index --use-wheel --find-links=wheelhouse \
  -r requirements.txt

Continue reading

Posted in ceph | Leave a comment

Testing if a jenkins container finished booting

When running Jenkins as a docker container for test purposes, it is necessary to verify the Jenkins master is fully functional before running the first test cases.
The http interface can be tested with a call to the API such as

curl --silent http://jenkins.host/api/json

It will first fail with Connection reset by peer, then with 503 Server Error: Service Unavailable and return a JSON output after a few seconds.
The Jenkins CLI can be tested by sending the help command.

$ wget -O /tmp/jenkins-cli.jar http://jenkins.host/jnlpJars/jenkins-cli.jar
$ java -jar /tmp/jenkins-cli.jar -s http://jenkins.host help

and it will use port 50000 to connect to the jenkins master. It will first fail with an error such as

SEVERE: I/O error in channel Chunked connection to http://jenkins.host/cli
java.io.StreamCorruptedException: invalid stream header: 0A0A0A0A

meaning the connection to port 50000 failed (the error message is misleading).
It will eventually succeed with

   add-job-to-view
    Adds jobs to view.
  build
...

If security is disabled (which is the default when running the container), a call to the CLI will succeed despite the following error message.

SEVERE: I/O error in channel CLI connection to http://jenkins.host
java.io.IOException: Unexpected termination of the channel
Posted in jenkins | Leave a comment

DNS spoofing with RPZ and bind9

When two web services reside on the same LAN, it may be convenient to spoof DNS entries to use the LAN IP instead of the public IP. It can be done using RPZ and bind9.
For instance workbench.dachary.org can be mapped to 10.0.2.21 with

$ cat /etc/bind/rpz.db
$TTL 60
@            IN    SOA  localhost. root.localhost.  (
                          2   ; serial
                          3H  ; refresh
                          1H  ; retry
                          1W  ; expiry
                          1H) ; minimum
                  IN    NS    localhost.

workbench.dachary.org        A    10.0.2.21

The zone is declared in

$ cat /etc/bind/named.conf.local
zone "rpz" {
      type master;
      file "/etc/bind/rpz.db";
      allow-query {none;};
};

and the response-policy is set in the options file with

$ cat /etc/bind/named.conf.options
...
	response-policy { zone "rpz"; };
};

When bind9 is restarted with /etc/init.d/bind9 restart, the mapping can be verified with

$ dig @127.0.0.1 workbench.dachary.org
workbench.dachary.org.	5	IN	A	10.0.2.21

If the bind9 server runs on a docker host, it can be used by docker containers with

docker run  ... --dns=172.17.42.1 ...
Posted in Uncategorized | Leave a comment

Mirror github pull requests locally

Each GitHub pull request is associated with a reference in the target repository. For instance the commit sent to pull request 3948 is the reference refs/pull/3948/head. If GitHub successfully merges the pull request in the target branch, another reference is associated with it, for instance refs/pull/3948/merge.
It is convenient to mirror pull requests in local branches, for instance to trigger GitLab CI or jenkins git-plugin because they only react to commits that belong to branches, not orphaned references. The tip of the branch can be named pull/XXX and its tip reset to the matching merge reference.
The tip of the branch could be set to the head reference but it would be less effective than using the merge because it is based on an older version of the target branch. Whatever test runs on the head, it may fail although it could succeed after the merge.
GitHub does not set the merge and the head reference atomically: the head can be set before GitHub even tries to merge. Similarly, when a pull request is rebased and forced push, there is a window of opportunity for the merge reference to still be about the previous head.
The following shell function takes care of all these border cases and keeps an up to date set of branches accurately reflecting all pull requests from a GitHub repository:

function import_pull_requests() {
    local remote=$1

    local ref
    local remote_head

    git fetch $remote +refs/pull/*:refs/remotes/$remote/pull/*

    git for-each-ref \
        --sort='-committerdate' \
        --format='%(refname) %(objectname)' \
        refs/remotes/$remote/pull/*/head | \
        while read ref remote_head ; do

        local pr=$(echo $ref | perl -pe 's:.*/pull/(.*)/head:$1:')

        # ignore pull requests that cannot merge
        local merge=$(git rev-parse --quiet --verify
            refs/remotes/$remote/pull/$pr/merge)
        test -z "$merge" && continue

        # ignore pull requests for which the merge does not match the
        # remote head, most likely because it has not been updated yet
        # after a rebase was pushed
        local merged_head=$(git rev-parse
            refs/remotes/$remote/pull/$pr/merge^2)
        test "$merged_head" != "$remote_head" && continue

        # nothing to do if the head did not change since we last saw
        # it
        local local_head=$(git rev-parse --quiet --verify
            refs/pull/$pr/head)
        test "$remote_head" = "$local_head" && continue

        # remember the head for the next round
        git update-ref refs/pull/$pr/head $remote_head

        # create/update a branch with the successfull merge of the
        # head
        git update-ref refs/heads/pull/$pr $merge
        echo branch pull/$pr
    done
}

Download the function and the associated tests

Posted in git, gitlab | Leave a comment

Using a cloud image with kvm

It would be convenient to have a virt-builder oneliner such as

$ virt-builder --arch i386 --ssh-inject ~/.ssh/id_rsa.pub fedora-21

to get an image suitable to run and login with

$ qemu-kvm -m 1024 -net user,hostfwd=tcp::2222-:22 \
  -drive file=fedora-21.qcow2 &
$ ssh -p 2222 localhost grep PRETTY /etc/os-release
PRETTY_NAME="Fedora 21 (Twenty One)"

Docker users have a simpler form because there is no need to ssh to enter the container:

$ docker run fedora:21 grep PRETTY /etc/os-release
PRETTY_NAME="Fedora 21 (Twenty One)"

It is not currently possible to use virt-builder as described above because

  • the set of images available by default is limited (no i386 architecture for instance)
  • the –inject-ssh option is only available in the development version

The libguestfs.org toolbox can however be used to implement a script modifying images prepared for the cloud (see ubuntu cloud images for instance):

  • wget the image
    wget -O my.img http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-i386-disk1.img
    
  • create a config-drive for cloud-init to feed it the ssh public key.
    mkdir -p config-drive
    cat > config-drive/user-data <<EOF
    #cloud-config
    ssh_authorized_keys:
     - $(cat ~/.ssh/id_rsa.pub)
    chpasswd: { expire: False }
    EOF
    cat > config-drive/meta-data <<EOF
    instance-id: iid-123459
    local-hostname: testhost
    EOF
    ( cd config-drive ; LIBGUESTFS_BACKEND=direct virt-make-fs \
      --type=msdos --label=cidata .  ../config-drive.img )
    
  • launch the image with the config drive attached and it will be auto detected
    qemu-kvm -m 1024 -net user,hostfwd=tcp::2222-:22 \
      -drive file=my.img -drive config-drive.img
    

Continue reading

Posted in Uncategorized | 2 Comments

Ceph make check in a ram disk

When running tests from the Ceph sources, the disk is used intensively and a ram disk can be used to reduce the latency. The kernel must be rebooted to set the ramdisk maximum size to 16GB. For instance on Ubuntu 14.04 in /etc/default/grub (the module name which could be rb or brd depending).

GRUB_CMDLINE_LINUX="brd.rd_size=16777216" # 16GB in KB

the grub configuration must then be updated with

sudo update-grub

After reboot the ram disk is formatted as an ext4 file system and mounted:

$ cat /sys/module/brd/parameters/rd_size
16777216
$ sudo mkfs -t ext4 /dev/ram1
$ sudo mount /dev/ram1 /srv
$ df -h /srv
Filesystem      Size  Used Avail Use% Mounted on
/dev/ram1        16G   44M   15G   1% /srv
$ free -g
             total       used       free     shared    buffers     cached
Mem:            31          0         31          0          0          0
-/+ buffers/cache:          0         31

Cloning ceph, compiling and running tests should now take less than 15 minutes with

$ git clone https://github.com/ceph/ceph
$ cd ceph
$ ./run-make-check.sh

When the ram disk is umounted, some of the memory used by the ram disk is still in use

$ free -g
             total       used       free     shared    buffers     cached
Mem:            31         27          4          0          0         17
-/+ buffers/cache:          9         22
$ sudo umount /srv
$ free -g
             total       used       free     shared    buffers     cached
Mem:            31         18         13          0          0          8
-/+ buffers/cache:          9         22

It can be flushed with

$ sudo blockdev --flushbufs /dev/ram1
$ free -g
             total       used       free     shared    buffers     cached
Mem:            31          9         22          0          0          8
-/+ buffers/cache:          0         31

Continue reading

Posted in ceph | Leave a comment

Upgrade nodejs on Ubuntu 14.04

To run gh a version of nodejs more recent than the one packaged by default on Ubuntu 14.04 is required:

$ apt-cache policy nodejs
nodejs:
  Installed: 0.10.25~dfsg2-2ubuntu1
  Candidate: 0.10.25~dfsg2-2ubuntu1
  Version table:
 *** 0.10.25~dfsg2-2ubuntu1 0
        500 http://fr.archive.ubuntu.com/ubuntu/ trusty/universe amd64 Packages
        100 /var/lib/dpkg/status
$ gh watch
fatal: Please update your NodeJS version: http://nodejs.org/download

The recommended way to upgrade is currently broken and the following can be used instead:

sudo add-apt-repository 'deb https://deb.nodesource.com/node trusty main'
sudo apt-get update
sudo apt-get install nodejs

If either apt-get update or apt-get install fail with a message like SSL: certificate subject name:

...
Err https://deb.nodesource.com trusty/main amd64 Packages
  SSL: certificate subject name (login.meteornetworks.com) does not match target host name 'deb.nodesource.com'
Ign http://ceph.com trusty/main Translation-en
Err https://deb.nodesource.com trusty/main i386 Packages
  SSL: certificate subject name (login.meteornetworks.com) does not match target host name 'deb.nodesource.com'
Ign https://deb.nodesource.com trusty/main Translation-en_US
Ign https://deb.nodesource.com trusty/main Translation-en
Ign http://get.docker.io docker/main Translation-en_US
Ign http://get.docker.io docker/main Translation-en
W: Failed to fetch https://deb.nodesource.com/node/dists/trusty/main/binary-amd64/Packages  SSL: certificate subject name (login.meteornetworks.com) does not match target host name 'deb.nodesource.com'
W: Failed to fetch https://deb.nodesource.com/node/dists/trusty/main/binary-i386/Packages  SSL: certificate subject name (login.meteornetworks.com) does not match target host name 'deb.nodesource.com'
E: Some index files failed to download. They have been ignored, or old ones used instead.

The following will fix it:

echo 'Acquire::https::deb.nodesource.com::Verify-Peer "false";' > /etc/apt/apt.conf.d/99verify

Alternatively a version of gh that does not require a recent version of nodejs can be installed with

sudo npm install -g gh@1.9.4
Posted in Uncategorized | Leave a comment

Provisionning a teuthology target with a given kernel

When a teuthology target (i.e. machine) is provisioned with teuthology-lock for the purpose of testing Ceph, there is no way to choose the kernel. But it can be installed afterwards using the following:

cat > kernel.yaml <<EOF
interactive-on-error: true
roles:
- - mon.a
  - client.0
kernel:
   branch: testing
tasks:
- interactive:
EOF

Assuming the target on which the new kernel is to be installed is vpm083, running

$ teuthology  --owner loic@dachary.org \
  kernel.yaml <(teuthology-lock --list-targets vpm083)
...
2015-03-09 17:47 INFO:teuthology.task.internal:Starting timer...
2015-03-09 17:47 INFO:teuthology.run_tasks:Running task interactive...
Ceph test interactive mode, use ctx to interact with the cluster
>>>

will install an alternate kernel and reboot the machine:

[ubuntu@vpm083 ~]$ uname -a
Linux vpm083 3.19.0-ceph-00029-gaf5b96e #1 SMP Thu Mar 5 01:04:25 GNU/Linux
[ubuntu@vpm083 ~]$ lsb_release -a
LSB Version:	:base-4.0-amd64:base-4.0-noarch:
Distributor ID:	RedHatEnterpriseServer
Description:  release 6.5 (Santiago)
Release:	6.5
Codename:	Santiago

Command line arguments to the kernel may be added to /boot/grub/grub.conf. For instance loop.max_part=16 to allow partition creation on /dev/loop devices:

default=0
timeout=5
splashimage=(hd0,0)/boot/grub/splash.xpm.gz
hiddenmenu
title rhel-6.5-cloudinit (3.19.0-ceph-00029-gaf5b96e)
        root (hd0,0)
        kernel /boot/vmlinuz-3.19.0 ro root=LABEL=79d3d2d4  loop.max_part=16
        initrd /boot/initramfs-3.19.0.img
Posted in ceph | Leave a comment

Ceph OSD uuid conversion to OSD id and vice versa

When handling a Ceph OSD, it is convenient to assign it a symbolic name that can be chosen even before it is created. That’s what the uuid argument for ceph osd create is for. Without a uuid argument, a random uuid will be assigned to the OSD and can be used later. Since the ceph osd create uuid is idempotent, it can also be used to lookup the id of a given OSD.

$ osd_uuid=b2e780fc-ec82-4a91-a29d-20cd9159e5f6
# convert the OSD uuid into an OSD id
$ ceph osd create $osd_uuid
0
# convert the OSD id into an OSD uuid
$ ./ceph --format json osd dump | jq '.osds[] | select(.osd==0) | .uuid'
"b2e780fc-ec82-4a91-a29d-20cd9159e5f6"
Posted in ceph | Leave a comment