Welcome to the Free Software contributions diary of Loïc Dachary. Although the posts look like blog entries, they really are technical reports about the work done during the day. They are meant to be used as a reference by co-developers and managers. Erasure Code Patents StreamScale.

How to display the commits from a merged branch ?

A branch gmock was proposed as pull request 483 on GitHub, accepted, merged into master and deleted. It had two commits:

  • bf05ec1 tests: replace existing gtest 1.5.0
  • 5cbe0c5 gmock: use Google C++ Mocking

In GitHub, the reference pull/483/head is preserved and points to the last commit of the branch that no longer exists.

    master +
           |
    849afe + merged and deleted branch gmock
           |\
    d5cb91 + \
           |  + bf05ec1 tests: replace existing gtest 1.5.0 (pull/483/head)
    5301b2 +  |
           |  + 5cbe0c5 gmock: use Google C++ Mocking
           | /
           |/
    5a6549 +
           |
           .
           .
           .

To list the commits that were merged we need to find the commit in master that is immediately after the former branch started.

base=$(git rev-list --topo-order master ^pull/483/head | tail -1)

The base variable contains 5301b2 which is the first commit of master that is not reachable from pull/483/head, in topological order instead of the default chronological order.

$ git rev-list --oneline $base^..pull/483/head
bf05ec1 tests: replace existing gtest 1.5.0
5cbe0c5 gmock: use Google C++ Mocking

Displays the commits of the former gmock branch. Note the ^ after $base that means the first parent of $base. If the graph is as shown above, it does not make a difference. But if it is as follows:

    master +
           |
    849afe + merged and deleted branch gmock
           |\
           | \
           |  + bf05ec1 tests: replace existing gtest 1.5.0 (pull/483/head)
           |  |
           |  + 5cbe0c5 gmock: use Google C++ Mocking
           | /
           |/
    5a6549 +
           |
           .
           .
           .

Then $base would be 849afe and git rev-list –oneline $base..pull/483/head (without the ^) would display nothing because pull/483/head is reachable from $base. Since $base^ is the first parent (i.e. the left parent on the graph above), it is 5a6549 and we get the desired result.

Posted in git | Leave a comment

retrieve github pull requests in JSON

The following python function returns a map associating each pull request number to its JSON description for the given repo. The OAuth token is needed so github will allow more requests to be processed during a given time frame. The result is cached in a file and refreshed every 24 hours.

import urllib2
import json
import re
import os
import time

def get_pull_requests(repo, token):
    # https://developer.github.com/v3/pulls/#list-pull-requests
    pulls_file = "/tmp/pulls.json"
    if ( not os.access(pulls_file, 0) or
         time.time() - os.stat(pulls_file).st_mtime > 24 * 60 * 60 ):
        pulls = {}
        url = ("https://api.github.com/repos/" + repo +
               "/pulls?state=all&access_token=" + token )
        while url:
            github = urllib2.Request(url=url)
            f = urllib2.urlopen(github)
            for pull in json.loads(f.read()):
                pulls[pull['number']] = pull
            url = None
            for link in f.info()['Link'].split(','):
                if 'rel="next"' in link:
                    m = re.search('<(.*)>', link)
                    if m:
                        url = m.group(1)
        with open(pulls_file, 'w') as f:
            json.dump(pulls, f)
    else:
        with open(pulls_file, 'r') as f:
            pulls = json.load(f)
    return pulls

For instance

pulls = get_pull_requests('ceph/ceph', '64933d355fda984108b4aad2c5cd4c4a224aad')

The same pagination logic applies to all API calls (see Web Linking RFC 5988 for more information) and parsing could use the LinkHeader module instead of rudimentary regexp parsing.

Posted in Uncategorized | Leave a comment

gf-complete test coverage report

The make coverage target is added to gf-complete to create a lcov report while running the tests with make check.

The full report is archived.

Posted in jerasure | Leave a comment

Teuthology docker targets hack (4/4)

The teuthology container hack is completed by adding a flag to retrieve packages from a user specified repository instead of gitbuilder.ceph.com. The user can build packages from sources and run a job, which will implicitly save a docker image with the package installed. The second time the same job is run, it will go faster because it reuses the image. For instance the following job:

machine_type: container
os_type: ubuntu
os_version: "14.04"
suite_path: /home/loic/software/ceph/ceph-qa-suite
roles:
- - mon.a
  - osd.0
  - osd.1
  - client.0
overrides:
  install:
    ceph:
      branch: master
  ceph:
    wait-for-scrub: false
tasks:
- install:
    repository_url: http://172.17.42.1/trusty
- ceph:

runs under one minute:

{duration: 47.98, flavor: basic, owner: loic@dachary.org, success: true}
Posted in ceph, docker | 3 Comments

Building Ceph Debian GNU/Linux packages

The following script explains how to create Debian GNU/Linux packages for Ceph from a clone of the sources.

releasedir=/tmp/release
rm -fr releasedir
mkdir -p $releasedir
#
# remove all files not under git so they are not
# included in the distribution.
#
git clean -dxf
#
# git describe provides a version that is
# a) human readable
# b) is unique for each commit
# c) compares higher than any previous commit
# d) contains the short hash of the commit
#
vers=`git describe --match "v*" | sed s/^v//`
#
# creating the distribution tarbal requires some configure
# options (otherwise parts of the source tree will be left out).
#
./autogen.sh
./configure --with-rocksdb --with-ocf --with-rest-bench \
    --with-nss --with-debug --enable-cephfs-java \
    --with-lttng --with-babeltrace
#
# use distdir= to set the name of the top level directory of the
# tarbal to match the desired version
#
make distdir=ceph-$vers dist
#
# rename the tarbal to match debian conventions and extract it
#
mv ceph-$vers.tar.gz $releasedir/ceph_$vers.orig.tar.gz
tar -C $releasedir zxf ceph_$vers.orig.tar.gz
#
# copy the debian directory over and remove -dbg packages
# because they are large and take time to build
#
cp -a debian $releasedir/ceph-$vers/debian
cd $releasedir
perl -ni -e 'print if(!(/^Package: .*-dbg$/../^$/))' ceph-$vers/debian/control
perl -pi -e 's/--dbg-package.*//' ceph-$vers/debian/rules
#
# always set the debian version to 1 which is ok because the debian
# directory is included in the sources and the upstream version will
# change each time it is modified.
#
dvers="$vers-1"
#
# update the changelog to match the desired version
#
cd ceph-$vers
chvers=`head -1 debian/changelog | perl -ne 's/.*\(//; s/\).*//; print'`
if [ "$chvers" != "$dvers" ]; then
   DEBEMAIL="contact@ceph.com" dch -b -v "$dvers" "new version"
fi
#
# create the packages
# a) with ccache to speed things up when building repeatedly
# b) do not sign the packages
# c) use half of the available processors
#
PATH=/usr/lib/ccache:$PATH dpkg-buildpackage -j$(($(nproc) / 2)) -uc -us

Continue reading

Posted in ceph, debian | 2 Comments

jerasure.org installation notes

The jerasure.org is setup to host the upstream repositories for the GF-complete and jerasure libraries. Contributors may sign-up or re-use their existing GitHub account. A companion continous integration server runs make check on each merge request.
Continue reading

Posted in gitlab, jerasure | Leave a comment

Customizing the gitlab home page

The customization of the Gitlab home page is a proprietary extension that is not available in the Free Software version. When running Gitlab from docker containers, the home page template needs to be moved to a file that won’t go away with the container:

$ layouts=/home/git/gitlab/app/views/layouts/
$ docker exec gitlab mkdir -p /home/git/data/$layouts
$ docker exec gitlab mv $layouts/devise.html.haml /home/git/data/$layouts
$ docker exec gitlab ln -s /home/git/data/$layouts/devise.html.haml \
   $layouts/devise.html.haml

The template can now be modifed in /opt/gitlab/data/home/git/gitlab/app/views/layouts/ from the host running the container. It is a HAML template which can have raw HTML as long as proper indentation is respected.

Continue reading

Posted in gitlab | 1 Comment

A make check bot for Ceph contributors

The automated make check for Ceph bot runs on Ceph pull requests. It is still experimental and will not be triggered by all pull requests yet.

It does the following:

A use case for developers is:

  • write a patch and send a pull request
  • switch to another branch and work on another patch while the bot is running
  • if the bot reports failure, switch back to the original branch and repush a fix: the bot will notice the repush and run again

It also helps reviewers who can wait until the bot succeeds before looking at the patch closely.
Continue reading

Posted in ceph, docker | Leave a comment

Teuthology docker targets hack (3/4)

The teuthology container hack is improved so each Ceph command is run via docker exec -i which can read from stdin as of docker 1.4 released in December 2014.
It can run the following job

machine_type: container
os_type: ubuntu
os_version: "14.04"
suite_path: /home/loic/software/ceph/ceph-qa-suite
roles:
- - mon.a
  - osd.0
  - osd.1
  - client.0
overrides:
  install:
    ceph:
      branch: master
  ceph:
    wait-for-scrub: false
tasks:
- install:
- ceph:

under one minute, when repeated a second time and the bulk of the installation can be reused.

{duration: 50.01510691642761, flavor: basic,
  owner: loic@dachary.org, success: true}

Continue reading

Posted in ceph, docker | Leave a comment

Why are by-partuuid symlinks missing or outdated ?

The ceph-disk script manages Ceph devices and rely on the content of the /dev/disk/by-partuuid directory which is updated by udev rules. For instance:

  • a new partition is created with /sbin/sgdisk --largest-new=1 --change-name=1:ceph data --partition-guid=1:83c14a9b-0493-4ccf-83ff-e3e07adae202 --typecode=1:89c57f98-2fe5-4dc0-89c1-f3ad0ceff2be -- /dev/loop4
  • the kernel is notified of the change with partprobe or partx and fires a udev event
  • the udev daemon receives UDEV [249708.246769] add /devices/virtual/block/loop4/loop4p1 (block) and the /lib/udev/rules.d/60-persistent-storage.rules script creates the corresponding symlink.

Let say the partition table is removed later (with sudo sgdisk --zap-all --clear --mbrtogpt -- /dev/loop4 for instance) and the kernel is not notified with partprobe or partx. If the first partition is created again and the kernel is notified as above, it will fail to notice any difference and will not send a udev event. As a result /dev/disk/by-partuuid will contain a symlink that is outdated.
The problem can be fixed by manually removing the stale symlink from /dev/disk/by-partuuid, clearing the partition table and notifying the kernel again. The events sent to udev can be displayed with:

# udevadm monitor
...
KERNEL[250902.072077] change   /devices/virtual/block/loop4 (block)
UDEV  [250902.100779] change   /devices/virtual/block/loop4 (block)
KERNEL[250902.101235] remove   /devices/virtual/block/loop4/loop4p1 (block)
UDEV  [250902.101421] remove   /devices/virtual/block/loop4/loop4p1 (block)
...

The environment and scripts used for a block device can be displayed with

# udevadm test /block/sdb/sdb1
...
udev_rules_apply_to_event: IMPORT '/sbin/blkid -o udev -p /dev/sdb/sdb1' /lib/udev/rules.d/60-ceph-partuuid-workaround.rules:28
udev_event_spawn: starting '/sbin/blkid -o udev -p /dev/sdb1'
...
Posted in ceph | Leave a comment