Welcome to the Free Software contributions diary of Loïc Dachary. Although the posts look like blog entries, they really are technical reports about the work done during the day. They are meant to be used as a reference by co-developers and managers. Erasure Code Patents StreamScale.

Gitlab CI installation

Assuming a GitLab container has been installed via Docker, a GitLab CI can be installed and associated with it. It needs a separate database server:

sudo mkdir -p /opt/mysql-ci/data
docker run --name=mysql-ci -d -e 'DB_NAME=gitlab_ci_production'  \
 -e 'DB_USER=gitlab_ci'  \
 -e 'DB_PASS=XXXXX'  \
 -v /opt/mysql-ci/data:/var/lib/mysql  sameersbn/mysql:latest

but it can re-use the redis server from GitLab

docker pull sameersbn/gitlab-ci
sudo mkdir -p /opt/gitlab-ci/data
docker run --name='gitlab-ci' -it --rm   \
  --link mysql-ci:mysql \
  --link redis:redisio \
  --link gitlab:gitlab \
  -e 'SMTP_ENABLED=true' \
  -e 'SMTP_USER=' \
  -e 'SMTP_HOST='  \
  -e 'SMTP_PORT=25'  \
  -e 'SMTP_STARTTLS=false'  \
  -e 'GITLAB_CI_PORT=8080'  \
  -e 'GITLAB_CI_HOST=workbench.dachary.org'  \
  -p 8080:80  \
  -v /var/run/docker.sock:/run/docker.sock  \
  -v /opt/gitlab-ci/data:/home/gitlab_ci/data  \
  -v $(which docker):/bin/docker  sameersbn/gitlab-ci

It uses port 8080 because port 80 is already in use by GitLab. The SMTP* are the same as when GitLab was installed.

The user and password are the same as with the associated GitLab.

Posted in docker, gitlab | Leave a comment

Copy a github pull request to gitlab

A mirror of a github repository is setup and contains two remotes:

gitlab	 git@workbench.dachary.org:tests/testrepo.git (push)
origin	 https://github.com/loic-bot/testrepo (push)

The github2gitlab command of gh (run from ~gitmirrors/repositories/Tests/testrepo) creates a merge request in gitlab by copying the designated pull request from github:

$ gh gg --user loic-bot --repo testrepo --number 3

Original github pull request

Matching gitlab merge request

Continue reading

Posted in gitlab | Leave a comment

Ceph read-only mirror on gitlab

The gitlab-mirrors scripts are installed to setup a a read-only Ceph mirror, updated hourly. It is used for permalinks such as src/osd/ClassHandler.cc#L170.
Continue reading

Posted in ceph, gitlab | Leave a comment

HOWTO debug a teuthology task

To debug a modification to a ceph-qa-suite task ( for instance repair_test.py), a teuthology target is locked with:

$ ./virtualenv/bin/teuthology-lock --lock-many 1 --owner loic@dachary.org
$ ./virtualenv/bin/teuthology-lock --list-targets --owner loic@dachary.org > targets.yaml

and used to run the test with:

./virtualenv/bin/teuthology \
  --suite-path $HOME/software/ceph/ceph-qa-suite \
  --owner loic@dachary.org \
  $HOME/software/ceph/ceph-qa-suite/suites/rados/basic/tasks/repair_test.yaml \

where roles.yaml sets all roles to one target:

- [mon.0, osd.0, osd.1, osd.2, osd.3, osd.4, client.0]

Each run requires the installation and deinstallation of all Ceph packages and takes minutes. The installation part of repair_test.yaml can be commented out and the packages installed manually.

$ cat repair.yaml
#- install:
- ceph:
- repair_test:

Continue reading

Posted in ceph | Leave a comment

Gitlab workbench

Gitlab is installed on http://workbench.dachary.org using docker images. redis is installed first, as an independant container:

docker pull sameersbn/redis:latest
docker run --name=redis -d sameersbn/redis:latest

then MySQL

docker pull sameersbn/mysql:latest
docker run --name=mysql -d \
  -e 'DB_NAME=gitlabhq_production' \
  -e 'DB_USER=gitlab' \
  -v /opt/mysql/data:/var/lib/mysql \

and finally gitlab

docker pull sameersbn/gitlab:latest
docker run --name='gitlab' -it -d  \
  --link mysql:mysql --link redis:redisio \
  -e 'GITLAB_EMAIL=gitlab@workbench.dachary.org'  \
  -e 'SMTP_ENABLED=true' \
  -e 'SMTP_DOMAIN=workbench.dachary.org' \
  -e 'SMTP_USER=' \
  -e 'SMTP_HOST=' \
  -e 'SMTP_PORT=25' \
  -e 'SMTP_STARTTLS=false' \
  -e 'GITLAB_SIGNUP=true' \
  -e 'GITLAB_PORT=80' \
  -e 'GITLAB_HOST=workbench.dachary.org' \
  -e 'OAUTH_ALLOW_SSO=true' \
  -e 'OAUTH_GITHUB_API_KEY=github Client ID'  \
  -e 'OAUTH_GITHUB_APP_SECRET=github Client Secret' \
  -e 'GITLAB_SSH_PORT=22' \
  -p 22:22 -p 80:80 \
  -v /var/run/docker.sock:/run/docker.sock \
  -v /opt/gitlab/data:/home/git/data \
  -v $(which docker):/bin/docker \

The ssh server of the server will need to bind another port by editing /etc/ssh/sshd_config, changing the Port value and restarting the server with stop ssh ; start ssh.
The OmniAuth single sign on is configured following gitlab instructions, except for editing the config.yml file: the OAUTH_GITHUB_* are set instead, using information found in the applications settings github page.
It uses the automagic dockerlinks to connect it to the MySQL and redis servers (–link mysql:mysql –link redis:redisio). The SMTP server is configured using variables from the documentation to point to the server running on the host ( is the IP of the docker0 bridge on which all containers are connected and in the same IP range as the dynamic IP they are given). A postfix server is installed on the host:

$ sudo apt-get install postfix
... chose internet server ...

and it is configured to accept to relay mails from any docker contain in the IP range:

$ cat /etc/postfix/main.cf
myhostname = workbench.dachary.org
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
myorigin = /etc/mailname
mydestination = workbench.dachary.org, localhost, localhost.localdomain, localhost
relayhost =
mynetworks = [::ffff:]/104 [::1]/128

A working SMTP server is required to allow sign up as required with GITLAB_SIGNUP=true. The gitlab persistent data is in /opt/mysql/data (bind mounted with -v /opt/mysql/data:/var/lib/mysql) for the MySQL database and /opt/gitlab/data (bind mounted with -v /opt/gitlab/data:/home/git/data) for repositories, gitlab assets etc. When the host reboots, the containers can be restarted as above, they only contains non persistent information.

Posted in gitlab | Leave a comment

Teuthology docker targets hack (2/4)

The teuthology container hack is improved to snapshot the container after Ceph and its dependencies have been installed. It helps quickly testing ceph-qa-suite tasks. A job doing nothing but install the Firefly version of Ceph takes 14 seconds after the initial installation (which can take between 5 to 15 minutes depending on how fast is the machine and how much bandwidth is available).

2014-11-17 01:21:00,067.067 INFO:teuthology.worker:Reserved job 42
2014-11-17 01:21:00,067.067 INFO:teuthology.worker:Config is:
machine_type: container
name: foo
os_type: ubuntu
os_version: '14.04'
    ceph: {branch: firefly}
owner: loic@dachary.org
priority: 1000
- [mon.a, osd.0, osd.1, client.0]
- {install: null}
tube: container
verbose: false

Fetching from upstream into /home/loic/src/ceph-qa-suite_master
completed on container001: sudo lsb_release '-is':  Ubuntu
reusing existing image ceph-base-ubuntu-14.04-firefly
running 'docker' 'stop' 'container001'
completed ('docker', 'stop', u'container001') on container001:  container001
2014-11-17 01:21:31,677.677 INFO:teuthology.run:Summary data:
{duration: 14, flavor: basic, success: true}
2014-11-17 01:21:31,677.677 INFO:teuthology.run:pass

Continue reading

Posted in ceph, docker | Leave a comment

Running make check on Ceph pull requests

Each Ceph contribution is expected to successfully run make check and pass all the unit tests it contains. The developer runs make check locally before submitting his changes but the result may be influenced by the development environment. A draft bot is proposed to watch the list of pull requests on a github repository and run a script based on github3.py each time a new patch is uploaded.

cephbot.py --user loic-bot --password XXXXX \
   --owner ceph --repository ceph \
   --script $HOME/makecheck/check.sh

If the script fails, it adds a comment with the output of the run to the pull request. Otherwise it reports success in the same way.
Continue reading

Posted in ceph | 2 Comments

make -j150 ceph

A power8 machine was recently donated to the GCC compile farm and /proc/cpuinfo shows 160 processors. Compiling Ceph from sources with make -j150 makes for a nice htop display.

The result of the compilation passes most of the unit tests, with a few minor exceptions such as bloom filter.

Posted in FSF, ceph | 1 Comment

Teuthology docker targets hack (1/4)

teuthology runs jobs testing the Ceph integration on targets that can either be virtual machines or bare metal. The container hack adds support for docker containers as a replacement.

Running task exec...
Executing custom commands...
Running commands on role mon.a host container002
running sudo 'TESTDIR=/home/ubuntu/cephtest' bash '-c' '/bin/true'
running docker exec container002 bash /tmp/tmp/tmptJ7hxa
Duration was 0.088931 seconds

Continue reading

Posted in ceph, docker | Leave a comment

Ceph make -j8 check in less than 3mn

The Ceph sources contain tests that can be run with make check. As of v0.85 then can only be run sequentially because some tests bind the same ports and use the same files. It takes around 18 minutes on a spinner and 12 minutes on a SSD because of some I/O intensive ones. It becomes problematic because it’s a long time to wait when adding code and trying to validate it works and also because it keeps increasing as more tests are added. It’s also a recurring frustration because they conflict with vstart.sh cluster running for manual testing.
The tests have been reworked to ensure that none of them use the the same port or the same files. It reduces the time to 12mn on a spinner and around 2mn on a SSD with make -j8 check.

[loic@rex001 src]$ time make -j8 check
make[4]: Entering directory `/home/loic/ceph/src'
./check_version ./.git_version
make[1]: Leaving directory `/home/loic/ceph/src'

real    2m21.907s
user    5m45.958s
sys     1m50.431s

A number of tests such as qa/workunits/cephtool/test.sh take a long time but do not require much CPU or disk I/O. When on a 4 core machine setting -j8 gives a chance for these tests to run while more CPU intensive tests are using most of the CPU power.
Using larger values (for instance -j16) does not help much because a few tests take around 3mn to run anyway.

Posted in ceph | Leave a comment