HOWTO debug a teuthology task

To debug a modification to a ceph-qa-suite task ( for instance repair_test.py), a teuthology target is locked with:

$ ./virtualenv/bin/teuthology-lock --lock-many 1 --owner loic@dachary.org
$ ./virtualenv/bin/teuthology-lock --list-targets --owner loic@dachary.org > targets.yaml

and used to run the test with:

./virtualenv/bin/teuthology \
  --suite-path $HOME/software/ceph/ceph-qa-suite \
  --owner loic@dachary.org \
  $HOME/software/ceph/ceph-qa-suite/suites/rados/basic/tasks/repair_test.yaml \
  roles.yaml

where roles.yaml sets all roles to one target:

roles:
- [mon.0, osd.0, osd.1, osd.2, osd.3, osd.4, client.0]

Each run requires the installation and deinstallation of all Ceph packages and takes minutes. The installation part of repair_test.yaml can be commented out and the packages installed manually.

$ cat repair.yaml
...
tasks:
#- install:
- ceph:
- repair_test:


The existing packages are removed:

sudo apt-get remove --purge python-ceph ceph-common librados2 librbd1 ceph-fuse libcephfs1-dbg libcephfs-java libcephfs1 libcephfs-jni ceph-fs-common

The repository is set to the desired branch (for instance wip-10018-primary-erasure-code-hinfo), as compiled by gitbuilder.cgi with:

$ echo deb http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic/ref/wip-10018-primary-erasure-code-hinfo trusty main | sudo tee /etc/apt/sources.list.d/ceph.list

The packages are installed:

$ sudo apt-get update
$ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph ceph-dbg ceph-mds ceph-mds-dbg ceph-common ceph-common-dbg ceph-fuse ceph-fuse-dbg ceph-test ceph-test-dbg radosgw radosgw-dbg python-ceph libcephfs1 libcephfs1-dbg libcephfs-java librados2 librados2-dbg librbd1 librbd1-dbg

and the repositories cleaned (this must be done again after each test completes):

$ rm -fr cephtest/ && sudo rm -fr /var/lib/ceph && sudo rm -fr /var/log/ceph/*

Utilities must be copied from the teuthology sources with (assuming vpm178 is the target):

$ scp teuthology/task/daemon-helper teuthology/task/adjust-ulimits ubuntu@vpm178:/tmp
$ ssh ubuntu@vpm178 sudo cp -a /tmp/{daemon-helper,adjust-ulimits} /usr/bin
This entry was posted in ceph. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>