HOWTO extract a stack trace from teuthology (take 2)

When a Ceph teuthology integration test fails (for instance a rados jobs), it will collect core dumps which can be downloaded from the same directory where the logs and config.yaml files can be found, under the remote/mira076/coredump directory.
The binary from which the core dump comes from can be displayed with:

$ file 1425077911.7304.core
ELF 64-bit LSB  core file x86-64, version 1, from 'ceph-osd -f -i 3'

The teuthology logs contains command lines that can be used to install the corresponding binaries:

$ echo deb http://gitbuilder.ceph.com/ceph-deb-trusty-x86_64-basic\
/sha1/e54834bfac3c38562987730b317cb1944a96005b trusty main | \
  sudo tee /etc/apt/sources.list.d/ceph.list
$ sudo apt-get update
$ sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes \
  -o Dpkg::Options::="--force-confdef" \
  -o Dpkg::Options::="--force-confold" install \
  ceph=0.80.8-75-ge54834b-1trusty \
  ceph-dbg=0.80.8-75-ge54834b-1trusty

The ceph-dbg package contains debug symbols that will automatically be used by gdb(1):

$ gdb /usr/bin/ceph-osd 1425077911.7304.core
...
Reading symbols from /usr/bin/ceph-osd...
Reading symbols from /usr/lib/debug//usr/bin/ceph-osd...done.
...
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `ceph-osd -f -i 3'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f59d6e9af07 in _dl_map_object_deps at dl-deps.c:528
(gdb) bt
#0  0x00007f59d6e9af07 in _dl_map_object_deps at dl-deps.c:528
#1  0x00007f59d6ea1aab in dl_open_worker at dl-open.c:272
#2  0x00007f59d6e9cff4 in _dl_catch_error  at dl-error.c:187
#3  0x00007f59d6ea13bb in _dl_open    at dl-addr.c:61
#5  __GI__dl_addr at dl-addr.c:137
#6  0x00007f59c06dcbc0 in ?? ()
#7  0x00007f59d70b11c8 in _r_debug ()
#8  0x00007f59c06dcba0 in ?? ()
#9  0x00007f59c06dcbb0 in ?? ()
#10 0x00007f59c06dcb90 in ?? ()
#11 0x00007f59c06dca94 in ?? ()
#12 0x0000000000000000 in ?? ()
This entry was posted in ceph. Bookmark the permalink.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>