Lowering Ceph scrub I/O priority

Note: the following does not currently work in Firefly because of http://tracker.ceph.com/issues/9677 . It has been backported to Firefly and will likely be in 0.80.8.

The disk I/O of a Ceph OSD thread scrubbing is the same as all other threads by default. It can be lowered with ioprio options for all OSDs with:

ceph tell osd.* injectargs '--osd_disk_thread_ioprio_priority 7'
ceph tell osd.* injectargs '--osd_disk_thread_ioprio_class idle'

All other threads in the OSD will be be (best effort) with priority 4 which is the default for daemons. The disk thread will show as idle:

$ sudo iotop --batch --iter 1 | grep 'ceph-osd -i 0' | grep -v be/4
 4156 idle loic        0.00 B/s    0.00 B/s  0.00 %  0.00 % ./ceph-osd -i 0 ..


The change will only be effective if the scheduler is cfq (it can safely be modified at runtime)

# cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
# echo cfq > /sys/block/sda/queue/scheduler

To display the current value for a given OSD the following can be run from the host on which it is running (because it uses the asok file found in /var/run/ceph):

$ ceph daemon osd.0 config get osd_disk_thread_ioprio_class
{ "osd_disk_thread_ioprio_class": "idle"}
$ ceph daemon osd.0 config get osd_disk_thread_ioprio_priority
{ "osd_disk_thread_ioprio_priority": "7"}
This entry was posted in ceph. Bookmark the permalink.

3 Responses to Lowering Ceph scrub I/O priority

  1. Lionel Bouton says:

    Is the change permanent? If not, how can me make it so?

    I’ve difficulties checking the current state on my servers, for example on the server runing osd.0 :
    # ceph daemon osd.0 config get osd_disk_thread_ioprio_class
    { “error”: “error getting ‘osd_disk_thread_ioprio_class’: (2) No such file or directory”}

    /var/run/ceph/ceph-osd.0.asock exists, this is 0.80.5 on Gentoo Linux packaged by the distribution.

Leave a Reply to Loic Dachary Cancel reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>