Category Archives: crush

Installing python-crush on CentOS 7 without network

To install python-crush on a CentOS 7 that does not have access to internet, the necessary files must be downloaded via an USB drive. The python34-pip package must be installed from the EPEL repository the machine uses for maintenance purposes. … Continue reading

Posted in crush | Leave a comment

A tool to rebalance uneven Ceph pools

The algorithm to fix uneven CRUSH distributions in Ceph was implemented as the crush optimize subcommand. Given the output of ceph report, crush analyze can show buckets that are over/under filled: $ ceph report > ceph_report.json $ crush analyze –crushmap … Continue reading

Posted in ceph, crush | Leave a comment

An algorithm to fix uneven CRUSH distributions in Ceph

The current CRUSH implementation in Ceph does not always provide an even distribution. The most common cause of unevenness is when only a few thousands PGs, or less, are mapped. This is not enough samples and the variations can be … Continue reading

Posted in ceph, crush, libcrush | Leave a comment

Ceph space lost due to overweight CRUSH items

When a CRUSH bucket contains five Ceph OSDs with the following weights: weight osd.0 5 osd.1 1 osd.2 1 osd.3 1 osd.4 1 20% of the space in osd.0 will never be used by a pool with two replicas. The … Continue reading

Posted in ceph, crush | Leave a comment

Ceph full ratio and uneven CRUSH distributions

A common CRUSH rule in Ceph is step chooseleaf firstn 0 type host meaning Placement Groups (PGs) will place replicas on different hosts so the cluster can sustain the failure of any host without losing data. The missing replicas are … Continue reading

Posted in ceph, crush, libcrush | Leave a comment

Improving PGs distribution with CRUSH weight sets

In a Ceph cluster with a single pool of 1024 Placement Groups (PGs), the PG distribution among devices will not be as expected. (see Predicting Ceph PG placement for details about this uneven distribution). In the following, the difference between … Continue reading

Posted in ceph, crush, libcrush | Leave a comment