Deactivate all Ceph OSDs on a node.

Disabling Ceph OSDs on a node for maintenance #

Oh wow this is a pain in the ass #

So, I needed to do some infra maintenance. and quickly discovered that the process of safely offlining the OSDs on a node was taking forever.

Scriptification for the win #

The steps we need to take are fairly simple.

The Steps to automate: #

Set ceph to not freak out about offline OSDs ↕

Enumerate all the local OSDs ↕

Stop each OSD ↕

Clear to perform Maintenance ↕

shutdown -h now

Do the needful ↕

Bring it all back online ↕

Disable Noout ↕

Tie it all together #

for osd in `ls -lA /var/lib/ceph/osd|awk '/ceph/{print $9}'|sed -e 's/ceph-//'`; do
  echo $osd;
  service "ceph-osd@${osd}" stop >/dev/null 2>&1 &
  for i in `seq 1 3`; do
    echo -n '.' ;
    sleep 1;
  done;
  echo;
done

Do Maintenance here

Start node back up

Validate ceph is healthy

/usr/bin/watch /usr/bin/ceph status

Disable Noout

ceph osd set noout