Ceph Radosgw Upgrade

Well well well…

What’s going on here? #

So, I was updating Ceph to Reef¹, on our ProxMox cluster, and stubbed my toe again.

Logs #

radosgw logs

root@pxm4:~# systemctl restart ceph-radosgw@pxm4.service; journalctl -fu ceph-radosgw@pxm4.service
Apr 21 13:38:19 pxm4 radosgw[1393158]: 2024-04-21T13:38:19.336-0500 7464287b0d80 -1 AuthRegistry(0x7fffb0c58c20) no keyring found at /etc/pve/priv/ceph.client.pxm4.keyring, disabling cephx
Apr 21 13:38:19 pxm4 radosgw[1393158]: failed to fetch mon config (--no-mon-config to skip)
Apr 21 13:38:19 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Main process exited, code=exited, status=1/FAILURE
Apr 21 13:38:19 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Failed with result 'exit-code'.
Apr 21 13:38:19 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Scheduled restart job, restart counter is at 5.
Apr 21 13:38:19 pxm4 systemd[1]: Stopped ceph-radosgw@pxm4.service - Ceph rados gateway.
Apr 21 13:38:19 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Start request repeated too quickly.
Apr 21 13:38:19 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Failed with result 'exit-code'.
Apr 21 13:38:19 pxm4 systemd[1]: Failed to start ceph-radosgw@pxm4.service - Ceph rados gateway.
Apr 22 14:48:31 pxm4 systemd[1]: Started ceph-radosgw@pxm4.service - Ceph rados gateway.
Apr 22 14:48:31 pxm4 radosgw[3122482]: 2024-04-22T14:48:31.723-0500 791e66cacd80 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.pxm4.keyring: (2) No such file or directory
Apr 22 14:48:31 pxm4 radosgw[3122482]: 2024-04-22T14:48:31.723-0500 791e66cacd80 -1 AuthRegistry(0x555fe9dddc40) no keyring found at /etc/pve/priv/ceph.client.pxm4.keyring, disabling cephx
Apr 22 14:48:31 pxm4 radosgw[3122482]: 2024-04-22T14:48:31.727-0500 791e66cacd80 -1 auth: unable to find a keyring on /etc/pve/priv/ceph.client.pxm4.keyring: (2) No such file or directory
Apr 22 14:48:31 pxm4 radosgw[3122482]: 2024-04-22T14:48:31.727-0500 791e66cacd80 -1 AuthRegistry(0x7fff6ab7fde0) no keyring found at /etc/pve/priv/ceph.client.pxm4.keyring, disabling cephx
Apr 22 14:48:31 pxm4 radosgw[3122482]: failed to fetch mon config (--no-mon-config to skip)
Apr 22 14:48:31 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Main process exited, code=exited, status=1/FAILURE
Apr 22 14:48:31 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Failed with result 'exit-code'.
Apr 22 14:48:32 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Scheduled restart job, restart counter is at 1.
Apr 22 14:48:32 pxm4 systemd[1]: Stopped ceph-radosgw@pxm4.service - Ceph rados gateway.
Apr 22 14:48:32 pxm4 systemd[1]: Started ceph-radosgw@pxm4.service - Ceph rados gateway.

This Looks… Familiar… #

I was pretty sure I’d fixed this before, an so it was a simple matter of remembering wth I did…

Fortunately I wrote it up! So it was a pretty quick fix..

Essentially the ExecStart line in /lib/systemd/system/ceph-radosgw@.service needed a little massaging.

ExecStart=/usr/bin/radosgw -f --cluster ${CLUSTER} --name client.%i --setuser ceph --setgroup ceph

It needed to change to:

ExecStart=/usr/bin/radosgw -f --setuser ceph --setgroup ceph --cluster ${CLUSTER} --name client.radosgw.%i --keyring=/etc/ceph/ceph.client.radosgw.keyring

Validating #

root@pxm4:~# systemctl daemon-reload
root@pxm4:~# systemctl restart ceph-radosgw@$(hostname).service
journalctl -fu ceph-radosgw@$(hostname).service

...Aaaaand We're back!

Apr 22 14:48:33 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Main process exited, code=exited, status=1/FAILURE
Apr 22 14:48:33 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Failed with result 'exit-code'.
Apr 22 14:48:33 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Scheduled restart job, restart counter is at 5.
Apr 22 14:48:33 pxm4 systemd[1]: Stopped ceph-radosgw@pxm4.service - Ceph rados gateway.
Apr 22 14:48:33 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Start request repeated too quickly.
Apr 22 14:48:33 pxm4 systemd[1]: ceph-radosgw@pxm4.service: Failed with result 'exit-code'.
Apr 22 14:48:33 pxm4 systemd[1]: Failed to start ceph-radosgw@pxm4.service - Ceph rados gateway.
...

Apr 22 15:15:35 pxm4 systemd[1]: Started ceph-radosgw@pxm4.service - Ceph rados gateway.

https://docs.ceph.com/en/latest/releases/reef/ ↩︎