Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Can't add journal after replacing disk


kpiti
29 Posts
November 25, 2025, 8:03 pm

Hi,

we had a disk failure (=lot of errors)  and when we got a replacement disk I replaced it like it's in the docs

https://docs.ceph.com/en/quincy/rados/operations/add-or-rm-osds/#replacing-an-osd

Basically

  1. empty disk
  2. stop osd, destroy osd
  3. replace disk
  4. lvm zap, prepare, activate
  5. start osd

The old disk had a journal and cache on external SSD/NvME respectively (cephfs pool). I can get a new OSD up and running but I can't seem to be able to add the journal and cache to the OSD neither via gui (nodes->disk) nor I could find any docs on that on CEPH docs..

The main question is in what point should I add the journal and cache. I remember doing it once before and I could add these in the Petasan GUI afterwards.. Should I just blank the disk and initialize it via the GUI to get these options?

running v 3.3.0

I also see that caching is getting deprecated in newer versions (we're on quincy) so if we upgrade to Petasan 4 (reef) can this become an issue?

Thanks

Jure

#1

admin
3,054 Posts
November 26, 2025, 4:36 am

The manual commands you show are for adding OSD without external journal or cache. You would need additional cli commands to add those manually.

Easier to use use the UI. Once you add a new replacement OSD, go to Node -> Physical Disk List and click on the + button next to the new drive. this open the Add Storage UI where you can select to add OSD with external journal and/or external cache. This assumes you have some free space/partitions on existing journal or cache disks, else you would need to add such devices and click on the + to define them as new journal or cache devices.

The deprecated cache you refer to is a Ceph pool level cache which is no longer used.  The cache supported in PetaSAN is a Linux block device write cache (dm-writecache) which works at the device level and is not deprecated. However as with any generic cache it may not be suitable for all workloads, if in doubt do not use it and just add OSD with external journal.

Last edited on November 26, 2025, 4:36 am by admin · #2

kpiti
29 Posts
November 26, 2025, 10:35 am

Yes, thanks, that worked. I did try it once but it always started *Adding* and at the end it didn't do. I think I found out why it didn't work in the first place - I had leftovers in LVM:

~# pvs

  WARNING: Couldn't find device with uuid xUUPz4-Ifmu-bwvg-YYFw-nzer-4xpg-NBzNkh.

  WARNING: VG ps-2e7a0a56-89a1-481d-b78b-7ed5a44f1881-wc-osd.17 is missing PV xUUPz4-Ifmu-bwvg-YYFw-nzer-4xpg-NBzNkh (last written to [unknown]).

Now I cleaned this up it worked as it should.. I just have to clean up the old one. Do I need to do anything else than

ceph osd purge 17 --yes-i-really-mean-it

In ceph docs there are some additional steps that aren't matched on Petasan..

Thanks a lot

Jure

#3

admin
3,054 Posts
November 26, 2025, 12:48 pm

Very good.

The write cache ( cache + OSD ) was not cleaned correctly using manual cli commands when you removed the OSDs, if you use the UI, it will handle all these cleanups for you.

You can follow all the steps in the docs while still using the UI, for example you may empty the OSD by setting its crush weight to 0 ( in the Maintenance menu), the wait for it to empty before stopping and deleting. But it your case, if the OSD is showing errors, it is better to skip the emptying step.

Note in versions before 4.0, we did not provide a button in UI to stop the OSD, so you had to manually stop it then proceed to delete it from UI.

#4

kpiti
29 Posts
November 26, 2025, 1:38 pm

Yes, I did it in cli as I couldn't find the option to do it in GUI in 3.3.0

Thanks for the explanation..

Cheers, Jure

#5