Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Recurring MDS memory growth / OOM on PetaSAN 4.1.0 (Ceph 18.2.7)


AlbertHakvoort
25 Posts
October 7, 2026, 6:06 pm

Environment

  • PetaSAN 4.1.0, Ceph 18.2.7 reef, kernel 6.12.0-03-petasan
  • 6 nodes, 71 HDD OSDs (11 to 12 per node), 93 GB RAM per node, no swap. Every node also runs OSDs; 3 of them run mon, mgr and MDS.
  • One CephFS (about 75M objects, 490 TB data) with Samba and CTDB on top, about 25 concurrent SMB sessions
  • 1 active MDS and 2 standby MDS
  • One scheduled CIFS snapshot per day with snapshots_to_keep set to 7
  • mds_cache_memory_limit was 8 GiB, later lowered to 5 GiB

1. Original symptom
HEALTH_WARN "N OSD(s) experiencing slow operations in BlueStore" (BLUESTORE_SLOW_OP_ALERT) kept coming back, always on OSDs of the same node.

2. Cause of the slow ops warning

  • The kernel log on that node showed repeated OOM kills of ceph-mds: 10 times in 3 weeks, RSS between 19 and 26 GB, getting worse over time.
  • At each kill the OSDs on that node flapped down and up within the same second. The MDS failed over to a standby in 5 to 7 seconds without data loss.
  • That short flap was enough to trigger the alert: bluestore_slow_ops_warn_threshold defaults to 1 and bluestore_slow_ops_warn_lifetime to 86400 seconds, so one blip keeps the warning for 24 hours.
  • Restarting the flagged OSDs clears it at once. I set threshold to 10 and lifetime to 300, in line with the false-positive reports in Ceph tracker issues 70010 and 70940.

3. MDS memory is not bounded by mds_cache_memory_limit

  • Lowering the limit from 8 to 5 GiB changed nothing. RSS still grew to 25 GB and the MDS was killed again.
  • Health showed MDS_CACHE_OVERSIZED (9GB/5GB) and at times MDS_CLIENT_RECALL.
  • perf dump showed about 175,000 strays and about 2.2M inodes in cache. The purge queue was idle and only a few thousand inodes were in use by clients.
  • "ceph mds fail" refuses to run while MDS_CACHE_OVERSIZED or MDS_TRIM is active and returns EPERM. Restarting the ceph-mds service directly on the node works and behaves like the earlier OOM kills.

4. Kernel slab leak on one node (separate from Ceph)

  • After the MDS crash-looped on a standby node (6 OOM kills in about 30 minutes), that node had only 405 MB MemAvailable, while the process list showed only about 27 GB of RSS out of 85 GB used.
  • meminfo showed Slab at 30 GB and SUnreclaim at 29.7 GB.
  • slabinfo showed about 108 million objects each in the cred, kmalloc-cg-16 and kmalloc-cg-32 caches. The nearly identical counts look like one allocation repeated, which suggests a cgroup or credential leak.
  • A reboot dropped SUnreclaim to 1 GB and raised MemAvailable from 405 MB to 45 GB.
  • Other nodes show the same effect at lower levels after long uptimes: SUnreclaim of 2.5, 4, 9.6 and 12.8 GB at uptimes of 12 to 27 weeks. The MDS crash loop may have accelerated it on the affected node, but that is a guess.

5. Missing pv package breaks snapshot deletion jobs

  • Every snapshot-delete job output contains "sh: pv: not found". The delete script pipes the snapshot removal command into pv for a progress bar, and pv is not installed on the nodes.
  • The removal itself still runs (only 7 snapshots exist, so rotation works), but progress and cleanup are broken and about 90 stale job files had piled up.
  • I installed pv on all nodes. It does not seem to be a dependency of the PetaSAN packages.

6. Current state: daily memory spike during snapshot rotation
The MDS is no longer OOM-killed, but it now ramps up every day while the 18:00 snapshot job runs (new snapshot, delete the 8th oldest). The delete job takes 50 to 60 minutes.

  • All day: flat at about 6 to 7 GB
  • 18:00: 6.5 GB
  • 18:30: 7.7 GB
  • 18:45: 10.5 GB
  • 19:00: 13.6 GB

On a day I let it run without intervention the peak was only about 7 GB, and RSS returned to about 6 GB afterwards. Some nodes have only 9 to 15 GB available, so a 14 GB MDS is not safe there. As a workaround a small watchdog restarts the MDS when RSS exceeds 12 GB and both standbys are healthy. It triggers most evenings.

My hypothesis, not verified: deleting the oldest snapshot lets the MDS process the strays and old inodes that snapshot was holding, which temporarily inflates memory.

Questions

  1. Is a large MDS memory spike during snapshot removal on a big CephFS tree known or expected? Is there a tunable to bound it?
  2. Is the cred and kmalloc-cg slab growth known for kernel 6.12.0-03-petasan, and is a fixed kernel planned?
  3. Would you recommend running the MDS only on nodes without OSDs? Can PetaSAN place the MDS role on a node without OSDs?
#1

admin
3,088 Posts
October 8, 2026, 4:53 am

Are you using SSD for journals to the HDD OSDs ?
Are you using SSD OSDs for cephfs metadata pool ?
On dashboard chart Node Statistics -> % Disk Util, are the disks very busy during snapshot delete?
Do you have cephfs clients or only using SMB/CIFS ?

what is value of
ceph tell osd.XX config show | grep snap_trim
(replace XX with osd id on node )

what is value of
ceph config get mds mds_cache_memory_limit
ceph config get mds mds_recall_max_caps
ceph config get mds mds_cache_reservation

Note that mds_cache_memory_limit is not a hard max limit but a target, too much load on slow system can have this limit exceeded.

You can install mds on a new node in PetaSAN manually using cli.

Last edited on October 8, 2026, 5:00 am by admin · #2

AlbertHakvoort
25 Posts
October 8, 2026, 5:59 pm

Thanks for the quick reply. Answers below.

Hardware (6 identical nodes)

  • Supermicro server, 2x Intel Xeon Silver 4215R (32 threads), 96 GB RAM, no swap
  • Broadcom/LSI SAS3008 HBA (SAS-3, Fusion-MPT) for all data and cache drives
  • Network: Intel X722 10GBASE-T and QLogic FastLinQ QL41000 10/25GbE (bonded)
  • Per node: 11-12 HDD OSDs, 71 in total
    • HDD: Western Digital Ultrastar SAS, mostly WUH721818AL5204 (18 TB, shown as 16.4 TiB), plus one WUH722020BL5204 (20 TB, shown as 18.2 TiB) in a few slots
    • BlueStore DB: 2x Intel Optane SSDPED1D480GA (NVMe, 480 GB) per node, 7 partitions of 60 GiB each, one partition per OSD
    • Writecache: 2x Seagate Nytro XA960LE10063 (960 GB SSD, SATA-class, behind the SAS HBA) per node, one LVM writecache of about 112 GiB per OSD

Your questions

  1. SSD for the HDD OSDs: yes, DB on NVMe plus an LVM writecache on SSD, as above.
  2. SSD OSDs for the CephFS metadata pool: no. All 71 OSDs are device class hdd, and the metadata pool (125 GB) uses the same CRUSH rule as the data pool, so metadata is on the HDD OSDs (with NVMe DB and SSD writecache). I do not have SSD OSDs to move it to.
  3. Disk utilization during the snapshot delete: not measured yet (the node statistics logs for that window were unreadable). I will measure during tonight’s 18:00 to 19:00 window and report back.
  4. CephFS clients: only 6 kernel clients (kernel 6.12.0-03-petasan), one on each PetaSAN node, mounted at the filesystem root and used by Samba/CTDB. No other CephFS clients.

Values

  • osd_snap_trim_priority 1, osd_snap_trim_sleep_hdd 5.0, osd_snap_trim_cost 1048576, osd_pg_max_concurrent_snap_trims 2
  • mds_cache_memory_limit 5368709120 (lowered from 8 GiB), mds_recall_max_caps 30000, mds_cache_reservation 0.05

Noted that mds_cache_memory_limit is a target, not a hard limit. In our case the MDS RSS stays flat at 6 to 7 GB all day and rises to 13 to 14 GB during the 18:00 snapshot job (new snapshot, delete the oldest), then falls back after the job finishes.

One of the six kernel clients holds about 497,000 caps while the other five hold 20 to 37. Samba on that node has only about 1,300 open file descriptors in total, so these are cached inodes, not open files. The MDS cannot trim inodes that carry client caps, which would explain why the cache stays oversized. Is there a recommended way to make the kernel client release caps sooner (for example the caps_max mount option or a higher vfs_cache_pressure), and would a large caps count on one client inflate MDS memory during snapshot creation and deletion?

#3

admin
3,088 Posts
October 8, 2026, 8:40 pm

Let me know what you observe for Disk % Util for HDDs as well as write cache disks.

#4

admin
3,088 Posts
October 12, 2026, 11:36 am

Have not heard back, but in all cases i would recommend you first start with slowing the snap trim process

ceph config set osd osd_snap_trim_sleep 0
ceph config set osd osd_snap_trim_sleep_hybrid 5

the second thing you should increase ram to 128G

after this you can try changing the mds parameters earlier to recall caps sooner

Last edited on October 12, 2026, 11:38 am by admin · #5

AlbertHakvoort
25 Posts
October 14, 2026, 6:06 pm

Quick follow-up on my earlier posts about the MDS memory growth. I think I finally found the cause, and it is in the snapshot delete script itself.

Our MDS sits at about 6 GB all day and then grows linearly from roughly 18:20 to 19:00, reaching 13 to 14 GB every single evening. That is exactly when the daily snapshot job runs (new snapshot at 18:00, oldest one removed). I first suspected strays, but I sampled the MDS every 2 minutes during that window on 9 October and the stray count stayed flat at about 1,270 the whole time and the purge queue was idle. What did grow was the number of inodes in the MDS cache, from 1.3 million at 18:26 to 2.8 million at 18:58. So something is walking the tree. The disks were not the problem either: HDDs averaged 25 to 35 percent (peaks around 55), the write cache SSDs about 1 percent and the NVMe DB devices 0.

Then I looked at how long the delete job actually runs. It starts at 18:00:01 and finishes after 56 to 60 minutes every day, while removing a snapshot with rmdir only takes seconds. Digging into delete_snapshot.py (lines 311 and 485 on 4.1.0) I found the reason: before removing the snapshot it runs a recursive du over the whole snapshot directory and counts the lines, and that number is only used as the total for the pv progress bar. Our oldest snapshot has 24,343,188 entries (about 21.6 million files, 171 TB), so that is 24 million stat calls through the kernel client, and each one pulls an inode into the MDS cache. That matches both the job duration and the memory curve.

The nice thing is that CephFS already keeps this number. Reading the ceph.dir.rentries virtual xattr on the snapshot directory returned 24,343,188 in about 1 ms, without walking anything. So I replaced the du call on both lines with that (it falls back to 0 if the attribute cannot be read, which would only mean no percentage in the progress bar) and kept a backup of the original script on every node. The first run with this change is the 18:00 job on 15 October, and I will report back here with the job duration and the MDS memory curve.

One more small thing I ran into on the way: the same script pipes the removal into pv, but pv is not installed on the nodes, so every delete job logged “pv: not found” and about 90 job output files had piled up. I installed pv by hand, but maybe it should be a dependency of the package.

Could you consider changing the delete script so it does not walk the whole snapshot for the progress total? On a big filesystem this turns the daily snapshot rotation into almost an hour of heavy MDS load, and on nodes that also run OSDs it can end in an MDS OOM kill, which is what started all this for us.

#6

admin
3,088 Posts
October 15, 2026, 5:59 pm

Thanks for the feedback on snapshot deletion, will look into your suggestion.

#7