Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

Not Recovering after Power loss on 6 nodes out off 16


raf.schandevyl@outlook.be
3 Posts
December 2, 2025, 9:50 am

Hi,

Yesterday we had a powerfailure on 6 nodes of our 17 node cluster.  All nodes came up and are running now but we are stuck in recoring.  All osd's are up and running at this moment.

we have an error recovery_unfound.

root@PetaSAN-01:~# ceph -w
cluster:
id: 3715a3a9-7800-4daa-b94b-3cd909bee0e2
health: HEALTH_ERR
6/25091800 objects unfound (0.000%)
Possible data damage: 4 pgs recovery_unfound
Degraded data redundancy: 13711/63912056 objects degraded (0.021%), 4 pgs degraded, 4 pgs undersized

services:
mon: 3 daemons, quorum PetaSAN-03,PetaSAN-01,PetaSAN-02 (age 25h)
mgr: PetaSAN-01(active, since 25h), standbys: PetaSAN-02, PetaSAN-03
osd: 137 osds: 137 up (since 24h), 137 in (since 24h); 4 remapped pgs

data:
pools: 13 pools, 2897 pgs
objects: 25.09M objects, 96 TiB
usage: 250 TiB used, 518 TiB / 767 TiB avail
pgs: 13711/63912056 objects degraded (0.021%)
3500/63912056 objects misplaced (0.005%)
6/25091800 objects unfound (0.000%)
2889 active+clean
4 active+clean+scrubbing+deep
3 active+recovery_unfound+undersized+degraded+remapped
1 active+forced_recovery+recovery_unfound+undersized+degraded+remapped

io:
client: 3.5 MiB/s rd, 34 MiB/s wr, 245 op/s rd, 1.74k op/s wr

Detailed info on from the pg :

ceph health detail
HEALTH_ERR 6/25091800 objects unfound (0.000%); Possible data damage: 4 pgs recovery_unfound; Degraded data redundancy: 13711/63912056 objects degraded (0.021%), 4 pgs degraded, 4 pgs undersized
[WRN] OBJECT_UNFOUND: 6/25091800 objects unfound (0.000%)
pg 13.5d has 2 unfound objects
pg 13.4f has 1 unfound objects
pg 13.bb has 1 unfound objects
pg 13.dc has 2 unfound objects
[ERR] PG_DAMAGED: Possible data damage: 4 pgs recovery_unfound
pg 13.4f is active+forced_recovery+recovery_unfound+undersized+degraded+remapped, acting [44], 1 unfound
pg 13.5d is active+recovery_unfound+undersized+degraded+remapped, acting [19], 2 unfound
pg 13.bb is active+recovery_unfound+undersized+degraded+remapped, acting [18], 1 unfound
pg 13.dc is active+recovery_unfound+undersized+degraded+remapped, acting [19], 2 unfound
[WRN] PG_DEGRADED: Degraded data redundancy: 13711/63912056 objects degraded (0.021%), 4 pgs degraded, 4 pgs undersized
pg 13.4f is stuck undersized for 24h, current state active+forced_recovery+recovery_unfound+undersized+degraded+remapped, last acting [44]
pg 13.5d is stuck undersized for 24h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [19]
pg 13.bb is stuck undersized for 24h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [18]
pg 13.dc is stuck undersized for 24h, current state active+recovery_unfound+undersized+degraded+remapped, last acting [19]

ceph pg 13.5d query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+recovery_unfound+undersized+degraded+remapped",
"epoch": 94261,
"up": [
19,
21
],
"acting": [
19
],
"backfill_targets": [
"21"
],
"acting_recovery_backfill": [
"19",
"21"
],
"info": {
"pgid": "13.5d",
"last_update": "94261'110911327",
"last_complete": "0'0",
"log_tail": "94246'110906858",
"last_user_version": 110911327,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94064,
"last_interval_started": 94063,
"last_epoch_clean": 93886,
"last_interval_clean": 93885,
"last_epoch_split": 5790,
"last_epoch_marked_full": 0,
"same_up_since": 94063,
"same_interval_since": 94063,
"same_primary_since": 94004,
"last_scrub": "93570'110521648",
"last_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_deep_scrub": "93570'110521648",
"last_deep_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_clean_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "94261'110911327",
"reported_seq": 138198325,
"reported_epoch": 94261,
"state": "active+recovery_unfound+undersized+degraded+remapped",
"last_fresh": "2025-12-02T10:45:07.109453+0100",
"last_change": "2025-12-01T10:31:50.831357+0100",
"last_active": "2025-12-02T10:45:07.109453+0100",
"last_peered": "2025-12-02T10:45:07.109453+0100",
"last_clean": "2025-12-01T09:44:32.411125+0100",
"last_became_active": "2025-12-01T10:31:50.610089+0100",
"last_became_peered": "2025-12-01T10:31:50.610089+0100",
"last_unstale": "2025-12-02T10:45:07.109453+0100",
"last_undegraded": "2025-12-01T10:31:50.710891+0100",
"last_fullsized": "2025-12-01T10:31:50.304039+0100",
"mapping_epoch": 94063,
"log_start": "94246'110906858",
"ondisk_log_start": "94246'110906858",
"created": 5348,
"last_epoch_clean": 93886,
"parent": "0.0",
"parent_split_bits": 7,
"last_scrub": "93570'110521648",
"last_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_deep_scrub": "93570'110521648",
"last_deep_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_clean_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"objects_scrubbed": 3389,
"log_size": 4469,
"log_dups_size": 0,
"ondisk_log_size": 4469,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 357,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 356.764345832,
"objects_trimmed": 0,
"snaptrim_duration": 0.0011844589999999999,
"stat_sum": {
"num_bytes": 14189470720,
"num_objects": 3389,
"num_object_clones": 0,
"num_object_copies": 6778,
"num_objects_missing_on_primary": 2,
"num_objects_missing": 2,
"num_objects_degraded": 3391,
"num_objects_misplaced": 0,
"num_objects_unfound": 2,
"num_objects_dirty": 3389,
"num_whiteouts": 0,
"num_read": 27148457,
"num_read_kb": 1216962196,
"num_write": 110744774,
"num_write_kb": 2007501838,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 553,
"num_bytes_recovered": 492805632,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
19,
21
],
"acting": [
19
],
"avail_no_missing": [
"6"
],
"object_location_counts": [
{
"shards": "19",
"objects": 3389
}
],
"blocked_by": [],
"up_primary": 19,
"acting_primary": 19,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 94064,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "3",
"pgid": "13.5d",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3389,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "6",
"pgid": "13.5d",
"last_update": "94062'110778626",
"last_complete": "94062'110778626",
"log_tail": "94031'110774858",
"last_user_version": 110778626,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94041,
"last_interval_started": 94040,
"last_epoch_clean": 93886,
"last_interval_clean": 93885,
"last_epoch_split": 5790,
"last_epoch_marked_full": 0,
"same_up_since": 94063,
"same_interval_since": 94063,
"same_primary_since": 94004,
"last_scrub": "93570'110521648",
"last_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_deep_scrub": "93570'110521648",
"last_deep_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_clean_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 94063,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
19,
21
],
"acting": [
19
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 19,
"acting_primary": 19,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 94041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "12",
"pgid": "13.5d",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3389,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "21",
"pgid": "13.5d",
"last_update": "94261'110911327",
"last_complete": "94261'110911327",
"log_tail": "94031'110774858",
"last_user_version": 110766871,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94064,
"last_interval_started": 94063,
"last_epoch_clean": 93886,
"last_interval_clean": 93885,
"last_epoch_split": 5790,
"last_epoch_marked_full": 0,
"same_up_since": 94063,
"same_interval_since": 94063,
"same_primary_since": 94004,
"last_scrub": "93570'110521648",
"last_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_deep_scrub": "93570'110521648",
"last_deep_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_clean_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "93984'110766871",
"reported_seq": 137971106,
"reported_epoch": 94003,
"state": "remapped+peering",
"last_fresh": "2025-12-01T10:14:53.394066+0100",
"last_change": "2025-12-01T10:14:53.394066+0100",
"last_active": "2025-12-01T10:14:53.393923+0100",
"last_peered": "2025-12-01T10:14:53.392269+0100",
"last_clean": "2025-12-01T09:44:32.411125+0100",
"last_became_active": "2025-12-01T10:05:28.935218+0100",
"last_became_peered": "2025-12-01T10:05:28.935218+0100",
"last_unstale": "2025-12-01T10:14:53.394066+0100",
"last_undegraded": "2025-12-01T10:14:53.394066+0100",
"last_fullsized": "2025-12-01T10:14:53.394066+0100",
"mapping_epoch": 94063,
"log_start": "93627'110763968",
"ondisk_log_start": "93627'110763968",
"created": 5348,
"last_epoch_clean": 93886,
"parent": "0.0",
"parent_split_bits": 7,
"last_scrub": "93570'110521648",
"last_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_deep_scrub": "93570'110521648",
"last_deep_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"last_clean_scrub_stamp": "2025-11-28T04:53:09.071538+0100",
"objects_scrubbed": 3389,
"log_size": 2903,
"log_dups_size": 115,
"ondisk_log_size": 2903,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 357,
"scrub_schedule": "periodic scrub scheduled @ 2025-12-07T10:56:19.782251+0000",
"scrub_duration": 356.764345832,
"objects_trimmed": 0,
"snaptrim_duration": 0.0011844589999999999,
"stat_sum": {
"num_bytes": 14193847808,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3389,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
19,
21
],
"acting": [
19
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [
3,
6,
19
],
"up_primary": 19,
"acting_primary": 19,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 94064,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2025-12-01T10:31:50.304086+0100",
"might_have_unfound": [
{
"osd": "3",
"status": "not queried"
},
{
"osd": "6",
"status": "already probed"
},
{
"osd": "12",
"status": "not queried"
},
{
"osd": "21",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [
"21"
],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"pull_from_peer": [],
"pushing": []
}
}
},
{
"name": "Started",
"enter_time": "2025-12-01T10:31:49.686435+0100"
}
],
"scrubber": {
"active": false,
"must_scrub": true,
"must_deep_scrub": true,
"must_repair": true,
"need_auto": false,
"scrub_reg_stamp": "1.000000",
"schedule": "queued for deep scrub"
},
"agent_state": {}
}
ceph pg 13.dc query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+recovery_unfound+undersized+degraded+remapped",
"epoch": 94261,
"up": [
19,
23
],
"acting": [
19
],
"backfill_targets": [
"23"
],
"acting_recovery_backfill": [
"19",
"23"
],
"info": {
"pgid": "13.dc",
"last_update": "94261'136564251",
"last_complete": "0'0",
"log_tail": "94246'136559790",
"last_user_version": 136564251,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 5784,
"epoch_pool_created": 5339,
"last_epoch_started": 94063,
"last_interval_started": 94062,
"last_epoch_clean": 93891,
"last_interval_clean": 93890,
"last_epoch_split": 5784,
"last_epoch_marked_full": 0,
"same_up_since": 94062,
"same_interval_since": 94062,
"same_primary_since": 94004,
"last_scrub": "93527'135654156",
"last_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"last_deep_scrub": "90682'133200229",
"last_deep_scrub_stamp": "2025-11-06T10:44:01.592877+0100",
"last_clean_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "94261'136564251",
"reported_seq": 186464958,
"reported_epoch": 94261,
"state": "active+recovery_unfound+undersized+degraded+remapped",
"last_fresh": "2025-12-02T10:47:51.339236+0100",
"last_change": "2025-12-01T10:31:50.304422+0100",
"last_active": "2025-12-02T10:47:51.339236+0100",
"last_peered": "2025-12-02T10:47:51.339236+0100",
"last_clean": "2025-12-01T09:44:32.678311+0100",
"last_became_active": "2025-12-01T10:31:49.818824+0100",
"last_became_peered": "2025-12-01T10:31:49.818824+0100",
"last_unstale": "2025-12-02T10:47:51.339236+0100",
"last_undegraded": "2025-12-01T10:31:49.922685+0100",
"last_fullsized": "2025-12-01T10:31:49.685839+0100",
"mapping_epoch": 94062,
"log_start": "94246'136559790",
"ondisk_log_start": "94246'136559790",
"created": 5784,
"last_epoch_clean": 93891,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "93527'135654156",
"last_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"last_deep_scrub": "90682'133200229",
"last_deep_scrub_stamp": "2025-11-06T10:44:01.592877+0100",
"last_clean_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"objects_scrubbed": 3467,
"log_size": 4461,
"log_dups_size": 1707,
"ondisk_log_size": 4461,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 43,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 42.499997208000003,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 14517697024,
"num_objects": 3467,
"num_object_clones": 0,
"num_object_copies": 6934,
"num_objects_missing_on_primary": 2,
"num_objects_missing": 2,
"num_objects_degraded": 3469,
"num_objects_misplaced": 0,
"num_objects_unfound": 2,
"num_objects_dirty": 3467,
"num_whiteouts": 0,
"num_read": 57273328,
"num_read_kb": 2923940539,
"num_write": 136424313,
"num_write_kb": 2160081354,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 4625,
"num_bytes_recovered": 17446706688,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
19,
23
],
"acting": [
19
],
"avail_no_missing": [
"24"
],
"object_location_counts": [
{
"shards": "19",
"objects": 3467
}
],
"blocked_by": [],
"up_primary": 19,
"acting_primary": 19,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 94063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "11",
"pgid": "13.dc",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3467,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "12",
"pgid": "13.dc",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3467,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "23",
"pgid": "13.dc",
"last_update": "94261'136564251",
"last_complete": "94261'136564251",
"log_tail": "94032'136456590",
"last_user_version": 136453260,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 5784,
"epoch_pool_created": 5339,
"last_epoch_started": 94063,
"last_interval_started": 94062,
"last_epoch_clean": 93891,
"last_interval_clean": 93890,
"last_epoch_split": 5784,
"last_epoch_marked_full": 0,
"same_up_since": 94062,
"same_interval_since": 94062,
"same_primary_since": 94004,
"last_scrub": "93527'135654156",
"last_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"last_deep_scrub": "90682'133200229",
"last_deep_scrub_stamp": "2025-11-06T10:44:01.592877+0100",
"last_clean_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "94011'136453259",
"reported_seq": 186272981,
"reported_epoch": 94011,
"state": "active+recovery_wait+degraded",
"last_fresh": "2025-12-01T10:15:02.173656+0100",
"last_change": "2025-12-01T10:14:55.408405+0100",
"last_active": "2025-12-01T10:15:02.173656+0100",
"last_peered": "2025-12-01T10:15:02.173656+0100",
"last_clean": "2025-12-01T09:44:32.678311+0100",
"last_became_active": "2025-12-01T10:14:55.408325+0100",
"last_became_peered": "2025-12-01T10:14:55.408325+0100",
"last_unstale": "2025-12-01T10:15:02.173656+0100",
"last_undegraded": "2025-12-01T10:14:55.407108+0100",
"last_fullsized": "2025-12-01T10:15:02.173656+0100",
"mapping_epoch": 94062,
"log_start": "93629'136449783",
"ondisk_log_start": "93629'136449783",
"created": 5784,
"last_epoch_clean": 93891,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "93527'135654156",
"last_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"last_deep_scrub": "90682'133200229",
"last_deep_scrub_stamp": "2025-11-06T10:44:01.592877+0100",
"last_clean_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"objects_scrubbed": 3467,
"log_size": 3476,
"log_dups_size": 1707,
"ondisk_log_size": 3476,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 43,
"scrub_schedule": "periodic scrub scheduled @ 2025-12-03T14:22:13.852076+0000",
"scrub_duration": 42.499997208000003,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 14518916608,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3467,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
19,
23
],
"acting": [
19
],
"avail_no_missing": [
"23",
"24"
],
"object_location_counts": [
{
"shards": "19,23",
"objects": 3465
},
{
"shards": "23",
"objects": 2
}
],
"blocked_by": [],
"up_primary": 19,
"acting_primary": 19,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 94063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "24",
"pgid": "13.dc",
"last_update": "94061'136460433",
"last_complete": "94061'136460433",
"log_tail": "94032'136456590",
"last_user_version": 136460433,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 5784,
"epoch_pool_created": 5339,
"last_epoch_started": 94041,
"last_interval_started": 94040,
"last_epoch_clean": 93891,
"last_interval_clean": 93890,
"last_epoch_split": 5784,
"last_epoch_marked_full": 0,
"same_up_since": 94062,
"same_interval_since": 94062,
"same_primary_since": 94004,
"last_scrub": "93527'135654156",
"last_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"last_deep_scrub": "90682'133200229",
"last_deep_scrub_stamp": "2025-11-06T10:44:01.592877+0100",
"last_clean_scrub_stamp": "2025-11-25T08:38:11.988865+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 94062,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
19,
23
],
"acting": [
19
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 19,
"acting_primary": 19,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 94041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2025-12-01T10:31:49.685868+0100",
"might_have_unfound": [
{
"osd": "11",
"status": "not queried"
},
{
"osd": "12",
"status": "not queried"
},
{
"osd": "23",
"status": "already probed"
},
{
"osd": "24",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [
"23"
],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"pull_from_peer": [],
"pushing": []
}
}
},
{
"name": "Started",
"enter_time": "2025-12-01T10:31:48.694854+0100"
}
],
"scrubber": {
"active": false,
"must_scrub": true,
"must_deep_scrub": true,
"must_repair": false,
"need_auto": false,
"scrub_reg_stamp": "1.000000",
"schedule": "queued for deep scrub"
},
"agent_state": {}
}

ceph pg 13.bb query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+recovery_unfound+undersized+degraded+remapped",
"epoch": 94261,
"up": [
23,
43
],
"acting": [
18
],
"backfill_targets": [
"23",
"43"
],
"acting_recovery_backfill": [
"18",
"23",
"43"
],
"info": {
"pgid": "13.bb",
"last_update": "94261'110689291",
"last_complete": "0'0",
"log_tail": "94254'110683961",
"last_user_version": 110689291,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94063,
"last_interval_started": 94062,
"last_epoch_clean": 93891,
"last_interval_clean": 93890,
"last_epoch_split": 5348,
"last_epoch_marked_full": 0,
"same_up_since": 94062,
"same_interval_since": 94062,
"same_primary_since": 94012,
"last_scrub": "93478'109774363",
"last_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_deep_scrub": "93478'109774363",
"last_deep_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_clean_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "94261'110689291",
"reported_seq": 142951531,
"reported_epoch": 94261,
"state": "active+recovery_unfound+undersized+degraded+remapped",
"last_fresh": "2025-12-02T10:43:10.100884+0100",
"last_change": "2025-12-01T10:31:50.305322+0100",
"last_active": "2025-12-02T10:43:10.100884+0100",
"last_peered": "2025-12-02T10:43:10.100884+0100",
"last_clean": "2025-12-01T09:44:32.674329+0100",
"last_became_active": "2025-12-01T10:31:49.838532+0100",
"last_became_peered": "2025-12-01T10:31:49.838532+0100",
"last_unstale": "2025-12-02T10:43:10.100884+0100",
"last_undegraded": "2025-12-01T10:31:50.224624+0100",
"last_fullsized": "2025-12-01T10:31:49.688665+0100",
"mapping_epoch": 94062,
"log_start": "94254'110683961",
"ondisk_log_start": "94254'110683961",
"created": 5348,
"last_epoch_clean": 93891,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "93478'109774363",
"last_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_deep_scrub": "93478'109774363",
"last_deep_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_clean_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"objects_scrubbed": 3501,
"log_size": 5330,
"log_dups_size": 0,
"ondisk_log_size": 5330,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 343,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 342.31573375900001,
"objects_trimmed": 0,
"snaptrim_duration": 0.00041727499999999999,
"stat_sum": {
"num_bytes": 14664203776,
"num_objects": 3501,
"num_object_clones": 0,
"num_object_copies": 7002,
"num_objects_missing_on_primary": 1,
"num_objects_missing": 1,
"num_objects_degraded": 3502,
"num_objects_misplaced": 3500,
"num_objects_unfound": 1,
"num_objects_dirty": 3501,
"num_whiteouts": 0,
"num_read": 32102495,
"num_read_kb": 1235259131,
"num_write": 110526746,
"num_write_kb": 2060999420,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 10379,
"num_bytes_recovered": 41608378368,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
23,
43
],
"acting": [
18
],
"avail_no_missing": [
"40"
],
"object_location_counts": [
{
"shards": "18",
"objects": 3501
}
],
"blocked_by": [],
"up_primary": 23,
"acting_primary": 18,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 94063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "4",
"pgid": "13.bb",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3501,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "23",
"pgid": "13.bb",
"last_update": "94261'110689291",
"last_complete": "94261'110689291",
"log_tail": "94032'110597661",
"last_user_version": 110591921,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94063,
"last_interval_started": 94062,
"last_epoch_clean": 93891,
"last_interval_clean": 93890,
"last_epoch_split": 5348,
"last_epoch_marked_full": 0,
"same_up_since": 94062,
"same_interval_since": 94062,
"same_primary_since": 94012,
"last_scrub": "93478'109774363",
"last_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_deep_scrub": "93478'109774363",
"last_deep_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_clean_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "94011'110591919",
"reported_seq": 142764176,
"reported_epoch": 94011,
"state": "active+recovery_wait+degraded+remapped",
"last_fresh": "2025-12-01T10:15:01.835427+0100",
"last_change": "2025-12-01T10:15:00.031374+0100",
"last_active": "2025-12-01T10:15:01.835427+0100",
"last_peered": "2025-12-01T10:15:01.835427+0100",
"last_clean": "2025-12-01T09:44:32.674329+0100",
"last_became_active": "2025-12-01T10:15:00.031261+0100",
"last_became_peered": "2025-12-01T10:15:00.031261+0100",
"last_unstale": "2025-12-01T10:15:01.835427+0100",
"last_undegraded": "2025-12-01T10:14:59.942040+0100",
"last_fullsized": "2025-12-01T10:15:01.835427+0100",
"mapping_epoch": 94062,
"log_start": "93793'110589298",
"ondisk_log_start": "93793'110589298",
"created": 5348,
"last_epoch_clean": 93891,
"parent": "0.0",
"parent_split_bits": 8,
"last_scrub": "93478'109774363",
"last_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_deep_scrub": "93478'109774363",
"last_deep_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_clean_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"objects_scrubbed": 3501,
"log_size": 2621,
"log_dups_size": 391,
"ondisk_log_size": 2621,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 343,
"scrub_schedule": "queued for scrub",
"scrub_duration": 342.31573375900001,
"objects_trimmed": 0,
"snaptrim_duration": 0.00041727499999999999,
"stat_sum": {
"num_bytes": 14660678144,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3501,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
23,
43
],
"acting": [
18
],
"avail_no_missing": [
"23",
"40"
],
"object_location_counts": [
{
"shards": "18,23",
"objects": 3500
},
{
"shards": "23",
"objects": 1
}
],
"blocked_by": [],
"up_primary": 23,
"acting_primary": 18,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 94063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "40",
"pgid": "13.bb",
"last_update": "94061'110601788",
"last_complete": "94061'110601788",
"log_tail": "94032'110597661",
"last_user_version": 110601788,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94040,
"last_interval_started": 94039,
"last_epoch_clean": 93891,
"last_interval_clean": 93890,
"last_epoch_split": 5348,
"last_epoch_marked_full": 0,
"same_up_since": 94062,
"same_interval_since": 94062,
"same_primary_since": 94012,
"last_scrub": "93478'109774363",
"last_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_deep_scrub": "93478'109774363",
"last_deep_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_clean_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 94062,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
23,
43
],
"acting": [
18
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 23,
"acting_primary": 18,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 94040,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "43",
"pgid": "13.bb",
"last_update": "94261'110689291",
"last_complete": "94261'110689291",
"log_tail": "94032'110597661",
"last_user_version": 110601788,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94063,
"last_interval_started": 94062,
"last_epoch_clean": 93891,
"last_interval_clean": 93890,
"last_epoch_split": 5348,
"last_epoch_marked_full": 0,
"same_up_since": 94062,
"same_interval_since": 94062,
"same_primary_since": 94012,
"last_scrub": "93478'109774363",
"last_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_deep_scrub": "93478'109774363",
"last_deep_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"last_clean_scrub_stamp": "2025-11-22T15:59:45.780502+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 94062,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3501,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
23,
43
],
"acting": [
18
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 23,
"acting_primary": 18,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 94063,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2025-12-01T10:31:49.688727+0100",
"might_have_unfound": [
{
"osd": "4",
"status": "not queried"
},
{
"osd": "23",
"status": "already probed"
},
{
"osd": "40",
"status": "already probed"
},
{
"osd": "43",
"status": "already probed"
}
],
"recovery_progress": {
"backfill_targets": [
"23",
"43"
],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"pull_from_peer": [],
"pushing": []
}
}
},
{
"name": "Started",
"enter_time": "2025-12-01T10:31:48.671415+0100"
}
],
"scrubber": {
"active": false,
"must_scrub": true,
"must_deep_scrub": true,
"must_repair": false,
"need_auto": false,
"scrub_reg_stamp": "1.000000",
"schedule": "queued for deep scrub"
},
"agent_state": {}
}
ceph pg 13.4f query
{
"snap_trimq": "[]",
"snap_trimq_len": 0,
"state": "active+forced_recovery+recovery_unfound+undersized+degraded+remapped",
"epoch": 94261,
"up": [
44,
21
],
"acting": [
44
],
"backfill_targets": [
"21"
],
"acting_recovery_backfill": [
"21",
"44"
],
"info": {
"pgid": "13.4f",
"last_update": "94261'113030372",
"last_complete": "0'0",
"log_tail": "94246'113025957",
"last_user_version": 113030372,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94064,
"last_interval_started": 94063,
"last_epoch_clean": 93896,
"last_interval_clean": 93895,
"last_epoch_split": 5705,
"last_epoch_marked_full": 0,
"same_up_since": 94063,
"same_interval_since": 94063,
"same_primary_since": 94003,
"last_scrub": "93505'112081441",
"last_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"last_deep_scrub": "90712'110192662",
"last_deep_scrub_stamp": "2025-11-08T08:43:39.071919+0100",
"last_clean_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "94261'113030372",
"reported_seq": 143656837,
"reported_epoch": 94261,
"state": "active+forced_recovery+recovery_unfound+undersized+degraded+remapped",
"last_fresh": "2025-12-02T10:44:01.454051+0100",
"last_change": "2025-12-01T15:23:58.535574+0100",
"last_active": "2025-12-02T10:44:01.454051+0100",
"last_peered": "2025-12-02T10:44:01.454051+0100",
"last_clean": "2025-12-01T09:44:33.180830+0100",
"last_became_active": "2025-12-01T10:31:50.661861+0100",
"last_became_peered": "2025-12-01T10:31:50.661861+0100",
"last_unstale": "2025-12-02T10:44:01.454051+0100",
"last_undegraded": "2025-12-01T10:31:55.591735+0100",
"last_fullsized": "2025-12-01T10:31:50.305914+0100",
"mapping_epoch": 94063,
"log_start": "94246'113025957",
"ondisk_log_start": "94246'113025957",
"created": 5348,
"last_epoch_clean": 93896,
"parent": "0.0",
"parent_split_bits": 7,
"last_scrub": "93505'112081441",
"last_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"last_deep_scrub": "90712'110192662",
"last_deep_scrub_stamp": "2025-11-08T08:43:39.071919+0100",
"last_clean_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"objects_scrubbed": 3348,
"log_size": 4415,
"log_dups_size": 0,
"ondisk_log_size": 4415,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 41,
"scrub_schedule": "queued for deep scrub",
"scrub_duration": 40.966387804999997,
"objects_trimmed": 0,
"snaptrim_duration": 0.00073771999999999998,
"stat_sum": {
"num_bytes": 14026813440,
"num_objects": 3348,
"num_object_clones": 0,
"num_object_copies": 6696,
"num_objects_missing_on_primary": 1,
"num_objects_missing": 1,
"num_objects_degraded": 3349,
"num_objects_misplaced": 0,
"num_objects_unfound": 1,
"num_objects_dirty": 3348,
"num_whiteouts": 0,
"num_read": 30491476,
"num_read_kb": 1172273573,
"num_write": 112900649,
"num_write_kb": 2009879236,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 1659,
"num_bytes_recovered": 4888453120,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
44,
21
],
"acting": [
44
],
"avail_no_missing": [
"4"
],
"object_location_counts": [
{
"shards": "44",
"objects": 3348
}
],
"blocked_by": [],
"up_primary": 44,
"acting_primary": 44,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 0,
"last_epoch_started": 94064,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
"peer_info": [
{
"peer": "4",
"pgid": "13.4f",
"last_update": "94061'112920022",
"last_complete": "94061'112920022",
"log_tail": "94032'112916557",
"last_user_version": 112920022,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94041,
"last_interval_started": 94040,
"last_epoch_clean": 93896,
"last_interval_clean": 93895,
"last_epoch_split": 5705,
"last_epoch_marked_full": 0,
"same_up_since": 94063,
"same_interval_since": 94063,
"same_primary_since": 94003,
"last_scrub": "93505'112081441",
"last_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"last_deep_scrub": "90712'110192662",
"last_deep_scrub_stamp": "2025-11-08T08:43:39.071919+0100",
"last_clean_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 94063,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 0,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
44,
21
],
"acting": [
44
],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": 44,
"acting_primary": 44,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 94041,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "21",
"pgid": "13.4f",
"last_update": "94261'113030372",
"last_complete": "94261'113030372",
"log_tail": "94032'112916557",
"last_user_version": 112902886,
"last_backfill": "MIN",
"purged_snaps": [],
"history": {
"epoch_created": 5348,
"epoch_pool_created": 5339,
"last_epoch_started": 94064,
"last_interval_started": 94063,
"last_epoch_clean": 93896,
"last_interval_clean": 93895,
"last_epoch_split": 5705,
"last_epoch_marked_full": 0,
"same_up_since": 94063,
"same_interval_since": 94063,
"same_primary_since": 94003,
"last_scrub": "93505'112081441",
"last_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"last_deep_scrub": "90712'110192662",
"last_deep_scrub_stamp": "2025-11-08T08:43:39.071919+0100",
"last_clean_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"prior_readable_until_ub": 0
},
"stats": {
"version": "94011'112902885",
"reported_seq": 143427186,
"reported_epoch": 94011,
"state": "active+recovery_wait+degraded",
"last_fresh": "2025-12-01T10:15:02.135139+0100",
"last_change": "2025-12-01T10:14:54.396548+0100",
"last_active": "2025-12-01T10:15:02.135139+0100",
"last_peered": "2025-12-01T10:15:02.135139+0100",
"last_clean": "2025-12-01T09:44:33.180830+0100",
"last_became_active": "2025-12-01T10:14:54.396460+0100",
"last_became_peered": "2025-12-01T10:14:54.396460+0100",
"last_unstale": "2025-12-01T10:15:02.135139+0100",
"last_undegraded": "2025-12-01T10:14:54.395277+0100",
"last_fullsized": "2025-12-01T10:15:02.135139+0100",
"mapping_epoch": 94063,
"log_start": "93716'112899560",
"ondisk_log_start": "93716'112899560",
"created": 5348,
"last_epoch_clean": 93896,
"parent": "0.0",
"parent_split_bits": 7,
"last_scrub": "93505'112081441",
"last_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"last_deep_scrub": "90712'110192662",
"last_deep_scrub_stamp": "2025-11-08T08:43:39.071919+0100",
"last_clean_scrub_stamp": "2025-11-24T06:28:26.200384+0100",
"objects_scrubbed": 3348,
"log_size": 3325,
"log_dups_size": 0,
"ondisk_log_size": 3325,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 41,
"scrub_schedule": "periodic scrub scheduled @ 2025-12-02T13:14:50.648568+0000",
"scrub_duration": 40.966387804999997,
"objects_trimmed": 0,
"snaptrim_duration": 0.00073771999999999998,
"stat_sum": {
"num_bytes": 14028208640,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3348,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [
44,
21
],
"acting": [
44
],
"avail_no_missing": [
"4",
"21"
],
"object_location_counts": [
{
"shards": "21",
"objects": 1
},
{
"shards": "21,44",
"objects": 3347
}
],
"blocked_by": [],
"up_primary": 44,
"acting_primary": 44,
"purged_snaps": []
},
"empty": 0,
"dne": 0,
"incomplete": 1,
"last_epoch_started": 94064,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "39",
"pgid": "13.4f",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3348,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "42",
"pgid": "13.4f",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3348,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
},
{
"peer": "43",
"pgid": "13.4f",
"last_update": "0'0",
"last_complete": "0'0",
"log_tail": "0'0",
"last_user_version": 0,
"last_backfill": "MAX",
"purged_snaps": [],
"history": {
"epoch_created": 0,
"epoch_pool_created": 0,
"last_epoch_started": 0,
"last_interval_started": 0,
"last_epoch_clean": 0,
"last_interval_clean": 0,
"last_epoch_split": 0,
"last_epoch_marked_full": 0,
"same_up_since": 0,
"same_interval_since": 0,
"same_primary_since": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"prior_readable_until_ub": 0
},
"stats": {
"version": "0'0",
"reported_seq": 0,
"reported_epoch": 0,
"state": "unknown",
"last_fresh": "0.000000",
"last_change": "0.000000",
"last_active": "0.000000",
"last_peered": "0.000000",
"last_clean": "0.000000",
"last_became_active": "0.000000",
"last_became_peered": "0.000000",
"last_unstale": "0.000000",
"last_undegraded": "0.000000",
"last_fullsized": "0.000000",
"mapping_epoch": 0,
"log_start": "0'0",
"ondisk_log_start": "0'0",
"created": 0,
"last_epoch_clean": 0,
"parent": "0.0",
"parent_split_bits": 0,
"last_scrub": "0'0",
"last_scrub_stamp": "0.000000",
"last_deep_scrub": "0'0",
"last_deep_scrub_stamp": "0.000000",
"last_clean_scrub_stamp": "0.000000",
"objects_scrubbed": 0,
"log_size": 0,
"log_dups_size": 0,
"ondisk_log_size": 0,
"stats_invalid": false,
"dirty_stats_invalid": false,
"omap_stats_invalid": false,
"hitset_stats_invalid": false,
"hitset_bytes_stats_invalid": false,
"pin_stats_invalid": false,
"manifest_stats_invalid": false,
"snaptrimq_len": 0,
"last_scrub_duration": 0,
"scrub_schedule": "--",
"scrub_duration": 0,
"objects_trimmed": 0,
"snaptrim_duration": 0,
"stat_sum": {
"num_bytes": 0,
"num_objects": 0,
"num_object_clones": 0,
"num_object_copies": 0,
"num_objects_missing_on_primary": 0,
"num_objects_missing": 3348,
"num_objects_degraded": 0,
"num_objects_misplaced": 0,
"num_objects_unfound": 0,
"num_objects_dirty": 0,
"num_whiteouts": 0,
"num_read": 0,
"num_read_kb": 0,
"num_write": 0,
"num_write_kb": 0,
"num_scrub_errors": 0,
"num_shallow_scrub_errors": 0,
"num_deep_scrub_errors": 0,
"num_objects_recovered": 0,
"num_bytes_recovered": 0,
"num_keys_recovered": 0,
"num_objects_omap": 0,
"num_objects_hit_set_archive": 0,
"num_bytes_hit_set_archive": 0,
"num_flush": 0,
"num_flush_kb": 0,
"num_evict": 0,
"num_evict_kb": 0,
"num_promote": 0,
"num_flush_mode_high": 0,
"num_flush_mode_low": 0,
"num_evict_mode_some": 0,
"num_evict_mode_full": 0,
"num_objects_pinned": 0,
"num_legacy_snapsets": 0,
"num_large_omap_objects": 0,
"num_objects_manifest": 0,
"num_omap_bytes": 0,
"num_omap_keys": 0,
"num_objects_repaired": 0
},
"up": [],
"acting": [],
"avail_no_missing": [],
"object_location_counts": [],
"blocked_by": [],
"up_primary": -1,
"acting_primary": -1,
"purged_snaps": []
},
"empty": 1,
"dne": 1,
"incomplete": 0,
"last_epoch_started": 0,
"hit_set_history": {
"current_last_update": "0'0",
"history": []
}
}
],
"recovery_state": [
{
"name": "Started/Primary/Active",
"enter_time": "2025-12-01T10:31:50.305946+0100",
"might_have_unfound": [
{
"osd": "4",
"status": "already probed"
},
{
"osd": "21",
"status": "already probed"
},
{
"osd": "39",
"status": "not queried"
},
{
"osd": "42",
"status": "not queried"
},
{
"osd": "43",
"status": "not queried"
}
],
"recovery_progress": {
"backfill_targets": [
"21"
],
"waiting_on_backfill": [],
"last_backfill_started": "MIN",
"backfill_info": {
"begin": "MIN",
"end": "MIN",
"objects": []
},
"peer_backfill_info": [],
"backfills_in_flight": [],
"recovering": [],
"pg_backend": {
"pull_from_peer": [],
"pushing": []
}
}
},
{
"name": "Started",
"enter_time": "2025-12-01T10:31:49.691348+0100"
}
],
"scrubber": {
"active": false,
"must_scrub": true,
"must_deep_scrub": true,
"must_repair": true,
"need_auto": false,
"scrub_reg_stamp": "1.000000",
"schedule": "queued for deep scrub"
},
"agent_state": {}
}

 

What is the fastest way to get the cluster healthy again.

Regards

Raf

 

 

 

#1

admin
3,054 Posts
December 2, 2025, 1:33 pm

You have 6 objects out of 25M that are not found.

You can first know the affected objects ( can related to rbd image sector or file )
ceph pg 13.5d list_unfound

try to revert to older version
ceph pg 13.5d mark_unfound_lost revert

else delete objects
ceph pg 13.5d mark_unfound_lost delete

similarly with other 3 pgs.

unfound objects can occur in power loss could be:
using 2x instead of 3x replication
media issues: non enterprise grade SSDs without PLP
disk controller cache without battery protection or controller firmware

#2

raf.schandevyl@outlook.be
3 Posts
December 2, 2025, 3:05 pm

We still have osd's not queried.

Do i need to wait till these ar queried ?  can force the query ?

regards

Raf

 

ceph pg 13.5d list_unfound
{
"num_missing": 2,
"num_unfound": 2,
"objects": [
{
"oid": {
"oid": "rbd_data.2129e811798f26.0000000000001b22",
"key": "",
"snapid": -2,
"hash": 2045912669,
"max": 0,
"pool": 13,
"namespace": ""
},
"need": "93984'110766871",
"have": "93924'110766840",
"flags": "none",
"clean_regions": "clean_offsets: [0~237568,241664~18446744073709309951], clean_omap: 1, new_object: 0",
"locations": []
},
{
"oid": {
"oid": "rbd_data.2129e811798f26.0000000000001c89",
"key": "",
"snapid": -2,
"hash": 2441173853,
"max": 0,
"pool": 13,
"namespace": ""
},
"need": "93984'110766870",
"have": "93795'110765723",
"flags": "none",
"clean_regions": "clean_offsets: [0~720896,1769472~18446744073707782143], clean_omap: 1, new_object: 0",
"locations": []
}
],
"state": "NotRecovering",
"available_might_have_unfound": true,
"might_have_unfound": [
{
"osd": "3",
"status": "not queried"
},
{
"osd": "12",
"status": "not queried"
}
],
"more": false
}

#3

admin
3,054 Posts
December 2, 2025, 4:02 pm

You can try to force a repeer
ceph pg repeer 13.5d
or for all pgs with unfound:
ceph pg ls recovery_unfound | cut -d ' ' -f1 | while read i; do echo $i; ceph pg repeer $i; done

Else see prev post.

#4

raf.schandevyl@outlook.be
3 Posts
December 3, 2025, 10:59 am

Hi,

Thanks for the help.  cluster is back healthy.  we are now removing our 2 way replicated pools.

Regards

Raf

#5

admin
3,054 Posts
December 3, 2025, 11:16 am

Very good.

Note that you can just increase the replicas to 3 on existing replica 2 pools via the UI, if you wish to keep them. You should also check the other factors listed earlier on SSD type and controllers.

Also to be more precise, it is not the x2 replica than can cause this, rather the min_size=1 is the root issue, which is typical setting when x2 replica is used. You can keep x2 replica and set min_size=2 and this will avoid such issues, but you will not have high availability.

Last edited on December 3, 2025, 11:24 am by admin · #6