Strong file descriptor leakage in petasan-iscsi and petasan-admin
dbutti
42 Posts
September 17, 2026, 1:43 pmQuote from dbutti on September 17, 2026, 1:43 pmHello; on our 4-node Petasan cluster I noticed over the years a tendency to instability in the web interface, so I started a systematic investigation on the running nodes, cross-referencing the observations with Petasan's source code. It turned out not only the admin process, but also the iSCSI admin process leaks file descriptors heavily, which affects response times and leads to periodic kill/restart of the process(es). Here I prepared a detailed description of the situation, it would be nice to have it fixed in upcoming upgrades:
PetaSAN 4.1.0 (Ceph 18.2.7, Ubuntu 22.04) leaks a librados client on every disk
enumeration that touches an erasure-coded RBD pool. In
PetaSAN/core/ceph/rbd_image_metadata.py, iSCSIMetadata.get_disks_metadata() reads the
omap object "petasan_metadata" from each pool returned by
CephAPI.get_active_rbd_pool_names(). An EC pool used as an RBD data pool carries the
"rbd" application tag, so it is included in that list, but the object cannot exist
there; the error handler therefore calls sync_disks_metadata() to rebuild it. That
function opens its connection at lines 424-426, outside its try block, and closes it at
lines 479-480, inside it. Its write of "petasan_metadata_backup" fails with errno 95
(EOPNOTSUPP), because an EC pool cannot store omap at all, so the except block logs the
error and does "return False" without calling ioctx.close() or cluster.shutdown().
rados.Rados does not define __del__ on this build, so the garbage collector never
reclaims the object, and roughly 14 file descriptors and 12 threads stay pinned for the
life of the process. The condition can never self-heal, because the object sync is
trying to create is one an EC pool is incapable of holding, so every subsequent
enumeration repeats it. Our log held 5128 such failures, 388 of them in a single hour.
Both long-lived services are affected. petasan-admin reaches the leak through
CephAPI.get_disks_meta() on the iSCSI Disks page, which list.js polls every ten seconds;
on our cluster, which has two EC pools carrying the rbd tag, the admin process reached
10240 of 10240 file descriptors and 8775 threads and stopped answering HTTP altogether,
with the thread counts exact multiples of 731 leaked clients. petasan-iscsi reaches it
through mange_path_assignment.py line 92 and reached 804 of 1024 descriptors after 354
hours of uptime, on a node with no management role. It reproduces in a few lines:
calling iSCSIMetadata(pool="<ec_pool>").get_disks_metadata() five times takes a Python
process from 5 file descriptors and 1 thread to 75 and 61. Critically, no exception is
ever raised, so no caller can catch it and nothing appears in the logs except the
underlying errno 95. The same shape, a connection opened before the try block but
released only inside it, is also present in RBDImageMetadata.set_disk_metadata (line 76)
and RBDImageMetadata.remove_disk_metadata (line 165); the whole file contains only three
shutdown() calls, all on success paths. The fix is to release the ioctx and the cluster
in a finally block in each of these functions, and it would be worth also excluding
pools that cannot hold an omap object from the metadata scan in the first place.
Please get back to me if you need additional information.
Thank-you and have a nice day,
Hello; on our 4-node Petasan cluster I noticed over the years a tendency to instability in the web interface, so I started a systematic investigation on the running nodes, cross-referencing the observations with Petasan's source code. It turned out not only the admin process, but also the iSCSI admin process leaks file descriptors heavily, which affects response times and leads to periodic kill/restart of the process(es). Here I prepared a detailed description of the situation, it would be nice to have it fixed in upcoming upgrades:
PetaSAN 4.1.0 (Ceph 18.2.7, Ubuntu 22.04) leaks a librados client on every disk
enumeration that touches an erasure-coded RBD pool. In
PetaSAN/core/ceph/rbd_image_metadata.py, iSCSIMetadata.get_disks_metadata() reads the
omap object "petasan_metadata" from each pool returned by
CephAPI.get_active_rbd_pool_names(). An EC pool used as an RBD data pool carries the
"rbd" application tag, so it is included in that list, but the object cannot exist
there; the error handler therefore calls sync_disks_metadata() to rebuild it. That
function opens its connection at lines 424-426, outside its try block, and closes it at
lines 479-480, inside it. Its write of "petasan_metadata_backup" fails with errno 95
(EOPNOTSUPP), because an EC pool cannot store omap at all, so the except block logs the
error and does "return False" without calling ioctx.close() or cluster.shutdown().
rados.Rados does not define __del__ on this build, so the garbage collector never
reclaims the object, and roughly 14 file descriptors and 12 threads stay pinned for the
life of the process. The condition can never self-heal, because the object sync is
trying to create is one an EC pool is incapable of holding, so every subsequent
enumeration repeats it. Our log held 5128 such failures, 388 of them in a single hour.
Both long-lived services are affected. petasan-admin reaches the leak through
CephAPI.get_disks_meta() on the iSCSI Disks page, which list.js polls every ten seconds;
on our cluster, which has two EC pools carrying the rbd tag, the admin process reached
10240 of 10240 file descriptors and 8775 threads and stopped answering HTTP altogether,
with the thread counts exact multiples of 731 leaked clients. petasan-iscsi reaches it
through mange_path_assignment.py line 92 and reached 804 of 1024 descriptors after 354
hours of uptime, on a node with no management role. It reproduces in a few lines:
calling iSCSIMetadata(pool="<ec_pool>").get_disks_metadata() five times takes a Python
process from 5 file descriptors and 1 thread to 75 and 61. Critically, no exception is
ever raised, so no caller can catch it and nothing appears in the logs except the
underlying errno 95. The same shape, a connection opened before the try block but
released only inside it, is also present in RBDImageMetadata.set_disk_metadata (line 76)
and RBDImageMetadata.remove_disk_metadata (line 165); the whole file contains only three
shutdown() calls, all on success paths. The fix is to release the ioctx and the cluster
in a finally block in each of these functions, and it would be worth also excluding
pools that cannot hold an omap object from the metadata scan in the first place.
Please get back to me if you need additional information.
Thank-you and have a nice day,
admin
3,088 Posts
September 17, 2026, 7:18 pmQuote from admin on September 17, 2026, 7:18 pmThank you so much for the detailed info/
Thank you so much for the detailed info/
Strong file descriptor leakage in petasan-iscsi and petasan-admin
dbutti
42 Posts
Quote from dbutti on September 17, 2026, 1:43 pmHello; on our 4-node Petasan cluster I noticed over the years a tendency to instability in the web interface, so I started a systematic investigation on the running nodes, cross-referencing the observations with Petasan's source code. It turned out not only the admin process, but also the iSCSI admin process leaks file descriptors heavily, which affects response times and leads to periodic kill/restart of the process(es). Here I prepared a detailed description of the situation, it would be nice to have it fixed in upcoming upgrades:
PetaSAN 4.1.0 (Ceph 18.2.7, Ubuntu 22.04) leaks a librados client on every disk
enumeration that touches an erasure-coded RBD pool. In
PetaSAN/core/ceph/rbd_image_metadata.py, iSCSIMetadata.get_disks_metadata() reads the
omap object "petasan_metadata" from each pool returned by
CephAPI.get_active_rbd_pool_names(). An EC pool used as an RBD data pool carries the
"rbd" application tag, so it is included in that list, but the object cannot exist
there; the error handler therefore calls sync_disks_metadata() to rebuild it. That
function opens its connection at lines 424-426, outside its try block, and closes it at
lines 479-480, inside it. Its write of "petasan_metadata_backup" fails with errno 95
(EOPNOTSUPP), because an EC pool cannot store omap at all, so the except block logs the
error and does "return False" without calling ioctx.close() or cluster.shutdown().
rados.Rados does not define __del__ on this build, so the garbage collector never
reclaims the object, and roughly 14 file descriptors and 12 threads stay pinned for the
life of the process. The condition can never self-heal, because the object sync is
trying to create is one an EC pool is incapable of holding, so every subsequent
enumeration repeats it. Our log held 5128 such failures, 388 of them in a single hour.Both long-lived services are affected. petasan-admin reaches the leak through
CephAPI.get_disks_meta() on the iSCSI Disks page, which list.js polls every ten seconds;
on our cluster, which has two EC pools carrying the rbd tag, the admin process reached
10240 of 10240 file descriptors and 8775 threads and stopped answering HTTP altogether,
with the thread counts exact multiples of 731 leaked clients. petasan-iscsi reaches it
through mange_path_assignment.py line 92 and reached 804 of 1024 descriptors after 354
hours of uptime, on a node with no management role. It reproduces in a few lines:
calling iSCSIMetadata(pool="<ec_pool>").get_disks_metadata() five times takes a Python
process from 5 file descriptors and 1 thread to 75 and 61. Critically, no exception is
ever raised, so no caller can catch it and nothing appears in the logs except the
underlying errno 95. The same shape, a connection opened before the try block but
released only inside it, is also present in RBDImageMetadata.set_disk_metadata (line 76)
and RBDImageMetadata.remove_disk_metadata (line 165); the whole file contains only three
shutdown() calls, all on success paths. The fix is to release the ioctx and the cluster
in a finally block in each of these functions, and it would be worth also excluding
pools that cannot hold an omap object from the metadata scan in the first place.Please get back to me if you need additional information.
Thank-you and have a nice day,
Hello; on our 4-node Petasan cluster I noticed over the years a tendency to instability in the web interface, so I started a systematic investigation on the running nodes, cross-referencing the observations with Petasan's source code. It turned out not only the admin process, but also the iSCSI admin process leaks file descriptors heavily, which affects response times and leads to periodic kill/restart of the process(es). Here I prepared a detailed description of the situation, it would be nice to have it fixed in upcoming upgrades:
PetaSAN 4.1.0 (Ceph 18.2.7, Ubuntu 22.04) leaks a librados client on every disk
enumeration that touches an erasure-coded RBD pool. In
PetaSAN/core/ceph/rbd_image_metadata.py, iSCSIMetadata.get_disks_metadata() reads the
omap object "petasan_metadata" from each pool returned by
CephAPI.get_active_rbd_pool_names(). An EC pool used as an RBD data pool carries the
"rbd" application tag, so it is included in that list, but the object cannot exist
there; the error handler therefore calls sync_disks_metadata() to rebuild it. That
function opens its connection at lines 424-426, outside its try block, and closes it at
lines 479-480, inside it. Its write of "petasan_metadata_backup" fails with errno 95
(EOPNOTSUPP), because an EC pool cannot store omap at all, so the except block logs the
error and does "return False" without calling ioctx.close() or cluster.shutdown().
rados.Rados does not define __del__ on this build, so the garbage collector never
reclaims the object, and roughly 14 file descriptors and 12 threads stay pinned for the
life of the process. The condition can never self-heal, because the object sync is
trying to create is one an EC pool is incapable of holding, so every subsequent
enumeration repeats it. Our log held 5128 such failures, 388 of them in a single hour.Both long-lived services are affected. petasan-admin reaches the leak through
CephAPI.get_disks_meta() on the iSCSI Disks page, which list.js polls every ten seconds;
on our cluster, which has two EC pools carrying the rbd tag, the admin process reached
10240 of 10240 file descriptors and 8775 threads and stopped answering HTTP altogether,
with the thread counts exact multiples of 731 leaked clients. petasan-iscsi reaches it
through mange_path_assignment.py line 92 and reached 804 of 1024 descriptors after 354
hours of uptime, on a node with no management role. It reproduces in a few lines:
calling iSCSIMetadata(pool="<ec_pool>").get_disks_metadata() five times takes a Python
process from 5 file descriptors and 1 thread to 75 and 61. Critically, no exception is
ever raised, so no caller can catch it and nothing appears in the logs except the
underlying errno 95. The same shape, a connection opened before the try block but
released only inside it, is also present in RBDImageMetadata.set_disk_metadata (line 76)
and RBDImageMetadata.remove_disk_metadata (line 165); the whole file contains only three
shutdown() calls, all on success paths. The fix is to release the ioctx and the cluster
in a finally block in each of these functions, and it would be worth also excluding
pools that cannot hold an omap object from the metadata scan in the first place.
Please get back to me if you need additional information.
Thank-you and have a nice day,
admin
3,088 Posts
Quote from admin on September 17, 2026, 7:18 pmThank you so much for the detailed info/
Thank you so much for the detailed info/