NFS server broken
Ste
151 Posts
August 25, 2025, 9:22 amQuote from Ste on August 25, 2025, 9:22 amGood morning,
today i found this issue in one PetaSAN cluster, 4 nodes, version 3.3.0. In the last node (#4) the NFS server fails to start, in the log there's this output:
KeyError: 'Id'
image_obj.id = image['Id']
File "/usr/lib/python3/dist-packages/PetaSAN/core/containers/container_manager.py", line 102, in get_images_list
podman_images = self.get_images_list()
File "/usr/lib/python3/dist-packages/PetaSAN/core/containers/container_manager.py", line 478, in clean_all_image_versions
container_mgr.clean_all_image_versions(self.IMAGE_NAME)
File "/usr/lib/python3/dist-packages/PetaSAN/core/nfs/nfs_server.py", line 505, in init_action
if not self.init_action():
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/lock_base.py", line 157, in start
service.start()
File "/opt/petasan/services/nfs_server.py", line 59, in start_nfs_service
Traceback (most recent call last):
25/08/2025 11:02:14 ERROR 'Id'
25/08/2025 11:02:14 ERROR Error start_nfs_service , Exception : 'Id'
25/08/2025 11:02:14 INFO Container Manager : deleting all old containers
25/08/2025 11:02:11 ERROR rm: cannot remove '/var/tmp/docker-tar*': No such file or directory
25/08/2025 11:02:11 ERROR NFSServer : Error running cmd : rm /var/tmp/docker-tar*
25/08/2025 11:02:11 INFO Starting NFSServer Service.
I didn't change anything, all nodes and OSDs are green, only a warning that some pgs are scrubbing. What is happened and why ? How can i fix it ? Are os updates running in background ?
Thanks. Ste
Good morning,
today i found this issue in one PetaSAN cluster, 4 nodes, version 3.3.0. In the last node (#4) the NFS server fails to start, in the log there's this output:
KeyError: 'Id'
image_obj.id = image['Id']
File "/usr/lib/python3/dist-packages/PetaSAN/core/containers/container_manager.py", line 102, in get_images_list
podman_images = self.get_images_list()
File "/usr/lib/python3/dist-packages/PetaSAN/core/containers/container_manager.py", line 478, in clean_all_image_versions
container_mgr.clean_all_image_versions(self.IMAGE_NAME)
File "/usr/lib/python3/dist-packages/PetaSAN/core/nfs/nfs_server.py", line 505, in init_action
if not self.init_action():
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/lock_base.py", line 157, in start
service.start()
File "/opt/petasan/services/nfs_server.py", line 59, in start_nfs_service
Traceback (most recent call last):
25/08/2025 11:02:14 ERROR 'Id'
25/08/2025 11:02:14 ERROR Error start_nfs_service , Exception : 'Id'
25/08/2025 11:02:14 INFO Container Manager : deleting all old containers
25/08/2025 11:02:11 ERROR rm: cannot remove '/var/tmp/docker-tar*': No such file or directory
25/08/2025 11:02:11 ERROR NFSServer : Error running cmd : rm /var/tmp/docker-tar*
25/08/2025 11:02:11 INFO Starting NFSServer Service.
I didn't change anything, all nodes and OSDs are green, only a warning that some pgs are scrubbing. What is happened and why ? How can i fix it ? Are os updates running in background ?
Thanks. Ste
Last edited on August 25, 2025, 9:27 am by Ste · #1
admin
3,054 Posts
August 25, 2025, 4:31 pmQuote from admin on August 25, 2025, 4:31 pmis this happening only on 1 server or all ?
is the log repeating or only once ?
is this happening only on 1 server or all ?
is the log repeating or only once ?
Ste
151 Posts
August 25, 2025, 5:31 pmQuote from Ste on August 25, 2025, 5:31 pmIs happening only on node #4 because only that node has enabled the NFS server. and in the log that group of lines is repeating, i guessed because it tries to restart the service at fixed interval of time.
Is happening only on node #4 because only that node has enabled the NFS server. and in the log that group of lines is repeating, i guessed because it tries to restart the service at fixed interval of time.
Last edited on August 25, 2025, 5:31 pm by Ste · #3
admin
3,054 Posts
August 26, 2025, 12:12 pmQuote from admin on August 26, 2025, 12:12 pmCan you show output of
dpkg -l | grep podman
podman images --format json
Can you show output of
dpkg -l | grep podman
podman images --format json
Ste
151 Posts
August 26, 2025, 3:14 pmQuote from Ste on August 26, 2025, 3:14 pmYes, here it is:
root@petasan04:~# dpkg -l | grep podman
ii podman 1.6.2-1~ubuntu18.04~ppa1 amd64 Manage pods, containers and container images.
root@petasan04:~#
root@petasan04:~# podman images --format json
[
{
"id": "45787403721a3a2ae9b3bee76083e346ce5e1fbf1af30d8397d88fef556a8424",
"names": [
"localhost/petasan-nfs-ganesha:3.2.0"
],
"digest": "sha256:d15fb141e39bfc1d261d6927d1007e2c1dfb267c1220cd3eb7b60739aaaee075",
"created": "2021-06-21T10:57:54.895068937Z",
"size": 105536086,
"readonly": false
}
]
Yes, here it is:
root@petasan04:~# dpkg -l | grep podman
ii podman 1.6.2-1~ubuntu18.04~ppa1 amd64 Manage pods, containers and container images.
root@petasan04:~#
root@petasan04:~# podman images --format json
[
{
"id": "45787403721a3a2ae9b3bee76083e346ce5e1fbf1af30d8397d88fef556a8424",
"names": [
"localhost/petasan-nfs-ganesha:3.2.0"
],
"digest": "sha256:d15fb141e39bfc1d261d6927d1007e2c1dfb267c1220cd3eb7b60739aaaee075",
"created": "2021-06-21T10:57:54.895068937Z",
"size": 105536086,
"readonly": false
}
]
admin
3,054 Posts
August 26, 2025, 5:46 pmQuote from admin on August 26, 2025, 5:46 pmit seems podman is not updated to correct version,
can you run
apt-cache policy podman
apt install podman
it seems podman is not updated to correct version,
can you run
apt-cache policy podman
apt install podman
Ste
151 Posts
August 27, 2025, 6:38 amQuote from Ste on August 27, 2025, 6:38 amIt worked ! Thanks. At first the NFS volume was still down (a problem with IP assignment), but after a service restart everything went green.
But why it suddenly needed this update ? I didn't change anything, i even was on holyday... 😀 Does it perform regular OS update in background ? I thought i only had to issue it manually.
Thanks and bye, S.
It worked ! Thanks. At first the NFS volume was still down (a problem with IP assignment), but after a service restart everything went green.
But why it suddenly needed this update ? I didn't change anything, i even was on holyday... 😀 Does it perform regular OS update in background ? I thought i only had to issue it manually.
Thanks and bye, S.
admin
3,054 Posts
August 27, 2025, 8:10 pmQuote from admin on August 27, 2025, 8:10 pmGood it is fixed.
No automatic updates done, you have to initiate the update. The podman update issue could happen if you started your cluster from a long time prior to v 2.6 but you can resolve it by explicitly updating it as you did.
Good it is fixed.
No automatic updates done, you have to initiate the update. The podman update issue could happen if you started your cluster from a long time prior to v 2.6 but you can resolve it by explicitly updating it as you did.
Ste
151 Posts
August 28, 2025, 4:28 pmQuote from Ste on August 28, 2025, 4:28 pmI was too quick 🙁 after seeing in the GUI the service coming up, but actually there's a problem with podman. Here is the log:
28/08/2025 18:21:21 ERROR NFSExportsServer : init_action() : Cannot create and run new container "NFS-10-220-12-211"
Error: error mounting storage for container 76d5d4cbe8e3c15107f4331a60eec2a8ed20ce299d4475c27d0513c4761ff069: error creating overlay mount to /var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LPSLG32S64Z7QKKBWOJDRL4YOF:/var/lib/containers/storage/overlay/l/HBRK3HU4IHW7PIMUTWHIE23ECL:/var/lib/containers/storage/overlay/l/IZPPKBMYFIKJPIQQLZ36RVCR3X,upperdir=/var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/diff,workdir=/var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/work": invalid argument
28/08/2025 18:21:21 ERROR time="2025-08-28T18:21:21+02:00" level=error msg="error unmounting /var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/merged: invalid argument"
28/08/2025 18:21:21 ERROR Container Manager : run() : Error running cmd : podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
28/08/2025 18:21:20 INFO Container Manager : creating NFS-10-220-12-211 container
28/08/2025 18:21:20 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
28/08/2025 18:21:20 ERROR Container Manager : delete() : Error running cmd : podman rm -f NFS-10-220-12-211
28/08/2025 18:21:20 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
28/08/2025 18:21:20 ERROR Container Manager : stop() : Error running cmd : podman stop NFS-10-220-12-211
28/08/2025 18:21:20 INFO Starting NFS Exports Service
28/08/2025 18:21:20 INFO NFSServer : waiting for the container NFS-10-220-12-211 to be up.
28/08/2025 18:21:20 INFO LockBase : Succeeded on acquiring the resource = NFS-10-220-12-211
28/08/2025 18:21:19 INFO LockBase : Try to acquire the resource = NFS-10-220-12-211.
28/08/2025 18:21:18 INFO NFSServer : sync Consul settings -> done
28/08/2025 18:21:18 INFO NFSServer : sync Consul settings
28/08/2025 18:21:16 INFO Clean all old local resources.
28/08/2025 18:21:16 INFO LockBase : Successfully dropped old sessions
28/08/2025 18:21:06 INFO LockBase : Dropping old sessions
28/08/2025 18:21:01 INFO Container Manager : Loading /opt/petasan/container-images/petasan-nfs-ganesha-3.2.0.tar.gz image into podman
28/08/2025 18:21:01 INFO Container Manager : deleting all old containers
28/08/2025 18:20:58 INFO sync_replication_node called on non-backup node
28/08/2025 18:20:57 INFO Starting NFSServer Service.
I tried to issue the commands in bash, with with no success. The container name seems in use, if i delete it is automatically re-created, so it is again in use:
root@petasan04:~#
root@petasan04:~# podman container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
Error: error creating container storage: the container name "NFS-10-220-12-211" is already in use by "a6b549a91c83e14072d2fdce9617494281fbe21e480a58946a94b6f595fd37a5". You have to remove that container to be able to reuse that name.: that name is already in use
root@petasan04:~#
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman rm -f NFS-10-220-12-211
65ab6274219d4cf4b9145bdb531ad1c3ce83f1e6d167586cd37a80ee0013e813
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/merged: invalid argument
Error: error mounting storage for container 3b7a86a99ceb8ca504aae30827125c6bb75db5ea8bb80d44347b09bf5f311888: error creating overlay mount to /var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LPSLG32S64Z7QKKBWOJDRL4YOF:/var/lib/containers/storage/overlay/l/HBRK3HU4IHW7PIMUTWHIE23ECL:/var/lib/containers/storage/overlay/l/IZPPKBMYFIKJPIQQLZ36RVCR3X,upperdir=/var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/diff,workdir=/var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/work": invalid argument
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
Error: error creating container storage: the container name "NFS-10-220-12-211" is already in use by "7d7a251721bc4faac28e58fa607e657bcfc39252373672432ab90d432497be9b". You have to remove that container to be able to reuse that name.: that name is already in use
How to get out of it ? I already removed and added again the NFS service from host, but does not solve the issue.
Ste
I was too quick 🙁 after seeing in the GUI the service coming up, but actually there's a problem with podman. Here is the log:
28/08/2025 18:21:21 ERROR NFSExportsServer : init_action() : Cannot create and run new container "NFS-10-220-12-211"
Error: error mounting storage for container 76d5d4cbe8e3c15107f4331a60eec2a8ed20ce299d4475c27d0513c4761ff069: error creating overlay mount to /var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LPSLG32S64Z7QKKBWOJDRL4YOF:/var/lib/containers/storage/overlay/l/HBRK3HU4IHW7PIMUTWHIE23ECL:/var/lib/containers/storage/overlay/l/IZPPKBMYFIKJPIQQLZ36RVCR3X,upperdir=/var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/diff,workdir=/var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/work": invalid argument
28/08/2025 18:21:21 ERROR time="2025-08-28T18:21:21+02:00" level=error msg="error unmounting /var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/merged: invalid argument"
28/08/2025 18:21:21 ERROR Container Manager : run() : Error running cmd : podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
28/08/2025 18:21:20 INFO Container Manager : creating NFS-10-220-12-211 container
28/08/2025 18:21:20 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
28/08/2025 18:21:20 ERROR Container Manager : delete() : Error running cmd : podman rm -f NFS-10-220-12-211
28/08/2025 18:21:20 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
28/08/2025 18:21:20 ERROR Container Manager : stop() : Error running cmd : podman stop NFS-10-220-12-211
28/08/2025 18:21:20 INFO Starting NFS Exports Service
28/08/2025 18:21:20 INFO NFSServer : waiting for the container NFS-10-220-12-211 to be up.
28/08/2025 18:21:20 INFO LockBase : Succeeded on acquiring the resource = NFS-10-220-12-211
28/08/2025 18:21:19 INFO LockBase : Try to acquire the resource = NFS-10-220-12-211.
28/08/2025 18:21:18 INFO NFSServer : sync Consul settings -> done
28/08/2025 18:21:18 INFO NFSServer : sync Consul settings
28/08/2025 18:21:16 INFO Clean all old local resources.
28/08/2025 18:21:16 INFO LockBase : Successfully dropped old sessions
28/08/2025 18:21:06 INFO LockBase : Dropping old sessions
28/08/2025 18:21:01 INFO Container Manager : Loading /opt/petasan/container-images/petasan-nfs-ganesha-3.2.0.tar.gz image into podman
28/08/2025 18:21:01 INFO Container Manager : deleting all old containers
28/08/2025 18:20:58 INFO sync_replication_node called on non-backup node
28/08/2025 18:20:57 INFO Starting NFSServer Service.
I tried to issue the commands in bash, with with no success. The container name seems in use, if i delete it is automatically re-created, so it is again in use:
root@petasan04:~#
root@petasan04:~# podman container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
Error: error creating container storage: the container name "NFS-10-220-12-211" is already in use by "a6b549a91c83e14072d2fdce9617494281fbe21e480a58946a94b6f595fd37a5". You have to remove that container to be able to reuse that name.: that name is already in use
root@petasan04:~#
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman rm -f NFS-10-220-12-211
65ab6274219d4cf4b9145bdb531ad1c3ce83f1e6d167586cd37a80ee0013e813
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/merged: invalid argument
Error: error mounting storage for container 3b7a86a99ceb8ca504aae30827125c6bb75db5ea8bb80d44347b09bf5f311888: error creating overlay mount to /var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LPSLG32S64Z7QKKBWOJDRL4YOF:/var/lib/containers/storage/overlay/l/HBRK3HU4IHW7PIMUTWHIE23ECL:/var/lib/containers/storage/overlay/l/IZPPKBMYFIKJPIQQLZ36RVCR3X,upperdir=/var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/diff,workdir=/var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/work": invalid argument
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
Error: error creating container storage: the container name "NFS-10-220-12-211" is already in use by "7d7a251721bc4faac28e58fa607e657bcfc39252373672432ab90d432497be9b". You have to remove that container to be able to reuse that name.: that name is already in use
How to get out of it ? I already removed and added again the NFS service from host, but does not solve the issue.
Ste
admin
3,054 Posts
August 28, 2025, 11:02 pmQuote from admin on August 28, 2025, 11:02 pmwhat is output of
apt-cache policy podman
apt install podman
what is output of
apt-cache policy podman
apt install podman
NFS server broken
Ste
151 Posts
Quote from Ste on August 25, 2025, 9:22 amGood morning,
today i found this issue in one PetaSAN cluster, 4 nodes, version 3.3.0. In the last node (#4) the NFS server fails to start, in the log there's this output:
KeyError: 'Id'
image_obj.id = image['Id']
File "/usr/lib/python3/dist-packages/PetaSAN/core/containers/container_manager.py", line 102, in get_images_list
podman_images = self.get_images_list()
File "/usr/lib/python3/dist-packages/PetaSAN/core/containers/container_manager.py", line 478, in clean_all_image_versions
container_mgr.clean_all_image_versions(self.IMAGE_NAME)
File "/usr/lib/python3/dist-packages/PetaSAN/core/nfs/nfs_server.py", line 505, in init_action
if not self.init_action():
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/lock_base.py", line 157, in start
service.start()
File "/opt/petasan/services/nfs_server.py", line 59, in start_nfs_service
Traceback (most recent call last):
25/08/2025 11:02:14 ERROR 'Id'
25/08/2025 11:02:14 ERROR Error start_nfs_service , Exception : 'Id'
25/08/2025 11:02:14 INFO Container Manager : deleting all old containers
25/08/2025 11:02:11 ERROR rm: cannot remove '/var/tmp/docker-tar*': No such file or directory
25/08/2025 11:02:11 ERROR NFSServer : Error running cmd : rm /var/tmp/docker-tar*
25/08/2025 11:02:11 INFO Starting NFSServer Service.
I didn't change anything, all nodes and OSDs are green, only a warning that some pgs are scrubbing. What is happened and why ? How can i fix it ? Are os updates running in background ?
Thanks. Ste
Good morning,
today i found this issue in one PetaSAN cluster, 4 nodes, version 3.3.0. In the last node (#4) the NFS server fails to start, in the log there's this output:
KeyError: 'Id'
image_obj.id = image['Id']
File "/usr/lib/python3/dist-packages/PetaSAN/core/containers/container_manager.py", line 102, in get_images_list
podman_images = self.get_images_list()
File "/usr/lib/python3/dist-packages/PetaSAN/core/containers/container_manager.py", line 478, in clean_all_image_versions
container_mgr.clean_all_image_versions(self.IMAGE_NAME)
File "/usr/lib/python3/dist-packages/PetaSAN/core/nfs/nfs_server.py", line 505, in init_action
if not self.init_action():
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/lock_base.py", line 157, in start
service.start()
File "/opt/petasan/services/nfs_server.py", line 59, in start_nfs_service
Traceback (most recent call last):
25/08/2025 11:02:14 ERROR 'Id'
25/08/2025 11:02:14 ERROR Error start_nfs_service , Exception : 'Id'
25/08/2025 11:02:14 INFO Container Manager : deleting all old containers
25/08/2025 11:02:11 ERROR rm: cannot remove '/var/tmp/docker-tar*': No such file or directory
25/08/2025 11:02:11 ERROR NFSServer : Error running cmd : rm /var/tmp/docker-tar*
25/08/2025 11:02:11 INFO Starting NFSServer Service.
I didn't change anything, all nodes and OSDs are green, only a warning that some pgs are scrubbing. What is happened and why ? How can i fix it ? Are os updates running in background ?
Thanks. Ste
admin
3,054 Posts
Quote from admin on August 25, 2025, 4:31 pmis this happening only on 1 server or all ?
is the log repeating or only once ?
is this happening only on 1 server or all ?
is the log repeating or only once ?
Ste
151 Posts
Quote from Ste on August 25, 2025, 5:31 pmIs happening only on node #4 because only that node has enabled the NFS server. and in the log that group of lines is repeating, i guessed because it tries to restart the service at fixed interval of time.
Is happening only on node #4 because only that node has enabled the NFS server. and in the log that group of lines is repeating, i guessed because it tries to restart the service at fixed interval of time.
admin
3,054 Posts
Quote from admin on August 26, 2025, 12:12 pmCan you show output of
dpkg -l | grep podman
podman images --format json
Can you show output of
dpkg -l | grep podman
podman images --format json
Ste
151 Posts
Quote from Ste on August 26, 2025, 3:14 pmYes, here it is:
root@petasan04:~# dpkg -l | grep podman
ii podman 1.6.2-1~ubuntu18.04~ppa1 amd64 Manage pods, containers and container images.
root@petasan04:~#
root@petasan04:~# podman images --format json
[
{
"id": "45787403721a3a2ae9b3bee76083e346ce5e1fbf1af30d8397d88fef556a8424",
"names": [
"localhost/petasan-nfs-ganesha:3.2.0"
],
"digest": "sha256:d15fb141e39bfc1d261d6927d1007e2c1dfb267c1220cd3eb7b60739aaaee075",
"created": "2021-06-21T10:57:54.895068937Z",
"size": 105536086,
"readonly": false
}
]
Yes, here it is:
root@petasan04:~# dpkg -l | grep podman
ii podman 1.6.2-1~ubuntu18.04~ppa1 amd64 Manage pods, containers and container images.
root@petasan04:~#
root@petasan04:~# podman images --format json
[
{
"id": "45787403721a3a2ae9b3bee76083e346ce5e1fbf1af30d8397d88fef556a8424",
"names": [
"localhost/petasan-nfs-ganesha:3.2.0"
],
"digest": "sha256:d15fb141e39bfc1d261d6927d1007e2c1dfb267c1220cd3eb7b60739aaaee075",
"created": "2021-06-21T10:57:54.895068937Z",
"size": 105536086,
"readonly": false
}
]
admin
3,054 Posts
Quote from admin on August 26, 2025, 5:46 pmit seems podman is not updated to correct version,
can you run
apt-cache policy podman
apt install podman
it seems podman is not updated to correct version,
can you run
apt-cache policy podman
apt install podman
Ste
151 Posts
Quote from Ste on August 27, 2025, 6:38 amIt worked ! Thanks. At first the NFS volume was still down (a problem with IP assignment), but after a service restart everything went green.
But why it suddenly needed this update ? I didn't change anything, i even was on holyday... 😀 Does it perform regular OS update in background ? I thought i only had to issue it manually.
Thanks and bye, S.
It worked ! Thanks. At first the NFS volume was still down (a problem with IP assignment), but after a service restart everything went green.
But why it suddenly needed this update ? I didn't change anything, i even was on holyday... 😀 Does it perform regular OS update in background ? I thought i only had to issue it manually.
Thanks and bye, S.
admin
3,054 Posts
Quote from admin on August 27, 2025, 8:10 pmGood it is fixed.
No automatic updates done, you have to initiate the update. The podman update issue could happen if you started your cluster from a long time prior to v 2.6 but you can resolve it by explicitly updating it as you did.
Good it is fixed.
No automatic updates done, you have to initiate the update. The podman update issue could happen if you started your cluster from a long time prior to v 2.6 but you can resolve it by explicitly updating it as you did.
Ste
151 Posts
Quote from Ste on August 28, 2025, 4:28 pmI was too quick 🙁 after seeing in the GUI the service coming up, but actually there's a problem with podman. Here is the log:
28/08/2025 18:21:21 ERROR NFSExportsServer : init_action() : Cannot create and run new container "NFS-10-220-12-211"
Error: error mounting storage for container 76d5d4cbe8e3c15107f4331a60eec2a8ed20ce299d4475c27d0513c4761ff069: error creating overlay mount to /var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LPSLG32S64Z7QKKBWOJDRL4YOF:/var/lib/containers/storage/overlay/l/HBRK3HU4IHW7PIMUTWHIE23ECL:/var/lib/containers/storage/overlay/l/IZPPKBMYFIKJPIQQLZ36RVCR3X,upperdir=/var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/diff,workdir=/var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/work": invalid argument
28/08/2025 18:21:21 ERROR time="2025-08-28T18:21:21+02:00" level=error msg="error unmounting /var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/merged: invalid argument"
28/08/2025 18:21:21 ERROR Container Manager : run() : Error running cmd : podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
28/08/2025 18:21:20 INFO Container Manager : creating NFS-10-220-12-211 container
28/08/2025 18:21:20 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
28/08/2025 18:21:20 ERROR Container Manager : delete() : Error running cmd : podman rm -f NFS-10-220-12-211
28/08/2025 18:21:20 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
28/08/2025 18:21:20 ERROR Container Manager : stop() : Error running cmd : podman stop NFS-10-220-12-211
28/08/2025 18:21:20 INFO Starting NFS Exports Service
28/08/2025 18:21:20 INFO NFSServer : waiting for the container NFS-10-220-12-211 to be up.
28/08/2025 18:21:20 INFO LockBase : Succeeded on acquiring the resource = NFS-10-220-12-211
28/08/2025 18:21:19 INFO LockBase : Try to acquire the resource = NFS-10-220-12-211.
28/08/2025 18:21:18 INFO NFSServer : sync Consul settings -> done
28/08/2025 18:21:18 INFO NFSServer : sync Consul settings
28/08/2025 18:21:16 INFO Clean all old local resources.
28/08/2025 18:21:16 INFO LockBase : Successfully dropped old sessions
28/08/2025 18:21:06 INFO LockBase : Dropping old sessions
28/08/2025 18:21:01 INFO Container Manager : Loading /opt/petasan/container-images/petasan-nfs-ganesha-3.2.0.tar.gz image into podman
28/08/2025 18:21:01 INFO Container Manager : deleting all old containers
28/08/2025 18:20:58 INFO sync_replication_node called on non-backup node
28/08/2025 18:20:57 INFO Starting NFSServer Service.
I tried to issue the commands in bash, with with no success. The container name seems in use, if i delete it is automatically re-created, so it is again in use:
root@petasan04:~#
root@petasan04:~# podman container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
Error: error creating container storage: the container name "NFS-10-220-12-211" is already in use by "a6b549a91c83e14072d2fdce9617494281fbe21e480a58946a94b6f595fd37a5". You have to remove that container to be able to reuse that name.: that name is already in use
root@petasan04:~#
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman rm -f NFS-10-220-12-211
65ab6274219d4cf4b9145bdb531ad1c3ce83f1e6d167586cd37a80ee0013e813
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/merged: invalid argument
Error: error mounting storage for container 3b7a86a99ceb8ca504aae30827125c6bb75db5ea8bb80d44347b09bf5f311888: error creating overlay mount to /var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LPSLG32S64Z7QKKBWOJDRL4YOF:/var/lib/containers/storage/overlay/l/HBRK3HU4IHW7PIMUTWHIE23ECL:/var/lib/containers/storage/overlay/l/IZPPKBMYFIKJPIQQLZ36RVCR3X,upperdir=/var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/diff,workdir=/var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/work": invalid argument
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
Error: error creating container storage: the container name "NFS-10-220-12-211" is already in use by "7d7a251721bc4faac28e58fa607e657bcfc39252373672432ab90d432497be9b". You have to remove that container to be able to reuse that name.: that name is already in useHow to get out of it ? I already removed and added again the NFS service from host, but does not solve the issue.
Ste
I was too quick 🙁 after seeing in the GUI the service coming up, but actually there's a problem with podman. Here is the log:
28/08/2025 18:21:21 ERROR NFSExportsServer : init_action() : Cannot create and run new container "NFS-10-220-12-211"
Error: error mounting storage for container 76d5d4cbe8e3c15107f4331a60eec2a8ed20ce299d4475c27d0513c4761ff069: error creating overlay mount to /var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LPSLG32S64Z7QKKBWOJDRL4YOF:/var/lib/containers/storage/overlay/l/HBRK3HU4IHW7PIMUTWHIE23ECL:/var/lib/containers/storage/overlay/l/IZPPKBMYFIKJPIQQLZ36RVCR3X,upperdir=/var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/diff,workdir=/var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/work": invalid argument
28/08/2025 18:21:21 ERROR time="2025-08-28T18:21:21+02:00" level=error msg="error unmounting /var/lib/containers/storage/overlay/247693c11e1142bab68a2a0f49ffb6a021c876c34d7b3249a73d8446230123f7/merged: invalid argument"
28/08/2025 18:21:21 ERROR Container Manager : run() : Error running cmd : podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
28/08/2025 18:21:20 INFO Container Manager : creating NFS-10-220-12-211 container
28/08/2025 18:21:20 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
28/08/2025 18:21:20 ERROR Container Manager : delete() : Error running cmd : podman rm -f NFS-10-220-12-211
28/08/2025 18:21:20 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
28/08/2025 18:21:20 ERROR Container Manager : stop() : Error running cmd : podman stop NFS-10-220-12-211
28/08/2025 18:21:20 INFO Starting NFS Exports Service
28/08/2025 18:21:20 INFO NFSServer : waiting for the container NFS-10-220-12-211 to be up.
28/08/2025 18:21:20 INFO LockBase : Succeeded on acquiring the resource = NFS-10-220-12-211
28/08/2025 18:21:19 INFO LockBase : Try to acquire the resource = NFS-10-220-12-211.
28/08/2025 18:21:18 INFO NFSServer : sync Consul settings -> done
28/08/2025 18:21:18 INFO NFSServer : sync Consul settings
28/08/2025 18:21:16 INFO Clean all old local resources.
28/08/2025 18:21:16 INFO LockBase : Successfully dropped old sessions
28/08/2025 18:21:06 INFO LockBase : Dropping old sessions
28/08/2025 18:21:01 INFO Container Manager : Loading /opt/petasan/container-images/petasan-nfs-ganesha-3.2.0.tar.gz image into podman
28/08/2025 18:21:01 INFO Container Manager : deleting all old containers
28/08/2025 18:20:58 INFO sync_replication_node called on non-backup node
28/08/2025 18:20:57 INFO Starting NFSServer Service.
I tried to issue the commands in bash, with with no success. The container name seems in use, if i delete it is automatically re-created, so it is again in use:
root@petasan04:~#
root@petasan04:~# podman container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
Error: error creating container storage: the container name "NFS-10-220-12-211" is already in use by "a6b549a91c83e14072d2fdce9617494281fbe21e480a58946a94b6f595fd37a5". You have to remove that container to be able to reuse that name.: that name is already in use
root@petasan04:~#
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman rm -f NFS-10-220-12-211
65ab6274219d4cf4b9145bdb531ad1c3ce83f1e6d167586cd37a80ee0013e813
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
ERRO[0000] error unmounting /var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/merged: invalid argument
Error: error mounting storage for container 3b7a86a99ceb8ca504aae30827125c6bb75db5ea8bb80d44347b09bf5f311888: error creating overlay mount to /var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LPSLG32S64Z7QKKBWOJDRL4YOF:/var/lib/containers/storage/overlay/l/HBRK3HU4IHW7PIMUTWHIE23ECL:/var/lib/containers/storage/overlay/l/IZPPKBMYFIKJPIQQLZ36RVCR3X,upperdir=/var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/diff,workdir=/var/lib/containers/storage/overlay/19abffcde68afef0ff3f975d0951b475230c53686eb2c7333c31642bdc92c0d1/work": invalid argument
root@petasan04:~#
root@petasan04:~#
root@petasan04:~# podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0
Error: error creating container storage: the container name "NFS-10-220-12-211" is already in use by "7d7a251721bc4faac28e58fa607e657bcfc39252373672432ab90d432497be9b". You have to remove that container to be able to reuse that name.: that name is already in use
How to get out of it ? I already removed and added again the NFS service from host, but does not solve the issue.
Ste
admin
3,054 Posts
Quote from admin on August 28, 2025, 11:02 pmwhat is output of
apt-cache policy podman
apt install podman
what is output of
apt-cache policy podman
apt install podman