Forums

Home / Forums

You need to log in to create posts and topics. Login · Register

NFS server broken

Pages: 1 2 3

admin
3,054 Posts
August 31, 2025, 3:00 pm

can you try

apt install fuse3

there seem to be an issue with fuse upgrade if cluster was built earlier than version 2.6

#11

Ste
151 Posts
September 2, 2025, 2:31 pm

Hi, here is the output of all the three commands, fuse says is alrready up to date:

root@petasan04:~# apt-cache policy podman
podman:
Installed: 100:3.4.2-5
Candidate: 100:3.4.2-5
Version table:
*** 100:3.4.2-5 700
700 http://archive.petasan.org/repo_v3 petasan-v3/updates amd64 Packages
100 /var/lib/dpkg/status

root@petasan04:~# apt install podman
Reading package lists... Done
Building dependency tree
Reading state information... Done
podman is already the newest version (100:3.4.2-5).
The following packages were automatically installed and are no longer required:
containers-golang containers-image cri-o-runc libleveldb1d liblttng-ust-ctl4 liblttng-ust0 libvarlink linux-headers-5.4.0-117 linux-headers-5.4.0-117-generic
linux-image-5.4.0-117-generic linux-modules-5.4.0-117-generic linux-modules-extra-5.4.0-117-generic
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@petasan04:~#
root@petasan04:~#
root@petasan04:~#

root@petasan04:~# apt install fuse3
Reading package lists... Done
Building dependency tree
Reading state information... Done
fuse3 is already the newest version (3.9.0-2).
fuse3 set to manually installed.
The following packages were automatically installed and are no longer required:
containers-golang containers-image cri-o-runc libleveldb1d liblttng-ust-ctl4 liblttng-ust0 libvarlink linux-headers-5.4.0-117 linux-headers-5.4.0-117-generic
linux-image-5.4.0-117-generic linux-modules-5.4.0-117-generic linux-modules-extra-5.4.0-117-generic
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

This is tho PetaSAN log for today, things go fine up to the start of NFS service:

02/09/2025 17:06:33 ERROR NFSExportsServer : init_action() : Cannot create and run new container "NFS-10-220-12-211"

Error: error mounting storage for container f23c2acff4909a300c77c9b35d94d4506dc1dc60649c345e5a42d83e688fcf1f: error creating overlay mount to /var/lib/containers/storage/overlay/062f537c0d3d73172a4e4e679047ee7bcf8f795685a65b1ae267ea7f5d942852/merged, mount_data="nodev,metacopy=on,lowerdir=/var/lib/containers/storage/overlay/l/LPSLG32S64Z7QKKBWOJDRL4YOF:/var/lib/containers/storage/overlay/l/HBRK3HU4IHW7PIMUTWHIE23ECL:/var/lib/containers/storage/overlay/l/IZPPKBMYFIKJPIQQLZ36RVCR3X,upperdir=/var/lib/containers/storage/overlay/062f537c0d3d73172a4e4e679047ee7bcf8f795685a65b1ae267ea7f5d942852/diff,workdir=/var/lib/containers/storage/overlay/062f537c0d3d73172a4e4e679047ee7bcf8f795685a65b1ae267ea7f5d942852/work": invalid argument

02/09/2025 17:06:33 ERROR time="2025-09-02T17:06:33+02:00" level=error msg="error unmounting /var/lib/containers/storage/overlay/062f537c0d3d73172a4e4e679047ee7bcf8f795685a65b1ae267ea7f5d942852/merged: invalid argument"

02/09/2025 17:06:33 ERROR Container Manager : run() : Error running cmd : podman run --name NFS-10-220-12-211 --hostname NFS-10-220-12-211 -p 10.220.12.211:2049:2049 -d -t localhost/petasan-nfs-ganesha:3.2.0

02/09/2025 17:06:33 INFO Container Manager : creating NFS-10-220-12-211 container

02/09/2025 17:06:33 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container

02/09/2025 17:06:33 ERROR Container Manager : delete() : Error running cmd : podman rm -f NFS-10-220-12-211

02/09/2025 17:06:33 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container

02/09/2025 17:06:33 ERROR Container Manager : stop() : Error running cmd : podman stop NFS-10-220-12-211

02/09/2025 17:06:33 INFO Starting NFS Exports Service

02/09/2025 17:06:33 INFO NFSServer : waiting for the container NFS-10-220-12-211 to be up.

02/09/2025 17:06:32 INFO LockBase : Succeeded on acquiring the resource = NFS-10-220-12-211

02/09/2025 17:06:32 INFO LockBase : Try to acquire the resource = NFS-10-220-12-211.

02/09/2025 17:06:31 INFO NFSServer : clean local resource : 3) Stop and delete container of resource : NFS-10-220-12-211

02/09/2025 17:06:31 INFO NFSServer : clean local resource : 2) Delete ip address of resource : NFS-10-220-12-211

02/09/2025 17:06:31 INFO NFSServer : clean local resource : 1) Stopping NFS Exports service of resource : NFS-10-220-12-211

02/09/2025 17:06:31 INFO NFSServer : clean local resource : NFS-10-220-12-211

Bye, S.

Last edited on September 2, 2025, 4:25 pm by Ste · #12

admin
3,054 Posts
September 2, 2025, 5:37 pm

Try

apt install podman

after

apt install fuse3

(not before, sorry for confusion)

#13

Ste
151 Posts
September 4, 2025, 8:48 am

I get for both commands that packages are already at the latest versions, in fact podman was reinstalled on the 27th of august following your suggestion, just few messages before in this thread:

podman is already the newest version (100:3.4.2-5).
fuse3 is already the newest version (3.9.0-2).

 

 

#14

admin
3,054 Posts
September 9, 2025, 5:07 pm

can you please show

dpkg -l | grep "fuse-overlayfs"

#15

Ste
151 Posts
September 10, 2025, 9:28 am

root@petasan04:~# dpkg -l | grep "fuse-overlayfs"
ii fuse-overlayfs 0.7.6-1 amd64 implementation of overlay+shiftfs in FUSE for rootless containers

 

#16

Ste
151 Posts
September 22, 2025, 10:25 am

Good morning, is there any update in order to make the NFS server work again ?

In the meanwhile, there's a colleague who has a quite urgent need to access some files stored in the NFS volume that i currently can't mount due to the server issue. My question is: is there a way to temporarily access and copy the content of the NFS volume, just to give him a copy of his data ?

Thanks,  Ste

#17

admin
3,054 Posts
September 22, 2025, 1:07 pm

the only suggestion was to make sure podman is installed correctly.

you can access the data if you mount the cephfs directly form a linux client.

i am not sure if you see the nfs issue on 1 node only, if so you can try giving the nfs role to different servers.

 

 

#18

Ste
151 Posts
September 23, 2025, 2:57 pm

Well, i enabled NFS service on all other 3 nodes, but i got the same result: NFS service does not start !

So i decided to replicate the whole procedure on a different node (node03), first removing NFS role and then installing the new version of podman:

 root@petasan03:~# apt-cache policy podman

podman:
Installed: 1.6.2-1~ubuntu18.04~ppa1
Candidate: 100:3.4.2-5
Version table:
100:3.4.2-5 700
700 http://archive.petasan.org/repo_v3 petasan-v3/updates amd64 Packages
*** 1.6.2-1~ubuntu18.04~ppa1 100
100 /var/lib/dpkg/status
root@petasan03:~#
root@petasan03:~#

root@petasan03:~# apt install podman
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
containers-golang containers-image cri-o-runc libleveldb1d liblttng-ust-ctl4 liblttng-ust0 libvarlink
Use 'apt autoremove' to remove them.
The following additional packages will be installed:
catatonit criu crun dns-root-data dnsmasq-base fuse-overlayfs fuse3 libidn11 libnet1 podman-machine-cni podman-plugins python3-protobuf
The following packages will be REMOVED:
fuse
The following NEW packages will be installed:
catatonit criu crun dns-root-data dnsmasq-base fuse-overlayfs fuse3 libidn11 libnet1 podman-machine-cni podman-plugins python3-protobuf
The following packages will be upgraded:
podman
1 upgraded, 12 newly installed, 1 to remove and 0 not upgraded.
Need to get 23.4 MB of archives.
After this operation, 64.6 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 http://archive.ubuntu.com/ubuntu focal/universe amd64 fuse3 amd64 3.9.0-2 [24.7 kB]
Get:2 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-protobuf amd64 3.6.1.3-2ubuntu5.2 [298 kB]
Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 libnet1 amd64 1.1.6+dfsg-3.1build1 [43.3 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 dns-root-data all 2024071801~ubuntu0.20.04.1 [6,128 B]
Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libidn11 amd64 1.33-2.2ubuntu2 [46.2 kB]
Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 dnsmasq-base amd64 2.90-0ubuntu0.20.04.1 [350 kB]
Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 fuse-overlayfs amd64 0.7.6-1 [38.7 kB]
Get:3 https://archive.petasan.org/repo_v3 petasan-v3/updates amd64 catatonit amd64 0.1.7-1 [275 kB]
Get:9 https://archive.petasan.org/repo_v3 petasan-v3/updates amd64 criu amd64 3.17.1-1 [685 kB]
Get:10 https://archive.petasan.org/repo_v3 petasan-v3/updates amd64 crun amd64 100:1.2-2 [252 kB]
Get:11 https://archive.petasan.org/repo_v3 petasan-v3/updates amd64 podman-plugins amd64 100:1.2.0-1 [1,035 kB]
Get:12 https://archive.petasan.org/repo_v3 petasan-v3/updates amd64 podman-machine-cni amd64 100:0.0.0-1 [1,640 kB]
Get:13 https://archive.petasan.org/repo_v3 petasan-v3/updates amd64 podman amd64 100:3.4.2-5 [18.7 MB]
Fetched 23.4 MB in 7s (3,475 kB/s)
dpkg: fuse: dependency problems, but removing anyway as you requested:
glusterfs-client depends on fuse.

(Reading database ... 54089 files and directories currently installed.)
Removing fuse (2.9.9-3) ...
update-initramfs: deferring update (trigger activated)
Selecting previously unselected package fuse3.
(Reading database ... 54079 files and directories currently installed.)
Preparing to unpack .../00-fuse3_3.9.0-2_amd64.deb ...
Unpacking fuse3 (3.9.0-2) ...
Selecting previously unselected package catatonit.
Preparing to unpack .../01-catatonit_0.1.7-1_amd64.deb ...
Unpacking catatonit (0.1.7-1) ...
Selecting previously unselected package python3-protobuf.
Preparing to unpack .../02-python3-protobuf_3.6.1.3-2ubuntu5.2_amd64.deb ...
Unpacking python3-protobuf (3.6.1.3-2ubuntu5.2) ...
Selecting previously unselected package libnet1:amd64.
Preparing to unpack .../03-libnet1_1.1.6+dfsg-3.1build1_amd64.deb ...
Unpacking libnet1:amd64 (1.1.6+dfsg-3.1build1) ...
Selecting previously unselected package criu.
Preparing to unpack .../04-criu_3.17.1-1_amd64.deb ...
Unpacking criu (3.17.1-1) ...
Selecting previously unselected package crun.
Preparing to unpack .../05-crun_100%3a1.2-2_amd64.deb ...
Unpacking crun (100:1.2-2) ...
Selecting previously unselected package dns-root-data.
Preparing to unpack .../06-dns-root-data_2024071801~ubuntu0.20.04.1_all.deb ...
Unpacking dns-root-data (2024071801~ubuntu0.20.04.1) ...
Selecting previously unselected package libidn11:amd64.
Preparing to unpack .../07-libidn11_1.33-2.2ubuntu2_amd64.deb ...
Unpacking libidn11:amd64 (1.33-2.2ubuntu2) ...
Selecting previously unselected package dnsmasq-base.
Preparing to unpack .../08-dnsmasq-base_2.90-0ubuntu0.20.04.1_amd64.deb ...
Unpacking dnsmasq-base (2.90-0ubuntu0.20.04.1) ...
Selecting previously unselected package fuse-overlayfs.
Preparing to unpack .../09-fuse-overlayfs_0.7.6-1_amd64.deb ...
Unpacking fuse-overlayfs (0.7.6-1) ...
Selecting previously unselected package podman-plugins.
Preparing to unpack .../10-podman-plugins_100%3a1.2.0-1_amd64.deb ...
Unpacking podman-plugins (100:1.2.0-1) ...
Selecting previously unselected package podman-machine-cni.
Preparing to unpack .../11-podman-machine-cni_100%3a0.0.0-1_amd64.deb ...
Unpacking podman-machine-cni (100:0.0.0-1) ...
Preparing to unpack .../12-podman_100%3a3.4.2-5_amd64.deb ...
Unpacking podman (100:3.4.2-5) over (1.6.2-1~ubuntu18.04~ppa1) ...
dpkg: warning: unable to delete old directory '/etc/cni/net.d': Directory not empty
dpkg: warning: unable to delete old directory '/etc/cni': Directory not empty
Setting up libnet1:amd64 (1.1.6+dfsg-3.1build1) ...
Setting up dns-root-data (2024071801~ubuntu0.20.04.1) ...
Setting up fuse3 (3.9.0-2) ...
Installing new version of config file /etc/fuse.conf ...
update-initramfs: deferring update (trigger activated)
Setting up libidn11:amd64 (1.33-2.2ubuntu2) ...
Setting up catatonit (0.1.7-1) ...
Setting up fuse-overlayfs (0.7.6-1) ...
Setting up python3-protobuf (3.6.1.3-2ubuntu5.2) ...
Setting up dnsmasq-base (2.90-0ubuntu0.20.04.1) ...
Setting up podman-machine-cni (100:0.0.0-1) ...
Setting up podman-plugins (100:1.2.0-1) ...
Setting up criu (3.17.1-1) ...
Setting up crun (100:1.2-2) ...
Setting up podman (100:3.4.2-5) ...
Processing triggers for initramfs-tools (0.136ubuntu6.7) ...
update-initramfs: Generating /boot/initrd.img-5.14.21-08-petasan
Processing triggers for libc-bin (2.31-0ubuntu9.17) ...
Processing triggers for dbus (1.12.16-2ubuntu2.3) ...
root@petasan03:~#
root@petasan03:~#

root@petasan03:~# apt install fuse3
Reading package lists... Done
Building dependency tree
Reading state information... Done
fuse3 is already the newest version (3.9.0-2).
fuse3 set to manually installed.
The following packages were automatically installed and are no longer required:
containers-golang containers-image cri-o-runc libleveldb1d liblttng-ust-ctl4 liblttng-ust0 libvarlink
Use 'apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
root@petasan03:~#

root@petasan03:~# apt autoremove
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
containers-golang containers-image cri-o-runc libleveldb1d liblttng-ust-ctl4 liblttng-ust0 libvarlink
0 upgraded, 0 newly installed, 7 to remove and 0 not upgraded.
After this operation, 7,552 kB disk space will be freed.
Do you want to continue? [Y/n]
(Reading database ... 54389 files and directories currently installed.)
Removing containers-golang (0.3.1-1~ubuntu18.04~ppa3) ...
Removing containers-image (5.0.0-1~ubuntu18.04~ppa1) ...
Removing cri-o-runc (1.0.0-rc8-1~ubuntu18.04~ppa1) ...
Removing libleveldb1d:amd64 (1.22-3ubuntu2) ...
Removing liblttng-ust0:amd64 (2.11.0-1) ...
Removing liblttng-ust-ctl4:amd64 (2.11.0-1) ...
Removing libvarlink (18-2~ubuntu18.04~ppa4) ...
Processing triggers for libc-bin (2.31-0ubuntu9.17) ...
root@petasan03:~#
root@petasan03:~#

Then i added again the NFS role to node03, this is the log output, in reverse time order:

23/09/2025 16:47:10 INFO NFSServer : sync Consul settings -> done
23/09/2025 16:47:10 INFO NFSServer : sync Consul settings
23/09/2025 16:47:06 INFO Container Manager : creating NFS-10-220-12-211 container
23/09/2025 16:47:06 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
23/09/2025 16:47:06 ERROR Container Manager : delete() : Error running cmd : podman rm -f NFS-10-220-12-211
23/09/2025 16:47:06 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
23/09/2025 16:47:06 ERROR Container Manager : stop() : Error running cmd : podman stop NFS-10-220-12-211
23/09/2025 16:47:06 INFO Starting NFS Exports Service
23/09/2025 16:47:06 INFO NFSServer : waiting for the container NFS-10-220-12-211 to be up.
23/09/2025 16:47:05 INFO LockBase : Succeeded on acquiring the resource = NFS-10-220-12-211
23/09/2025 16:47:05 INFO LockBase : Try to acquire the resource = NFS-10-220-12-211.
23/09/2025 16:47:04 INFO NFSServer : sync Consul settings -> done
23/09/2025 16:47:04 INFO NFSServer : sync Consul settings
23/09/2025 16:47:02 INFO Clean all old local resources.
23/09/2025 16:47:02 INFO LockBase : Successfully dropped old sessions
23/09/2025 16:46:52 INFO LockBase : Dropping old sessions
23/09/2025 16:46:50 INFO Container Manager : Loading /opt/petasan/container-images/petasan-nfs-ganesha-3.2.0.tar.gz image into podman
23/09/2025 16:46:49 INFO Container Manager : deleting all old containers
23/09/2025 16:46:45 INFO Starting NFSServer Service.
23/09/2025 16:46:15 INFO Clean all local resources.
consul.base.ConsulException: 500 rpc error making call: invalid session "f27e22b8-7a8c-0788-cf3b-ac0ad1741452"
raise ConsulException("%d %s" % (response.code, response.body))
File "/usr/lib/python3/dist-packages/consul/base.py", line 188, in _status
CB._status(response, allow_404=allow_404)
File "/usr/lib/python3/dist-packages/consul/base.py", line 223, in cb
return callback(res)
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 98, in put
raise e
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 109, in put
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
raise value
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
raise attempt.get()
File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
return self.agent.http.put(
File "/usr/lib/python3/dist-packages/consul/base.py", line 620, in put
result = cons.kv.put(key=key, value=value, cas=cas, flags=flags, acquire=acquire, release=release,
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 161, in put
return consul_obj.put(key, data, None, None, acquire=str(session))
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 211, in lock_key
ret = ConsulAPI().lock_key(resource_key, self.session, resource_value)
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/lock_base.py", line 820, in do_acquire_resource
status = self.do_acquire_resource(resource)
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/lock_base.py", line 779, in acquire_resource
self.acquire_resource()
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/lock_base.py", line 378, in do_process
while not self.do_process(): # if it returns False , try again
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/lock_base.py", line 326, in process
self.process()
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/lock_base.py", line 269, in start
Traceback (most recent call last):
23/09/2025 16:46:15 ERROR 500 rpc error making call: invalid session "f27e22b8-7a8c-0788-cf3b-ac0ad1741452"
23/09/2025 16:46:15 ERROR LockBase : start() --> Error during process.
23/09/2025 16:46:15 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:46:15 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:46:15 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:46:15 INFO Checking backend latencies :
23/09/2025 16:46:15 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:45:28 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:45:28 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:45:28 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:45:28 INFO Checking backend latencies :
23/09/2025 16:45:28 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:44:57 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:44:57 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:44:57 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:44:57 INFO Checking backend latencies :
23/09/2025 16:44:57 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:44:34 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:44:34 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:44:34 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:44:34 INFO Checking backend latencies :
23/09/2025 16:44:34 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:44:15 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:44:15 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:44:15 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:44:15 INFO Checking backend latencies :
23/09/2025 16:44:15 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:58 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:58 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:58 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:58 INFO Checking backend latencies :
23/09/2025 16:43:58 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:42 INFO LockBase : Try to acquire the resource = NFS-10-220-12-211.
23/09/2025 16:43:42 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
23/09/2025 16:43:42 ERROR Container Manager : delete() : Error running cmd : podman rm -f NFS-10-220-12-211
23/09/2025 16:43:42 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
23/09/2025 16:43:42 ERROR Container Manager : stop() : Error running cmd : podman stop NFS-10-220-12-211
23/09/2025 16:43:42 INFO NFSServer : clean local resource : 3) Stop and delete container of resource : NFS-10-220-12-211
23/09/2025 16:43:42 INFO NFSServer : clean local resource : 2) Delete ip address of resource : NFS-10-220-12-211
23/09/2025 16:43:41 INFO Stopping NFS Exports Service
23/09/2025 16:43:40 INFO NFSServer : clean local resource : 1) Stopping NFS Exports service of resource : NFS-10-220-12-211
23/09/2025 16:43:40 INFO NFSServer : clean local resource : NFS-10-220-12-211
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:36 INFO Checking backend latencies :
23/09/2025 16:43:36 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:36 INFO Checking backend latencies :
23/09/2025 16:43:36 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:36 INFO Checking backend latencies :
23/09/2025 16:43:36 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:36 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:36 INFO Checking backend latencies :
23/09/2025 16:43:36 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:35 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:35 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:35 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:35 INFO Checking backend latencies :
23/09/2025 16:43:35 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:33 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:33 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:33 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:33 INFO Checking backend latencies :
23/09/2025 16:43:33 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:15 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:15 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:15 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:15 INFO Checking backend latencies :
23/09/2025 16:43:15 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:13 INFO WatchBase unfound key action
PetaSAN.core.consul.ps_consul.RetryConsulException
raise RetryConsulException()
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get
raise e
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 86, in get
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
raise value
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
raise attempt.get()
File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
return self.agent.http.get(
File "/usr/lib/python3/dist-packages/consul/base.py", line 551, in get
index, data = cons.kv.get(key=key, index=index, recurse=recurse, wait=wait,
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 119, in get
version, data = consul_obj.get(key, lock_index, recurse, timeout)
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/api.py", line 141, in get_key_blocking
Traceback (most recent call last):
23/09/2025 16:43:13 ERROR
23/09/2025 16:43:13 ERROR ConsulAPI error in "get_key_blocking", could not find key resource.
23/09/2025 16:43:13 INFO Network latency for backend 192.168.110.3 = 4.08 us
23/09/2025 16:43:13 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:13 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:13 INFO Checking backend latencies :
23/09/2025 16:43:13 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:12 INFO Checking backend latencies :
23/09/2025 16:43:12 INFO Checking backend latencies :
23/09/2025 16:43:12 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:12 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.3 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:12 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:12 INFO Checking backend latencies :
23/09/2025 16:43:12 INFO Checking backend latencies :
23/09/2025 16:43:12 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:12 INFO Consul is not responding, will check cluster backend ips:
PetaSAN.core.consul.ps_consul.RetryConsulException
raise RetryConsulException()
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 74, in get
raise e
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/ps_consul.py", line 86, in get
attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
File "/usr/lib/python3/dist-packages/retrying.py", line 200, in call
raise value
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
six.reraise(self.value[0], self.value[1], self.value[2])
File "/usr/lib/python3/dist-packages/retrying.py", line 247, in get
raise attempt.get()
File "/usr/lib/python3/dist-packages/retrying.py", line 212, in call
return Retrying(*dargs, **dkw).call(f, *args, **kw)
File "/usr/lib/python3/dist-packages/retrying.py", line 49, in wrapped_f
return self.agent.http.get(
File "/usr/lib/python3/dist-packages/consul/base.py", line 551, in get
index, data = cons.kv.get(key, index=current_index, recurse=True)
File "/usr/lib/python3/dist-packages/PetaSAN/core/consul/base.py", line 77, in watch
index, data = base.watch(self.root_path, current_index)
File "/usr/lib/python3/dist-packages/PetaSAN/backend/file_sync_manager.py", line 81, in sync
Traceback (most recent call last):
23/09/2025 16:43:11 ERROR
23/09/2025 16:43:11 INFO Network latency for backend 192.168.110.3 = 4.1 us
23/09/2025 16:43:11 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:11 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:11 INFO Checking backend latencies :
23/09/2025 16:43:11 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:43:09 INFO Network latency for backend 192.168.110.3 = 4.21 us
23/09/2025 16:43:09 INFO Network latency for backend 192.168.110.1 =
23/09/2025 16:43:09 INFO Network latency for backend 192.168.110.2 =
23/09/2025 16:43:09 INFO Checking backend latencies :
23/09/2025 16:43:09 INFO Consul is not responding, will check cluster backend ips:
23/09/2025 16:42:41 INFO Container Manager : creating NFS-10-220-12-211 container
23/09/2025 16:42:41 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
23/09/2025 16:42:41 ERROR Container Manager : delete() : Error running cmd : podman rm -f NFS-10-220-12-211
23/09/2025 16:42:41 ERROR Error: no container with name or ID "NFS-10-220-12-211" found: no such container
23/09/2025 16:42:41 ERROR Container Manager : stop() : Error running cmd : podman stop NFS-10-220-12-211
23/09/2025 16:42:41 INFO Starting NFS Exports Service
23/09/2025 16:42:41 INFO NFSServer : waiting for the container NFS-10-220-12-211 to be up.
23/09/2025 16:42:40 INFO LockBase : Succeeded on acquiring the resource = NFS-10-220-12-211
23/09/2025 16:42:40 INFO LockBase : Try to acquire the resource = NFS-10-220-12-211.
23/09/2025 16:42:39 INFO NFSServer : sync Consul settings -> done
23/09/2025 16:42:39 INFO NFSServer : sync Consul settings
23/09/2025 16:42:37 INFO Clean all old local resources.
23/09/2025 16:42:37 INFO LockBase : Successfully dropped old sessions
23/09/2025 16:42:27 INFO LockBase : Dropping old sessions
23/09/2025 16:42:24 INFO Container Manager : Loading /opt/petasan/container-images/petasan-nfs-ganesha-3.2.0.tar.gz image into podman
23/09/2025 16:42:24 INFO Container Manager : deleting all old containers
23/09/2025 16:42:21 INFO sync_replication_node completed
23/09/2025 16:42:21 INFO syncing replication users ok
23/09/2025 16:42:21 INFO syncing cron ok
23/09/2025 16:42:20 INFO sync_replication_node starting
23/09/2025 16:42:20 ERROR rm: cannot remove '/var/tmp/docker-tar*': No such file or directory
23/09/2025 16:42:20 ERROR NFSServer : Error running cmd : rm /var/tmp/docker-tar*
23/09/2025 16:42:20 INFO Starting NFSServer Service.
23/09/2025 16:42:19 INFO Update node info in file
23/09/2025 16:42:19 INFO Starting PetaSAN-nfs service
23/09/2025 16:42:19 INFO Update node nfs role to true
23/09/2025 16:42:19 INFO Update roles.

From the linux client this is what happens (10.220.12.211 is the IP reserved for NFS):

[stefano@vega1:~]$ rpcinfo -p 10.220.12.211
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper

[stefano@vega1:~]$ sudo mount -vvv -t nfs4 10.220.12.211:/vega /mnt/nfs/vega1
mount.nfs4: timeout set for Tue Sep 23 14:52:17 2025
mount.nfs4: trying text-based options 'vers=4.2,addr=10.220.12.211,clientaddr=10.220.13.184'
mount.nfs4: mount(2): Device or resource busy

Any other suggestion ? Thanks

 

Last edited on September 23, 2025, 2:59 pm by Ste · #19

admin
3,054 Posts
September 24, 2025, 12:14 pm

Difficult to say what is happening. If you wish, you can get support from us as it will requires time to debug.

Note that as per prev suggestion, you can access the data from cephs directly mounted on a linux client without nfs

 

#20
Pages: 1 2 3