@florent
I performed another test to reproduce the issue and evaluate the recovery mechanism introduced in PR #10587.
Unfortunately, I encountered a limitation that may prevent the new disable-remote command from being used in the exact failure scenario it is intended to address.
1. Reproducing the issue
With the NFS remote enabled, I shut down the NAS.
Initially, Xen Orchestra remained accessible, but one libuv worker became blocked in rpc_wait_bit_killable.
After some time, all four libuv workers were blocked:
TID STAT WCHAN COMMAND
863 Dsl rpc_wait_bit_killable libuv-worker
864 Dsl rpc_wait_bit_killable libuv-worker
865 Dsl rpc_wait_bit_killable libuv-worker
866 Dsl rpc_wait_bit_killable libuv-worker
The xo-server process was still running, but the libuv worker pool was completely blocked.
I also observed multiple df -kP processes attempting to access the unavailable NFS mount.
2. Attempting to stop xo-server
I then executed:
systemctl stop xo-server
However, systemd was unable to terminate the service, even after sending SIGKILL.
The final status was:
Active: failed (Result: timeout)
Main PID: 738
Processes still around after SIGKILL. Ignoring.
State 'final-sigterm' timed out. Killing.
Killing process 738 (MainThread) with signal SIGKILL.
Processes still around after final SIGKILL. Entering failed mode.
Unit process 738 (MainThread) remains running after unit stopped.
Checking the remaining process:
ps -p 738 -o pid,ppid,stat,wchan:32,comm
Returned:
PID PPID STAT WCHAN COMMAND
738 1 Ds rpc_wait_bit_killable MainThread
All other Node.js threads had disappeared, but the main thread remained stuck in an uninterruptible kernel wait.
3. Attempting a lazy unmount
I then tried:
umount -l /run/xo-server/mounts/9fc232df-4d0c-4a15-93b5-c0585c1dd9c9
The command completed successfully, and findmnt -t nfs,nfs4 no longer showed the NFS mount.
However, the Node.js process remained blocked:
PID STAT WCHAN COMMAND
738 Ds rpc_wait_bit_killable MainThread
Therefore, detaching the NFS mount did not release the already pending kernel operation.
4. Implications for PR #10587
As I understand it, the new xo-server-db disable-remote <id> command is intended to be executed while xo-server is stopped.
However, this test shows that when an NFS hard remote becomes unavailable:
All four libuv workers can become blocked.
xo-server becomes unresponsive.
systemctl stop xo-server can fail, even after SIGKILL.
The Node.js process can remain stuck in an uninterruptible kernel wait.
Even umount -l does not necessarily release the blocked process.
5. Question about the intended recovery procedure
If xo-server cannot be fully stopped because its process remains stuck in state D, how is the new disable-remote command expected to be used safely?
Is the intended procedure to execute the database command while the old Node.js process is still present but systemd reports the service as failed?
Or is a VM reboot / restoration of NFS connectivity still required before the recovery command can be used?
I have not executed disable-remote in this condition yet, because the original Node.js process is still present and I wanted to avoid modifying the database while it might still be in use.
My concern is that the new CLI provides a way to prevent the problematic remote from being accessed on the next startup, but does not necessarily provide a complete recovery path when the existing xo-server process cannot terminate.
I would appreciate clarification on the intended recovery procedure for this specific scenario.
