XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Xen Orchestra web UI becomes unresponsive when an NFS backup remote goes offline

    Scheduled Pinned Locked Moved Unsolved Xen Orchestra
    18 Posts 4 Posters 761 Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • florentF
      florent Vates 🪐 XO Team
      last edited by

      when a NFS munt in hard mount mode fails, it is quite tricky to handle on user land, and falling back to a soft mount means we accept to loose data without knowing

      To unstuck a XO with a brken NFS , we added a cli tool that allow you to disable a remote here https://github.com/vatesfr/xen-orchestra/pull/10587

      F 1 Reply Last reply
      Reply Quote 0
      • F
        frb94 @florent
        last edited by

        @florent

        I performed another test to reproduce the issue and evaluate the recovery mechanism introduced in PR #10587.

        Unfortunately, I encountered a limitation that may prevent the new disable-remote command from being used in the exact failure scenario it is intended to address.

        1. Reproducing the issue

        With the NFS remote enabled, I shut down the NAS.

        Initially, Xen Orchestra remained accessible, but one libuv worker became blocked in rpc_wait_bit_killable.

        After some time, all four libuv workers were blocked:

        TID  STAT  WCHAN                    COMMAND
        863  Dsl   rpc_wait_bit_killable    libuv-worker
        864  Dsl   rpc_wait_bit_killable    libuv-worker
        865  Dsl   rpc_wait_bit_killable    libuv-worker
        866  Dsl   rpc_wait_bit_killable    libuv-worker
        

        The xo-server process was still running, but the libuv worker pool was completely blocked.

        I also observed multiple df -kP processes attempting to access the unavailable NFS mount.

        2. Attempting to stop xo-server

        I then executed:

        systemctl stop xo-server
        

        However, systemd was unable to terminate the service, even after sending SIGKILL.

        The final status was:

        Active: failed (Result: timeout)
        Main PID: 738
        
        Processes still around after SIGKILL. Ignoring.
        State 'final-sigterm' timed out. Killing.
        Killing process 738 (MainThread) with signal SIGKILL.
        Processes still around after final SIGKILL. Entering failed mode.
        Unit process 738 (MainThread) remains running after unit stopped.
        

        Checking the remaining process:

        ps -p 738 -o pid,ppid,stat,wchan:32,comm
        

        Returned:

        PID  PPID  STAT  WCHAN                    COMMAND
        738     1  Ds    rpc_wait_bit_killable    MainThread
        

        All other Node.js threads had disappeared, but the main thread remained stuck in an uninterruptible kernel wait.

        3. Attempting a lazy unmount

        I then tried:

        umount -l /run/xo-server/mounts/9fc232df-4d0c-4a15-93b5-c0585c1dd9c9
        

        The command completed successfully, and findmnt -t nfs,nfs4 no longer showed the NFS mount.

        However, the Node.js process remained blocked:

        PID  STAT  WCHAN                    COMMAND
        738  Ds    rpc_wait_bit_killable    MainThread
        

        Therefore, detaching the NFS mount did not release the already pending kernel operation.

        4. Implications for PR #10587

        As I understand it, the new xo-server-db disable-remote <id> command is intended to be executed while xo-server is stopped.

        However, this test shows that when an NFS hard remote becomes unavailable:

        1. All four libuv workers can become blocked.
        2. xo-server becomes unresponsive.
        3. systemctl stop xo-server can fail, even after SIGKILL.
        4. The Node.js process can remain stuck in an uninterruptible kernel wait.
        5. Even umount -l does not necessarily release the blocked process.

        5. Question about the intended recovery procedure

        If xo-server cannot be fully stopped because its process remains stuck in state D, how is the new disable-remote command expected to be used safely?

        Is the intended procedure to execute the database command while the old Node.js process is still present but systemd reports the service as failed?

        Or is a VM reboot / restoration of NFS connectivity still required before the recovery command can be used?

        I have not executed disable-remote in this condition yet, because the original Node.js process is still present and I wanted to avoid modifying the database while it might still be in use.

        My concern is that the new CLI provides a way to prevent the problematic remote from being accessed on the next startup, but does not necessarily provide a complete recovery path when the existing xo-server process cannot terminate.

        I would appreciate clarification on the intended recovery procedure for this specific scenario.

        1 Reply Last reply
        Reply Quote 0
        • F
          frb94 @poddingue
          last edited by

          @poddingue @florent

          I really do not want to look like i'm not appreciating the platform. I am a sysadmin currently evaluating XCP-ng and Xen Orchestra as a replacement for VMware, and I have to say this issue is making me a bit concerned. Of course for now it's my homelab, but it is already a show stopper for me.

          I understand that NFS hard mounts can block when the server goes offline, but I really wouldn't expect an unavailable backup repository to make the entire XO Web UI unresponsive, especially when no backup jobs are running.

          To me, this feels like vCenter becoming unusable just because one of the backup repositories is down. That's a pretty big deal in a production environment. Also, i have not yet tested what happens to vms running on local SR and management plane of a host when the main NFS SR may become unavailable.

          What worries me even more is that, as my tests showed, once XO gets stuck, even SIGKILL can't terminate the process, so recovering isn't straightforward either.

          Am I missing something here? Is there something wrong with my setup, or is this actually expected behavior?

          I really like XCP-ng and XO so far, but I'm trying to understand how companies handle this kind of situation in production. Surely a failed backup destination shouldn't take down the whole management interface?

          acebmxerA 1 Reply Last reply
          Reply Quote 1
          • acebmxerA
            acebmxer @frb94
            last edited by

            @frb94

            just to confirm you are disabling the Backup NFS SR? Before it goes off line? I can try this again in my setup just not sure if i still kept the script i created that the synology would used to send the disable the remote before shutdown.

            F 1 Reply Last reply
            Reply Quote 0
            • F
              frb94 @acebmxer
              last edited by frb94

              @acebmxer Hello, no. Let me describe my setup.

              Esxi + vcenter + veeam. Vcenter wakes up veaam at 9am. Veeam sends wol packet to the synology nas at 9.15. At 9.30 the backup starts and ends more or less at 10.00
              Due to inactivity synology stops the hard disks, then after some time it shut itself off. Veeam just see the repo as unavailable. At 12.00 veeam auto shuts down.

              Migrating to xcpng + xo, veeam is retired. At first i added the remote to XO, believing it's like veeam. If the backend goes down it simply shows unavailable. Then i discovered is not at all like that when i found an inaccessible XO. After some chatgpt-ing i found out this "bug". I thought that the script would be sufficient, but it's not. Waiting for ping to fail and then disable the remote (cron every 5mins) still ends up in

              ps -T -p 738 -o pid,tid,stat,wchan:32,comm
                  PID     TID STAT WCHAN                            COMMAND
                  738     738 Ssl  do_epoll_wait                    MainThread
                  738     753 Ssl  do_epoll_wait                    DelayedTaskSche
                  738     754 Ssl  futex_wait_queue                 V8Worker
                  738     755 Ssl  futex_wait_queue                 V8Worker
                  738     756 Ssl  futex_wait_queue                 V8Worker
                  738     757 Ssl  futex_wait_queue                 V8Worker
                  738     758 Ssl  futex_wait_queue                 SignalInspector
                  738     863 Dsl  rpc_wait_bit_killable            libuv-worker
                  738     864 Dsl  rpc_wait_bit_killable            libuv-worker
                  738     865 Dsl  rpc_wait_bit_killable            libuv-worker
                  738     866 Dsl  rpc_wait_bit_killable            libuv-worker
              

              And it's even worse because not only XO becomes unavailable, but even if i disable manually (or via script) the remote as soon as the backup ends, with the synology still ON, the linux os does not unmount the nfs.
              Today the remote was disabled before the nas shutdown and correctly XO kept working. But if at this point i want to restart the XO vm i must do this before shutdown

              umount -l /run/xo-server/mounts/9fc232df-4d0c-4a15-93b5-c0585c1dd9c9
              

              otherwise it stays like this for at least 15 minutes, then finally reboots

              image.jpeg

              The ideal behaviour would be that XO does all by itself, but since we can't have it all, the closest thing i have in mind is

              • Script that sends WOL to the nas, when it pings than enables the remote
              • Script that monitors the backup acrivity. when the last job is done, it must disable the remote and also umount the linux os nfs mount.

              Then the nas could shut down automatically without issue.

              Since i'm not a dev i need to take my time with chatgpt and see i what i can do.

              Having said that, my concern is not about my homelab, it's about production. In my environment i can and want to simulate all kinds of malfuncions. In the previous message i said i still need to try to shut down the main nfs SR for vms and see how the host behaves.
              But if i decide to deploy XEN at my company, i cant have 1 remote unavailability cause the whole XO (or XOA in that case) fall apart.

              Hope it helps!!

              1 Reply Last reply
              Reply Quote 0
              • acebmxerA
                acebmxer
                last edited by acebmxer

                So have disabled the synology device from the remotes page.

                Screenshot_20261112_162543.png

                I then logged into the synology device and did a normal shutdown. (had maintenance work to do anyway). So far the synology device has been powered off for going on 30 min and XO is still responding just fine. So far the only thing i am seeing is you can not see the backups / restore points as the remote is offline. (expected) Will keep it off and see when it stops (if it stops responding). I did migrate xo and it stayed up. I can try a reboot and / or apply the latest commits while the synology device is off.

                While discussing this with Claude as I was having issues seeing the obvious. That disabling the remote(s) xo does not unmount the remote(s) all the time... As shown above I have two remotes that are disabled and one remote is still mounted, shown below. Also not knowing the need to disable the remote before it goes offline or when an unexpected outage occurs.

                This is also possibly what I ran into last time and didnt bother to follow up with about it.

                ❯ findmnt -t nfs,nfs4
                TARGET                                 SOURCE  FSTYPE OPTIONS
                /run/xo-server/mounts/2ab02323-0d47-43a5-9081-b2454e036c70
                                                       10.100.2.23:/volume1/backup
                                                               nfs4   rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=
                

                Update: I re-tested with the Synology powered on the whole time. I enabled both remotes, then disabled both. Both are still mounted:

                ❯ findmnt -t nfs,nfs4
                TARGET                                  SOURCE    FSTYPE OPTIONS
                /run/xo-server/mounts/2ab02323-0d47-43a5-9081-b2454e036c70
                                                        10.100.2.23:/volume1/backup
                                                                  nfs4   rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,s
                /run/xo-server/mounts/588145e7-1e41-4e61-a717-c0b4cce7646a
                                                        10.100.253.23:/volume1/Backups
                                                                  nfs4   rw,relatime,vers=4.1,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,s
                

                xo-server still has files open on both mounts (deleted keeper files, renamed .nfs… by NFS), which keeps them busy so they can't be unmounted:

                ❯ sudo ls -l /proc/$(systemctl show -p MainPID --value xo-server)/fd | grep xo-server/mounts
                l-wx------ 1 root root 64 Nov 12 22:15 30 -> /run/xo-server/mounts/2ab02323-0d47-43a5-9081-b2454e036c70/.nfs000000000000443400000005
                l-wx------ 1 root root 64 Nov 12 22:15 34 -> /run/xo-server/mounts/2ab02323-0d47-43a5-9081-b2454e036c70/.nfs000000000000443e00000009
                l-wx------ 1 root root 64 Nov 12 22:15 36 -> /run/xo-server/mounts/2ab02323-0d47-43a5-9081-b2454e036c70/.nfs000000000000443d00000008
                l-wx------ 1 root root 64 Nov 12 22:15 39 -> /run/xo-server/mounts/588145e7-1e41-4e61-a717-c0b4cce7646a/.nfs00000000000036b70000000b
                l-wx------ 1 root root 64 Nov 12 22:15 40 -> /run/xo-server/mounts/588145e7-1e41-4e61-a717-c0b4cce7646a/.nfs00000000000036b80000000c
                

                XO from sources, commit 67ed8f7eb (2026-11-12), Node v26.13.0, Debian 13 (trixie).

                F 1 Reply Last reply
                Reply Quote 0
                • F
                  frb94 @acebmxer
                  last edited by frb94

                  @acebmxer that is exactley what i was saying before. if you disable the remote before nas shutdown you are good on the XO side and will work indefinitely, but if you don't execute

                  umount -l /run/xo-server/mounts/2ab02323-0d47-43a5-9081-b2454e036c70
                  

                  you will have the mount stuck on the os side. You will be able to reboot XO vm eventually but you will have to wait for a long timeout during XO shutdown.

                  if you want to simulate my issue just disable the remote 5 minutes after shutting down the synology, get the pid from

                  service xo-server status
                  

                  then

                  ps -T pid
                  

                  after a while you'll have 4 "rpc_wait_bit_killable" and XO will stop responding

                  poddingueP 1 Reply Last reply
                  Reply Quote 0
                  • poddingueP
                    poddingue Vates 🪐 @frb94
                    last edited by

                    @frb94 thanks for testing it that far, and no, I don't think you're missing something obvious. The question at the end of your last test, how to run disable-remote when xo-server won't stop, is really one for @florent, so I'll leave it with him.

                    @acebmxer the .nfs… files still held open after you disabled the remotes (your post) look like a separate finding, and worth having next to the rest. The GitHub issue (https://github.com/vatesfr/xen-orchestra/issues/10741) only has the original report and Florent's reply so far. If either of you has a minute to add these two tests there, it keeps everything in one place for the devs. 🙏

                    Happy to copy them over myself if that's easier. 🤷

                    F acebmxerA 2 Replies Last reply
                    Reply Quote 0
                    • F
                      frb94 @poddingue
                      last edited by

                      @poddingue Hello, i added my part regarding both disable-remote and umount -l

                      Just as an update for everyone. This nfs behaviour was driving me crazy so i decided to go back to veeam community edition as my backup solution.

                      Now XO does only metadata backup on a local folder, then veeam covers everything, XO included.

                      What i've lost

                      • Fully integrated solution
                      • Easy management with tagging
                      • Very low resource use

                      What i've gained

                      • Significantly less storage space used on the backup repo
                      • Easy to identify backup files (no uuid)
                      • Two additional vm to manage (veeam + veeam worker)

                      It's not as complete as it was with vmware, for obvious reasons, but i have to say Vates and Veeam did an amazing job. Looking forward for application aware processing !!

                      1 Reply Last reply
                      Reply Quote 0
                      • acebmxerA
                        acebmxer @poddingue
                        last edited by

                        @poddingue

                        added my info as well.

                        1 Reply Last reply
                        Reply Quote 0

                        Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                        Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                        With your input, this post could be even better 💗

                        Register Login
                        • First post
                          Last post