<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Xen Orchestra web UI becomes unresponsive when an NFS backup remote goes offline]]></title><description><![CDATA[<p dir="auto">Hi everyone,</p>
<p dir="auto">I am a very new user of XCPNG in my homelab. I think it's a fantastic product and i'm very happy with it. I have an issue that i did not find any reference online. I tried to troubleshoot it with my knowledge, but i did not go far. The following paragraph was written by chatgpt with the finding "we" had. It's all correct as far as i can tell.</p>
<p dir="auto">I'm experiencing a reproducible issue with Xen Orchestra when an NFS backup remote becomes unavailable.</p>
<p dir="auto">My backup storage is hosted on a Synology NAS configured with Auto Poweroff, which automatically shuts down the NAS after a period of inactivity.</p>
<p dir="auto">Xen Orchestra remains powered on 24/7.</p>
<p dir="auto">The expectation is that an unavailable backup remote should not affect the availability of the Xen Orchestra management interface, especially when no backup job is running.</p>
<p dir="auto">However, after the NAS has been powered off for some time, the entire XO web interface becomes unresponsive.</p>
<p dir="auto">After investigating the issue, I found that multiple Node.js libuv worker threads become blocked inside NFS kernel operations.</p>
<p dir="auto"><strong>Environment</strong></p>
<ul>
<li>Xen Orchestra: Installed from source (ronivay script)</li>
<li></li>
<li>XO build: xen-orchestra-202611011738 (November 1, 2026)</li>
<li></li>
<li>XO server: Linux, managed by systemd</li>
<li></li>
<li>XO server IP: 10.0.0.30</li>
<li></li>
<li>Backup storage: Synology NAS</li>
<li></li>
<li>NAS IP: 10.0.0.28</li>
<li></li>
<li>Remote type: NFS</li>
<li></li>
<li>NFS version: 4.0</li>
<li></li>
<li>NFS mount options: hard,timeo=600,retrans=2</li>
<li></li>
<li>NAS power management: Automatic shutdown after inactivity</li>
</ul>
<p dir="auto"><strong>Steps to reproduce</strong></p>
<ul>
<li>Configure an NFS backup remote in Xen Orchestra.</li>
<li></li>
<li>Run backup jobs successfully.</li>
<li></li>
<li>Wait until all backup operations have finished.</li>
<li></li>
<li>Allow the Synology NAS to shut down automatically.</li>
<li></li>
<li>Keep Xen Orchestra running.</li>
<li></li>
<li>After some time, try opening the XO web interface.</li>
</ul>
<p dir="auto">Same behaviour when trying to enable the already-disabled remote while nas is off.</p>
<p dir="auto">Observed result: Chrome keeps loading indefinitely, without displaying the login page.</p>
<p dir="auto">The issue occurs even when accessing XO directly by its IP address.</p>
<p dir="auto"><strong>Investigation</strong></p>
<ol>
<li>The NFS mount remains active</li>
</ol>
<p dir="auto">Even though the NAS is offline, the NFS remote remains mounted:</p>
<p dir="auto">10.0.0.28:/volume2/xobackup<br />
/run/xo-server/mounts/9fc232df-4d0c-4a15-93b5-c0585c1dd9c9<br />
nfs4 rw,vers=4.0,hard,proto=tcp,timeo=600,retrans=2</p>
<p dir="auto">The Linux kernel repeatedly reports:</p>
<p dir="auto">nfs: server 10.0.0.28 not responding, timed out<br />
nfs: server 10.0.0.28 not responding, still trying</p>
<p dir="auto">These messages continue for hours while the NAS is unavailable.</p>
<ol start="2">
<li>XO remains running</li>
</ol>
<p dir="auto">The service is still reported as active:</p>
<p dir="auto">xo-server.service - XO Server<br />
Active: active (running)<br />
Main PID: 729 (MainThread)</p>
<p dir="auto">The main Node.js thread is not blocked in NFS I/O:</p>
<p dir="auto">PID  STAT  WCHAN<br />
729  Ssl   do_epoll_wait</p>
<ol start="3">
<li>Multiple libuv workers are blocked</li>
</ol>
<p dir="auto">This is the most interesting finding.</p>
<p dir="auto">Running:</p>
<p dir="auto">ps -T -p 729 -o pid,tid,stat,wchan:32,comm</p>
<p dir="auto">produces:</p>
<p dir="auto">PID  TID   STAT  WCHAN                    COMMAND<br />
729  729   Ssl   do_epoll_wait            MainThread<br />
729  862   Dsl   rpc_wait_bit_killable    libuv-worker<br />
729  863   Dsl   rpc_wait_bit_killable    libuv-worker<br />
729  864   Dsl   rpc_wait_bit_killable    libuv-worker<br />
729  865   Dsl   rpc_wait_bit_killable    libuv-worker<br />
729  2840  Ssl   futex_wait_queue         libuv-worker</p>
<p dir="auto">Four libuv workers are blocked in uninterruptible sleep, waiting for NFS RPC operations.</p>
<ol start="4">
<li>HTTP works without compression</li>
</ol>
<p dir="auto">Interestingly, XO still responds correctly to uncompressed HTTP requests.</p>
<p dir="auto">From the XO machine:</p>
<p dir="auto">curl -k <a href="https://127.0.0.1/signin" target="_blank" rel="noopener noreferrer nofollow ugc">https://127.0.0.1/signin</a></p>
<p dir="auto">Result:</p>
<p dir="auto">HTTP: 200<br />
Time: 0.005740s</p>
<p dir="auto">From another Windows PC:</p>
<p dir="auto">curl.exe -k -v <a href="https://10.0.0.30/signin" target="_blank" rel="noopener noreferrer nofollow ugc">https://10.0.0.30/signin</a> -o NUL</p>
<p dir="auto">Result:</p>
<p dir="auto">HTTP/1.1 200 OK<br />
Content-Length: 5664<br />
100 5664</p>
<p dir="auto">The entire HTML document is transferred successfully.</p>
<ol start="5">
<li>HTTP compression hangs</li>
</ol>
<p dir="auto">However, enabling HTTP compression causes the request to hang.</p>
<p dir="auto">From Windows:</p>
<p dir="auto">curl.exe -k -v --compressed --max-time 10 <a href="https://10.0.0.30/signin" target="_blank" rel="noopener noreferrer nofollow ugc">https://10.0.0.30/signin</a> -o NUL</p>
<p dir="auto">Result:</p>
<p dir="auto">Accept-Encoding: deflate, gzip<br />
Request completely sent off</p>
<p dir="auto">Operation timed out after 10012 milliseconds<br />
with 0 bytes received</p>
<p dir="auto">The same behavior can be reproduced locally on the XO server using curl --compressed.</p>
<p dir="auto">This appears to explain why Chrome cannot display the login page, while basic uncompressed HTTP requests still succeed.</p>
<ol start="6">
<li>XO logs reference backup remote operations</li>
</ol>
<p dir="auto">The xo-server logs also show stack traces involving:</p>
<p dir="auto">BackupNg._listVmBackupsOnRemote<br />
BackupNg._listVmBackupsWithBackoff</p>
<p dir="auto">This suggests XO is attempting to access backup metadata on the unavailable remote.</p>
<p dir="auto">Possible root cause</p>
<p dir="auto">My hypothesis is that filesystem operations against the unavailable NFS hard mount block libuv worker threads.</p>
<p dir="auto">Node.js uses the libuv thread pool for asynchronous filesystem operations and some other operations, including zlib compression.</p>
<p dir="auto">If the available workers become blocked waiting for NFS RPC responses, HTTP compression tasks may be unable to complete.</p>
<p dir="auto">This would explain the observed behavior:</p>
<ul>
<li>Uncompressed HTTP requests work.</li>
<li></li>
<li>Compressed HTTP requests hang.</li>
<li></li>
<li>The browser cannot load the GUI.</li>
<li></li>
<li>Several libuv workers are stuck in rpc_wait_bit_killable.</li>
</ul>
<p dir="auto">I have confirmed the blocked workers and the compression-dependent HTTP timeout, but I have not yet confirmed the internal scheduling mechanism within Node.js.</p>
<p dir="auto"><strong>Recovery attempt</strong></p>
<p dir="auto">I attempted to terminate the blocked worker threads with kill -9.</p>
<p dir="auto">This caused the main Node.js thread to enter a zombie state, while one worker remained blocked:</p>
<p dir="auto">PID  TID  STAT  WCHAN<br />
729  729  Zsl   -<br />
729  864  Dsl   rpc_wait_bit_killable</p>
<p dir="auto">I also tried a lazy unmount:</p>
<p dir="auto">umount -l /run/xo-server/mounts/9fc232df-4d0c-4a15-93b5-c0585c1dd9c9</p>
<p dir="auto">The command completed, but the blocked worker remained in the same state.</p>
<p dir="auto">These were recovery experiments, not recommended procedures.</p>
<p dir="auto"><strong>Expected behavior</strong></p>
<p dir="auto">Ideally, if an NFS backup remote becomes unavailable:</p>
<ul>
<li>Backup operations involving that remote should report an error or remain isolated.</li>
<li></li>
<li>The management GUI should remain responsive.</li>
<li></li>
<li>Unrelated operations should continue functioning.</li>
<li></li>
<li>An unavailable backup repository should not exhaust the Node.js worker pool.</li>
<li></li>
<li>Temporary unavailability of the backup storage should not require force restarting the XO VM.</li>
</ul>
<p dir="auto"><strong>Questions</strong></p>
<p dir="auto">Is this a known issue with Xen Orchestra's handling of unavailable NFS remotes?</p>
<p dir="auto">Is there a recommended configuration for NFS repositories hosted on NAS devices that may power off automatically?</p>
<p dir="auto">Does Xen Orchestra periodically access or keep NFS remotes mounted even when no backup jobs are running?</p>
<p dir="auto">Would increasing UV_THREADPOOL_SIZE be a reasonable mitigation, or could filesystem operations still exhaust the larger pool?</p>
<p dir="auto">Could the remote filesystem operations be isolated from the main xo-server process to prevent this scenario?</p>
<p dir="auto">I would prefer to retain NFS hard mounts for backup integrity rather than switching to soft.</p>
<p dir="auto">I can provide additional logs and diagnostics if useful.</p>
<p dir="auto">Thanks!</p>
]]></description><link>https://xcp-ng.org/forum/topic/12563/xen-orchestra-web-ui-becomes-unresponsive-when-an-nfs-backup-remote-goes-offline</link><generator>RSS for Node</generator><lastBuildDate>Thu, 05 Nov 2026 15:35:04 GMT</lastBuildDate><atom:link href="https://xcp-ng.org/forum/topic/12563.rss" rel="self" type="application/rss+xml"/><pubDate>Thu, 05 Nov 2026 13:47:10 GMT</pubDate><ttl>60</ttl><item><title><![CDATA[Reply to Xen Orchestra web UI becomes unresponsive when an NFS backup remote goes offline on Thu, 05 Nov 2026 15:21:14 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/frb94" aria-label="Profile: frb94">@<bdi>frb94</bdi></a></p>
<p dir="auto">Just leaving it on 24/7 <img src="https://xcp-ng.org/forum/assets/plugins/nodebb-plugin-emoji/emoji/android/1f61e.png?v=041a08e19dc" class="not-responsive emoji emoji-android emoji--disappointed" style="height:1.4em;width:auto;vertical-align:middle" title=":(" alt="😞" /> winter is here now or so we thought (its 75f outside today) .  If there is any update on this i would try again. but as it seems you are having the same issue i did then, it does not appear to be fixed.</p>
]]></description><link>https://xcp-ng.org/forum/post/109513</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/109513</guid><dc:creator><![CDATA[acebmxer]]></dc:creator><pubDate>Thu, 05 Nov 2026 15:21:14 GMT</pubDate></item><item><title><![CDATA[Reply to Xen Orchestra web UI becomes unresponsive when an NFS backup remote goes offline on Thu, 05 Nov 2026 15:11:15 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/acebmxer" aria-label="Profile: acebmxer">@<bdi>acebmxer</bdi></a> So what are you doing now? Given the manual, NFS is the way, but i can't have the remote available h24 and also i would like to have a functioning XO <img src="https://xcp-ng.org/forum/assets/plugins/nodebb-plugin-emoji/emoji/android/1f600.png?v=041a08e19dc" class="not-responsive emoji emoji-android emoji--grinning" style="height:1.4em;width:auto;vertical-align:middle" title=":grinning:" alt="😀" /></p>
]]></description><link>https://xcp-ng.org/forum/post/109512</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/109512</guid><dc:creator><![CDATA[frb94]]></dc:creator><pubDate>Thu, 05 Nov 2026 15:11:15 GMT</pubDate></item><item><title><![CDATA[Reply to Xen Orchestra web UI becomes unresponsive when an NFS backup remote goes offline on Thu, 05 Nov 2026 14:53:06 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/poddingue" aria-label="Profile: poddingue">@<bdi>poddingue</bdi></a> <a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/frb94" aria-label="Profile: frb94">@<bdi>frb94</bdi></a></p>
<p dir="auto">Yes if I recall (I may not have stated on the forum) that I still ended up with issues and just abounded the idea.  Not sure on the time line when they attempted a patch to when i stopped disabling the remote.</p>
]]></description><link>https://xcp-ng.org/forum/post/109510</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/109510</guid><dc:creator><![CDATA[acebmxer]]></dc:creator><pubDate>Thu, 05 Nov 2026 14:53:06 GMT</pubDate></item><item><title><![CDATA[Reply to Xen Orchestra web UI becomes unresponsive when an NFS backup remote goes offline on Thu, 05 Nov 2026 14:39:30 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/poddingue" aria-label="Profile: poddingue">@<bdi>poddingue</bdi></a></p>
<p dir="auto">Thank you very much for the answer.</p>
<p dir="auto">This is my idea for now, but i'll look into better ways to do this.</p>
<pre><code>*/5 * * * * /opt/scripts/enabledisableremote.sh
</code></pre>
<pre><code>#!/bin/bash

export PATH="/usr/local/bin:/usr/bin:/bin:$HOME/.npm-global/bin:$PATH"

REMOTE="9fc232df-4d0c-4a15-93b5-c0585c1dd9c9"
NAS="10.0.0.28"

# Stato attuale del Remote
CURRENT=$(/usr/local/bin/xo-cli remote.getAll --json | jq -r --arg id "$REMOTE" '.[] | select(.id == $id) | .enabled')

# Stato desiderato in base alla raggiungibilità del NAS
if ping -c 1 -W 2 "$NAS" &gt; /dev/null 2&gt;&amp;1; then
    DESIRED=true
else
    DESIRED=false
fi

# Cambia stato solo se necessario
if [[ "$CURRENT" == "true" || "$CURRENT" == "false" ]]; then
    if [[ "$CURRENT" != "$DESIRED" ]]; then
        /usr/local/bin/xo-cli remote.set id="$REMOTE" enabled="$DESIRED"
    fi
fi
</code></pre>
]]></description><link>https://xcp-ng.org/forum/post/109507</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/109507</guid><dc:creator><![CDATA[frb94]]></dc:creator><pubDate>Thu, 05 Nov 2026 14:39:30 GMT</pubDate></item><item><title><![CDATA[Reply to Xen Orchestra web UI becomes unresponsive when an NFS backup remote goes offline on Thu, 05 Nov 2026 14:33:19 GMT]]></title><description><![CDATA[<p dir="auto">This looks like the same thing <a class="plugin-mentions-user plugin-mentions-a" href="/forum/user/acebmxer" aria-label="Profile: acebmxer">@<bdi>acebmxer</bdi></a> ran into in July (<a href="https://xcp-ng.org/forum/topic/12339">https://xcp-ng.org/forum/topic/12339</a>), and your libuv reading matches what the XO devs wrote in <a href="https://github.com/vatesfr/xen-orchestra/pull/10587" target="_blank" rel="noopener noreferrer nofollow ugc">https://github.com/vatesfr/xen-orchestra/pull/10587</a>.</p>
<p dir="auto">As I read their write-up, with a hard mount every background operation on the dead remote (listing backups, retries) holds one of xo-server's file-access threads until the NAS comes back, and once they're all taken the web UI can't be served either.</p>
<p dir="auto">Olivier's advice back then (<a href="https://xcp-ng.org/forum/post/106632">https://xcp-ng.org/forum/post/106632</a>) was to disable the remote while the NAS is still up and only then power it off, and acebmxer scripted that on the Synology so the remote gets disabled at shutdown and enabled again at startup through the XO API.</p>
<p dir="auto">If XO is already stuck, XO 6.10.0 added <code>xo-server-db disable-remote &lt;id&gt;</code>, which you run while xo-server is stopped so the remote isn't mounted again on the next start; your November 1 build should have it if I read the dates right, but I haven't tried it myself.</p>
<p dir="auto">About <code>UV_THREADPOOL_SIZE</code> I'm only guessing, but since a hung NFS call never gives its thread back I'd expect a bigger pool to just take longer to fill, and the PR says they tried capping the threads one remote can use and dropped the idea for now. <a class="plugin-mentions-group plugin-mentions-a" href="/forum/groups/team-xo-backend" aria-label="Profile: Team-XO-Backend">@<bdi>Team-XO-Backend</bdi></a> will know better than me whether more is planned.</p>
]]></description><link>https://xcp-ng.org/forum/post/109504</link><guid isPermaLink="true">https://xcp-ng.org/forum/post/109504</guid><dc:creator><![CDATA[poddingue]]></dc:creator><pubDate>Thu, 05 Nov 2026 14:33:19 GMT</pubDate></item></channel></rss>