Installed on my usual test hosts once again. This time i made sure to test backup job behaviour multiple times across the same two pools that gave me trouble last time. All backup job runs (4 runs on one pool, 2 on another) have been successful.

@jr-m4 I had this issue on one of the pools i tested on as well and was able to fix the issue by following the fix here:
https://xcp-ng.org/forum/topic/10603/sr_backend_failure_46-the-vdi-is-not-available-not-detached-cleanly/3

@rzr I was worried about this because of XAPI. Luckily i have a host thats not currently running any VMs so i tried it there first hoping it was a small enough change that i could get away with it. Good to know i could have just disabled NBD for now though!

@acebmxer I was able to run successfully one time after multiple failures. I don't think its consistently reliable. Interesting that disabling NBD helped though. I didn't try that before reverting.

@acebmxer

I was able to revert by running

 yum history list

Finding the transaction from todays updates then running

 yum history undo <id>

I did this on both pools, doing the slaves first then the master. (Reverse order of updating) and so far everything is back to normal.

Installed on my usual test hosts.

Backups seem to have major ussues for me, much less stable with these updates. On two different test pools i ran into a similar issue. These pools are different hardware, one pool uses CBT for backups, one does not.

Pool 1:
Started a backup after running the updates, 2 VMs in the job hung on first run requiring restarting XO and the hosts tool stack. When trying to clear the leftover snapshots one VM got stuck with "VBD.unplug_force", restarting the toolstack didn't help with this i had to reboot the entire host to remove it. Second attempt backups ran ok.

Third attempt, VM hung again, same story as above, had to reboot the whole host to clear the "VBD.unplug_force"

Pool 2:

Backup job completed however again two VMs got stuck at the end with "VBD.unplug_force" and "VDIs attached to Control Domain"
On one VM the backup showed "Error: operation timed out" despite it otherwise finishing.

Is there a way to revert these updates for now? They are test hosts but i do like backing up the test VMs on them.

Installed on my usual test hosts without any issues so far.

@dinhngtu We did initially run into this issue a number of years ago and were able to fix it by following this kb article at the time:
https://support.citrix.com/external/article?articleUrl=CTX692813-xenserver-potential-performance-issues-on-imported-windows-vms&language=en_US

Checking s few random migrated windows VMs and it appears these flags are still set as they should be.

@dinhngtu Thanks for clarifying. I can confirm our windows VMs that were migrated with V2V with "Other Install media" do show having this enabled.

I will deploy to my usual batch of test hosts and update this post with my results.

@gleh said:

Disable Viridian in the "Other install media" template by default. As a reminder, Windows VMs must use Windows templates.

Working on testing this update out however this change has me a bit concerned. Will existing Windows VMs using "Other install media" be affected by this? Or is this just for VMs created going forward? Back in 2024 when we migrated from VMware using V2V all Windows VMs we migrated were created with "Other Install Media" as I'm guessing that's just how the migration process worked at the time.

@poddingue This was fixed a number of weeks ago via a ticket. I believe @florent may know the exact fix.

Doing some more testing it doesn't seem to matter how many pools i have in the metadata backup job, it will hang and time out after 20 mins. I am also seeing this on my home install with XO from sources after updating to the latest commit.

@Danp I would normally do this and report back however i am running a custom patch from @florent at the moment that he deployed to fix the "Fetch Failed" backup proxy issue. Once 6.8.2 has been released which that patch i should be able to switch between versions and test this!

Since upgrading to XOA 6.8 i'm experiencing XOA pool metadata backup failures. The task will run and a number of pools will successfully backup however a random set of pools will stall until the job eventually fails after 20 mins. The error is "Body Timeout Error". I have tried restarting the toolstack on the hosts as well as restarting XOA. It is not always the same hosts each time i retry either.

I would attach the log but it is quite long as it seems to show successes as well as failures

"id": "0mtln3sci-stmgt9yih8i",
      "start": 1788446895426,
      "status": "failure",
      "tasks": [
        {
          "id": "0mtln3sde-8lp6c7ei6un",
          "start": 1788446895458,
          "status": "failure",
          "end": 1788448100196,
          "result": {
            "name": "BodyTimeoutError",
            "code": "UND_ERR_BODY_TIMEOUT",
            "message": "Body Timeout Error",
            "stack": "BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/dispatcher/client-h1.js:703:28)\n    at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:585:17)\n    at processTimers (node:internal/timers:521:7)"
          },
          "message": "Starting metadata backup for the pool (19bda980-70ec-bada-5d9d-6f35c8a2122b) for the remote (5b9abfa7-f5f1-4e80-bfde-cabe82864988). (5220a296-765d-4023-acc7-70d4ae66fe1e)",
          "data": {
            "id": "5b9abfa7-f5f1-4e80-bfde-cabe82864988",
            "type": "remote",
            "progress": 0
          }

@florent Im having the same issue on 6.8.1 as well. XO and Proxies both updated.

The backup fails with "Fetch Failed" and the following appears in the log:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1788359113454",
  "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2",
  "jobName": "TestLab-14Day-VMVault03",
  "message": "backup",
  "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222",
  "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63",
  "start": 1788359113454,
  "status": "failure",
  "end": 1788359173436,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

However the backup appears to carry on the in the background after i receive the email of failure / the backup shows as failed on the backups page in XO5.

I submitted a ticket about this as well but posting here for visibility

When running backup job that both backups up and replicates a VM the job will fail with the error "Error: fetch failed" both XOA and all proxies in the environment were upgraded. I also notice regular backups that do not have a replication running as well now show "NBT" instead of "NBD" when backing up. (Screenshot attached)

Screenshot 2026-08-31

Log attached:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1788197619873",
  "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2",
  "jobName": "TestLab-14Day-VMVault03",
  "message": "backup",
  "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222",
  "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63",
  "start": 1788197619873,
  "status": "failure",
  "end": 1788197679884,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

Rolling back to 6.7.1 results in the job functioning again.

Installed on my usual hosts without issues. I am only making limited use of QCOW2 however.

@semarie If that's the case then the update will likely go smoothly once its released to the stable repos and users use Xen Orchestra to run a rolling pool update which follows that workflow. Since this was a test pool and using the test repos i ran the update via yum without evacuating the host first.

@semarie In my case the pool i tested these updates on was a 2 host pool, i patched each host using yum and then kicked off a rolling pool reboot. So the VMs would have been resident on the pool master since it is the first rebooted when running a rolling pool reboot. I was unable to manually migrate the VMs resident on this host as well, attempting to do so would show the same error.

In my case though restarting the tool stack on both hosts allowed migrations to complete and a rolling pool reboot to run successfully.

Installed on my usual test hosts. Stand alone hosts all rebooted fine.

My test pool of 2 servers failed to rolling pool reboot until i restarted the tool stack on both hosts. Error log is attached.
2026-08-14T17_30_10.463Z - XO.txt