@poddingue This was fixed a number of weeks ago via a ticket. I believe @florent may know the exact fix.

Doing some more testing it doesn't seem to matter how many pools i have in the metadata backup job, it will hang and time out after 20 mins. I am also seeing this on my home install with XO from sources after updating to the latest commit.

@Danp I would normally do this and report back however i am running a custom patch from @florent at the moment that he deployed to fix the "Fetch Failed" backup proxy issue. Once 6.8.2 has been released which that patch i should be able to switch between versions and test this!

Since upgrading to XOA 6.8 i'm experiencing XOA pool metadata backup failures. The task will run and a number of pools will successfully backup however a random set of pools will stall until the job eventually fails after 20 mins. The error is "Body Timeout Error". I have tried restarting the toolstack on the hosts as well as restarting XOA. It is not always the same hosts each time i retry either.

I would attach the log but it is quite long as it seems to show successes as well as failures

"id": "0mtln3sci-stmgt9yih8i",
      "start": 1788446895426,
      "status": "failure",
      "tasks": [
        {
          "id": "0mtln3sde-8lp6c7ei6un",
          "start": 1788446895458,
          "status": "failure",
          "end": 1788448100196,
          "result": {
            "name": "BodyTimeoutError",
            "code": "UND_ERR_BODY_TIMEOUT",
            "message": "Body Timeout Error",
            "stack": "BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/dispatcher/client-h1.js:703:28)\n    at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:585:17)\n    at processTimers (node:internal/timers:521:7)"
          },
          "message": "Starting metadata backup for the pool (19bda980-70ec-bada-5d9d-6f35c8a2122b) for the remote (5b9abfa7-f5f1-4e80-bfde-cabe82864988). (5220a296-765d-4023-acc7-70d4ae66fe1e)",
          "data": {
            "id": "5b9abfa7-f5f1-4e80-bfde-cabe82864988",
            "type": "remote",
            "progress": 0
          }

@florent Im having the same issue on 6.8.1 as well. XO and Proxies both updated.

The backup fails with "Fetch Failed" and the following appears in the log:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1788359113454",
  "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2",
  "jobName": "TestLab-14Day-VMVault03",
  "message": "backup",
  "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222",
  "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63",
  "start": 1788359113454,
  "status": "failure",
  "end": 1788359173436,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

However the backup appears to carry on the in the background after i receive the email of failure / the backup shows as failed on the backups page in XO5.

I submitted a ticket about this as well but posting here for visibility

When running backup job that both backups up and replicates a VM the job will fail with the error "Error: fetch failed" both XOA and all proxies in the environment were upgraded. I also notice regular backups that do not have a replication running as well now show "NBT" instead of "NBD" when backing up. (Screenshot attached)

Screenshot 2026-08-31

Log attached:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1788197619873",
  "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2",
  "jobName": "TestLab-14Day-VMVault03",
  "message": "backup",
  "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222",
  "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63",
  "start": 1788197619873,
  "status": "failure",
  "end": 1788197679884,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

Rolling back to 6.7.1 results in the job functioning again.

Installed on my usual hosts without issues. I am only making limited use of QCOW2 however.

@semarie If that's the case then the update will likely go smoothly once its released to the stable repos and users use Xen Orchestra to run a rolling pool update which follows that workflow. Since this was a test pool and using the test repos i ran the update via yum without evacuating the host first.

@semarie In my case the pool i tested these updates on was a 2 host pool, i patched each host using yum and then kicked off a rolling pool reboot. So the VMs would have been resident on the pool master since it is the first rebooted when running a rolling pool reboot. I was unable to manually migrate the VMs resident on this host as well, attempting to do so would show the same error.

In my case though restarting the tool stack on both hosts allowed migrations to complete and a rolling pool reboot to run successfully.

Installed on my usual test hosts. Stand alone hosts all rebooted fine.

My test pool of 2 servers failed to rolling pool reboot until i restarted the tool stack on both hosts. Error log is attached.
2026-08-14T17_30_10.463Z - XO.txt

@bleader I am running into the same issue on a pair of Lenovo SR630 V4s with Intel Xeon 6530P CPUs

[08:23 xcpng-dr-2 ~]# xenpm get-cpufreq-para
[CPU0] failed to get cpufreq parameter

Playing with different workload profiles in the bios but so far the server seems to pretty aggressively clock itself down even while CPU intensive tasks like backups are running. I am of course not able to run something like xenpm set-scaling-governor performance either.

EDIT: The solution here is to get "Power/Performance Bias:" to OS Controlled in the servers BIOS. This restores functionality.

@andriy.sultanov

Running it now shows no error! And no output so looks like i have no issues!

@andriy.sultanov

Thanks

I ran that second script on the master and it did indeed find entries:

./snapshot-fixer.py rewrite
INFO:root:Check HA...
INFO:root:Shutting down xapi...
INFO:root:Regenerating database...
INFO:root:The VM d6befe36-fea7-04f3-25f1-feada684702b has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM e013d9ed-63b1-9c9f-a228-d48a6047688c has Ref:89 as its "snapshot_of" value, changing to null.
INFO:root:The VM 060df7ee-8af4-4c93-8265-bfa3024ccacd has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM da7b6ff6-d722-2120-7cc4-a1e0038a216e has Ref:307 as its "snapshot_of" value, changing to null.
INFO:root:The VM 4178b057-efee-427e-bcad-67323419ac7a has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM 8d624e49-ca99-4af6-b17c-38cad0646194 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM dc72da43-f144-3343-5ee5-a0dbffd209f6 has Ref:251 as its "snapshot_of" value, changing to null.
INFO:root:The VM e8c6376b-49e1-422b-551a-35420f70af2a has Ref:8 as its "snapshot_of" value, changing to null.
INFO:root:The VM 25eb14ab-e943-0d18-2137-c681e1191346 has Ref:11 as its "snapshot_of" value, changing to null.
INFO:root:The VM 6f819166-b008-4a03-a692-fcbd75018be9 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9d1a57e8-3531-4b55-9682-f775215866df has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 0ac88f7c-d79b-281e-cf1f-a2392ba594ed has Ref:29 as its "snapshot_of" value, changing to null.
INFO:root:The VM 1f3541c4-8c81-4959-9cbe-edbec9481220 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:Writing database to /var/lib/xcp/state.db
INFO:root:Starting up xapi...

after which it shows clean now:

./snapshot-fixer.py dry-run
INFO:root:Regenerating database...

However i am still getting the same error when trying to run leaked_vbs:

./leaked_vbds 
Traceback (most recent call last):
  File "./leaked_vbds", line 23, in <module>
    snap = xapi.VM.get_parent(snap)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 317, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 198, in xenapi_request
    result = _parse_result(getattr(self, methodname)(*full_params))
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 292, in _parse_result
    raise Failure(result['ErrorDescription'])
XenAPI.Failure: ['HANDLE_INVALID', 'VM', 'Ref:29']

@rzr

Is there something specific needed for the leaked_vbds script?

When i run it i get:

[09:37 xcpng-prd-01 ~]# ./leaked_vbds 
Traceback (most recent call last):
  File "./leaked_vbds", line 23, in <module>
    snap = xapi.VM.get_parent(snap)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 317, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 198, in xenapi_request
    result = _parse_result(getattr(self, methodname)(*full_params))
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 292, in _parse_result
    raise Failure(result['ErrorDescription'])
XenAPI.Failure: ['HANDLE_INVALID', 'VM', 'Ref:29']

No issues to report so far.

No issues on the same test hosts i used for the last 2 batches of updates.

Installed on my usual test hosts. No issues so far.

Installed on my usual hosts, one of which has an E810 and used the ICE driver, no issues so far however i am not using LACP bonding on that host.

Installed on my usual batch of test hosts, no issues so far.

Installed on my usual batch of test hosts, so far so good.