flakpyro

Doing some more testing it doesn't seem to matter how many pools i have in the metadata backup job, it will hang and time out after 20 mins. I am also seeing this on my home install with XO from sources after updating to the latest commit.

@Danp I would normally do this and report back however i am running a custom patch from @florent at the moment that he deployed to fix the "Fetch Failed" backup proxy issue. Once 6.8.2 has been released which that patch i should be able to switch between versions and test this!

Since upgrading to XOA 6.8 i'm experiencing XOA pool metadata backup failures. The task will run and a number of pools will successfully backup however a random set of pools will stall until the job eventually fails after 20 mins. The error is "Body Timeout Error". I have tried restarting the toolstack on the hosts as well as restarting XOA. It is not always the same hosts each time i retry either.

I would attach the log but it is quite long as it seems to show successes as well as failures

"id": "0mtln3sci-stmgt9yih8i",
      "start": 1788446895426,
      "status": "failure",
      "tasks": [
        {
          "id": "0mtln3sde-8lp6c7ei6un",
          "start": 1788446895458,
          "status": "failure",
          "end": 1788448100196,
          "result": {
            "name": "BodyTimeoutError",
            "code": "UND_ERR_BODY_TIMEOUT",
            "message": "Body Timeout Error",
            "stack": "BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/dispatcher/client-h1.js:703:28)\n    at Timeout.onTick [as _onTimeout] (/usr/local/lib/node_modules/xo-server/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:585:17)\n    at processTimers (node:internal/timers:521:7)"
          },
          "message": "Starting metadata backup for the pool (19bda980-70ec-bada-5d9d-6f35c8a2122b) for the remote (5b9abfa7-f5f1-4e80-bfde-cabe82864988). (5220a296-765d-4023-acc7-70d4ae66fe1e)",
          "data": {
            "id": "5b9abfa7-f5f1-4e80-bfde-cabe82864988",
            "type": "remote",
            "progress": 0
          }

@florent Im having the same issue on 6.8.1 as well. XO and Proxies both updated.

The backup fails with "Fetch Failed" and the following appears in the log:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1788359113454",
  "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2",
  "jobName": "TestLab-14Day-VMVault03",
  "message": "backup",
  "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222",
  "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63",
  "start": 1788359113454,
  "status": "failure",
  "end": 1788359173436,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

However the backup appears to carry on the in the background after i receive the email of failure / the backup shows as failed on the backups page in XO5.

I submitted a ticket about this as well but posting here for visibility

When running backup job that both backups up and replicates a VM the job will fail with the error "Error: fetch failed" both XOA and all proxies in the environment were upgraded. I also notice regular backups that do not have a replication running as well now show "NBT" instead of "NBD" when backing up. (Screenshot attached)

Screenshot 2026-08-31

Log attached:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1788197619873",
  "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2",
  "jobName": "TestLab-14Day-VMVault03",
  "message": "backup",
  "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222",
  "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63",
  "start": 1788197619873,
  "status": "failure",
  "end": 1788197679884,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

Rolling back to 6.7.1 results in the job functioning again.

Installed on my usual hosts without issues. I am only making limited use of QCOW2 however.

@semarie If that's the case then the update will likely go smoothly once its released to the stable repos and users use Xen Orchestra to run a rolling pool update which follows that workflow. Since this was a test pool and using the test repos i ran the update via yum without evacuating the host first.

@semarie In my case the pool i tested these updates on was a 2 host pool, i patched each host using yum and then kicked off a rolling pool reboot. So the VMs would have been resident on the pool master since it is the first rebooted when running a rolling pool reboot. I was unable to manually migrate the VMs resident on this host as well, attempting to do so would show the same error.

In my case though restarting the tool stack on both hosts allowed migrations to complete and a rolling pool reboot to run successfully.

Installed on my usual test hosts. Stand alone hosts all rebooted fine.

My test pool of 2 servers failed to rolling pool reboot until i restarted the tool stack on both hosts. Error log is attached.
2026-08-14T17_30_10.463Z - XO.txt

@bleader I am running into the same issue on a pair of Lenovo SR630 V4s with Intel Xeon 6530P CPUs

[08:23 xcpng-dr-2 ~]# xenpm get-cpufreq-para
[CPU0] failed to get cpufreq parameter

Playing with different workload profiles in the bios but so far the server seems to pretty aggressively clock itself down even while CPU intensive tasks like backups are running. I am of course not able to run something like xenpm set-scaling-governor performance either.

EDIT: The solution here is to get "Power/Performance Bias:" to OS Controlled in the servers BIOS. This restores functionality.