XOA 6.8 causes backup / replication failure

[flakpyro]

I submitted a ticket about this as well but posting here for visibility

When running backup job that both backups up and replicates a VM the job will fail with the error "Error: fetch failed" both XOA and all proxies in the environment were upgraded. I also notice regular backups that do not have a replication running as well now show "NBT" instead of "NBD" when backing up. (Screenshot attached)

Screenshot 2026-08-31

Log attached:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1788197619873",
  "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2",
  "jobName": "TestLab-14Day-VMVault03",
  "message": "backup",
  "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222",
  "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63",
  "start": 1788197619873,
  "status": "failure",
  "end": 1788197679884,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

Rolling back to 6.7.1 results in the job functioning again.

[acebmxer]

Same only remote pools failed. Main XOA pool completed with no errors.

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "compactMjml",
    "hideSuccessfulItems": true
  },
  "id": "1788224400022",
  "jobId": "c0e31730-7d45-4f3a-8733-8cea444b127d",
  "jobName": "Salem - Delta Backups",
  "message": "backup",
  "proxyId": "7840039e-a3cb-41ed-acee-3690f89a1426",
  "scheduleId": "ce3c801c-4384-4726-bda1-f6bae788e2f4",
  "start": 1788224400022,
  "status": "failure",
  "end": 1788224460395,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

[acebmxer]

updated to 6.8.1 still get backup error for remote pools...

Side not my v6 dashboard not loading

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "mjml",
    "hideSuccessfulItems": true
  },
  "id": "1788343797030",
  "jobId": "bb0adfc7-93b9-45c9-a90f-7aa327f95a05",
  "jobName": "Tilton - Delta Backups",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "249a8339-da2b-4031-b85d-dc3d9526efa5",
  "start": 1788343797030,
  "status": "failure",
  "end": 1788343857537,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

[poddingue]

There's an open issue that looks more or less like the same failure: vatesfr/xen-orchestra#10255, filed against a source build.
The reporter there says the proxy does get the command and finishes the export on the host, and that the failure is the UI dropping its connection rather than the backup dying.
I haven't reproduced that myself, so I don't know whether it's your case, but it seemed worth saying before more jobs get turned off.

One thing to watch if you go checking: xe task-list only shows tasks running right now.
On an idle 8.3 host here, it prints nothing at all, so running it after a job has already failed should give you an empty list, which may read like confirmation the backup never ran.
It'd have to be during the job, or check whether the replica actually landed on the destination SR.

Might be worth a mention to @Team-XO-Backend.

[acebmxer]

@poddingue

In the task log i see this...

Task: API call: backupNg.runJob

{
  "error": "Having multiple authentication methods is not supported, please choose one"
}

[pierrebrunet]

@acebmxer @flakpyro Hi, there was a patch release this morning for a few issues. Can you update your XO and check if it still happen?

[acebmxer]

@pierrebrunet

My post this morning after that update.

I see the same error in task view prior to the update but under the scheduled task as it was the actual backup job not me tring.

API call: schedule.runSequence

I am starting a fresh backup instead of retrying the last failed backup.

Edit - From fresh backup....

Also i have support ticket up about v6 dash board not loading and support tunnel open if you want to look. Ticket#7763637

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "mjml",
    "hideSuccessfulItems": true
  },
  "id": "1788347692784",
  "jobId": "bb0adfc7-93b9-45c9-a90f-7aa327f95a05",
  "jobName": "Tilton - Delta Backups",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "249a8339-da2b-4031-b85d-dc3d9526efa5",
  "start": 1788347692784,
  "status": "failure",
  "end": 1788347752961,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}