XOA 6.8 causes backup / replication failure

acebmxer

Same only remote pools failed. Main XOA pool completed with no errors.

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "compactMjml",
    "hideSuccessfulItems": true
  },
  "id": "1788224400022",
  "jobId": "c0e31730-7d45-4f3a-8733-8cea444b127d",
  "jobName": "Salem - Delta Backups",
  "message": "backup",
  "proxyId": "7840039e-a3cb-41ed-acee-3690f89a1426",
  "scheduleId": "ce3c801c-4384-4726-bda1-f6bae788e2f4",
  "start": 1788224400022,
  "status": "failure",
  "end": 1788224460395,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

acebmxer

updated to 6.8.1 still get backup error for remote pools...

Side not my v6 dashboard not loading

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "mjml",
    "hideSuccessfulItems": true
  },
  "id": "1788343797030",
  "jobId": "bb0adfc7-93b9-45c9-a90f-7aa327f95a05",
  "jobName": "Tilton - Delta Backups",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "249a8339-da2b-4031-b85d-dc3d9526efa5",
  "start": 1788343797030,
  "status": "failure",
  "end": 1788343857537,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

poddingue

There's an open issue that looks more or less like the same failure: vatesfr/xen-orchestra#10255, filed against a source build.
The reporter there says the proxy does get the command and finishes the export on the host, and that the failure is the UI dropping its connection rather than the backup dying.
I haven't reproduced that myself, so I don't know whether it's your case, but it seemed worth saying before more jobs get turned off.

One thing to watch if you go checking: xe task-list only shows tasks running right now.
On an idle 8.3 host here, it prints nothing at all, so running it after a job has already failed should give you an empty list, which may read like confirmation the backup never ran.
It'd have to be during the job, or check whether the replica actually landed on the destination SR.

Might be worth a mention to @Team-XO-Backend.

acebmxer

@poddingue

In the task log i see this...

Task: API call: backupNg.runJob

{
  "error": "Having multiple authentication methods is not supported, please choose one"
}

pierrebrunet

@acebmxer @flakpyro Hi, there was a patch release this morning for a few issues. Can you update your XO and check if it still happen?

acebmxer

@pierrebrunet

My post this morning after that update.

I see the same error in task view prior to the update but under the scheduled task as it was the actual backup job not me tring.

API call: schedule.runSequence

I am starting a fresh backup instead of retrying the last failed backup.

Edit - From fresh backup....

Also i have support ticket up about v6 dash board not loading and support tunnel open if you want to look. Ticket#7763637

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "mjml",
    "hideSuccessfulItems": true
  },
  "id": "1788347692784",
  "jobId": "bb0adfc7-93b9-45c9-a90f-7aa327f95a05",
  "jobName": "Tilton - Delta Backups",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "249a8339-da2b-4031-b85d-dc3d9526efa5",
  "start": 1788347692784,
  "status": "failure",
  "end": 1788347752961,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

pierrebrunet

@acebmxer Hi, I just checked your infra. It seems there was a connection error on the proxy around 2h ago: "Xen Orchestra Server could not be reached. Please check your connection."

Can you retry a backup whenever it is possible for you?

Edit: for this part, "error": "Having multiple authentication methods is not supported, please choose one"
It is because you are connecting with two connections in parallel (one with a token and one basic).

acebmxer

@pierrebrunet

v6 dahsboard is now connected and working.

Both Proxies are connected and vms are visible from the remote pools...

Backup job still fails. same error.

How and why does it have to many conections? To where XOA? to the Proxy? to the Backup remote?

I did recently have two seperate issues each different at both remote sites recenlty but the inital backup error seemed similar to the OP. And all backups worked until recent updates.

Edit -

This is soemthing new....

THis is still an active tranfer when the job reported failed....
Reran backup again. It does appear to be doing the backup but it stil fails after creating snapshots or around that point. Once the job fails the backup is still happening and health checks happening.

Screenshot 2026-09-02 100207.png

florent

@acebmxer Hi am testing an hypothesis where removing an old library introduced a implicit compression and buffering between the proxy and the how.

Can I provide a branch to test, or can I deploy a fix through a support tunnel ?

flakpyro

@florent Im having the same issue on 6.8.1 as well. XO and Proxies both updated.

The backup fails with "Fetch Failed" and the following appears in the log:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1788359113454",
  "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2",
  "jobName": "TestLab-14Day-VMVault03",
  "message": "backup",
  "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222",
  "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63",
  "start": 1788359113454,
  "status": "failure",
  "end": 1788359173436,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

However the backup appears to carry on the in the background after i receive the email of failure / the backup shows as failed on the backups page in XO5.

acebmxer

@florent Yes you may deploy a patch thought the support tunnel.

florent

@acebmxer can you give me the support tunnel number by chat ? patch is ready