XOA 6.8 causes backup / replication failure

pierrebrunet

@acebmxer @flakpyro Hi, there was a patch release this morning for a few issues. Can you update your XO and check if it still happen?

acebmxer

@pierrebrunet

My post this morning after that update.

I see the same error in task view prior to the update but under the scheduled task as it was the actual backup job not me tring.

API call: schedule.runSequence

I am starting a fresh backup instead of retrying the last failed backup.

Edit - From fresh backup....

Also i have support ticket up about v6 dash board not loading and support tunnel open if you want to look. Ticket#7763637

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "mjml",
    "hideSuccessfulItems": true
  },
  "id": "1788347692784",
  "jobId": "bb0adfc7-93b9-45c9-a90f-7aa327f95a05",
  "jobName": "Tilton - Delta Backups",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "249a8339-da2b-4031-b85d-dc3d9526efa5",
  "start": 1788347692784,
  "status": "failure",
  "end": 1788347752961,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

pierrebrunet

@acebmxer Hi, I just checked your infra. It seems there was a connection error on the proxy around 2h ago: "Xen Orchestra Server could not be reached. Please check your connection."

Can you retry a backup whenever it is possible for you?

Edit: for this part, "error": "Having multiple authentication methods is not supported, please choose one"
It is because you are connecting with two connections in parallel (one with a token and one basic).

acebmxer

@pierrebrunet

v6 dahsboard is now connected and working.

Both Proxies are connected and vms are visible from the remote pools...

Backup job still fails. same error.

How and why does it have to many conections? To where XOA? to the Proxy? to the Backup remote?

I did recently have two seperate issues each different at both remote sites recenlty but the inital backup error seemed similar to the OP. And all backups worked until recent updates.

Edit -

This is soemthing new....

THis is still an active tranfer when the job reported failed....
Reran backup again. It does appear to be doing the backup but it stil fails after creating snapshots or around that point. Once the job fails the backup is still happening and health checks happening.

Screenshot 2026-09-02 100207.png

florent

@acebmxer Hi am testing an hypothesis where removing an old library introduced a implicit compression and buffering between the proxy and the how.

Can I provide a branch to test, or can I deploy a fix through a support tunnel ?

flakpyro

@florent Im having the same issue on 6.8.1 as well. XO and Proxies both updated.

The backup fails with "Fetch Failed" and the following appears in the log:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1788359113454",
  "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2",
  "jobName": "TestLab-14Day-VMVault03",
  "message": "backup",
  "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222",
  "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63",
  "start": 1788359113454,
  "status": "failure",
  "end": 1788359173436,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

However the backup appears to carry on the in the background after i receive the email of failure / the backup shows as failed on the backups page in XO5.

acebmxer

@florent Yes you may deploy a patch thought the support tunnel.

florent

@acebmxer can you give me the support tunnel number by chat ? patch is ready

florent

fix is here https://github.com/vatesfr/xen-orchestra/pull/10346 , it works on @flakpyro proxies

the root cause is a timing condition between compression buffering data silently and timeout watching any stall connection

acebmxer

@florent I sent pm with support tunnel. Are you able to apply patch or will patch be pushed live soon?