XOA 6.8 causes backup / replication failure
-
There's an open issue that looks more or less like the same failure: vatesfr/xen-orchestra#10255, filed against a source build.
The reporter there says the proxy does get the command and finishes the export on the host, and that the failure is the UI dropping its connection rather than the backup dying.
I haven't reproduced that myself, so I don't know whether it's your case, but it seemed worth saying before more jobs get turned off.
One thing to watch if you go checking:
xe task-listonly shows tasks running right now.
On an idle 8.3 host here, it prints nothing at all, so running it after a job has already failed should give you an empty list, which may read like confirmation the backup never ran.
It'd have to be during the job, or check whether the replica actually landed on the destination SR.Might be worth a mention to @Team-XO-Backend.
-
In the task log i see this...
Task: API call: backupNg.runJob
{ "error": "Having multiple authentication methods is not supported, please choose one" } -
-
My post this morning after that update.
I see the same error in task view prior to the update but under the scheduled task as it was the actual backup job not me tring.
API call: schedule.runSequenceI am starting a fresh backup instead of retrying the last failed backup.
Edit - From fresh backup....
Also i have support ticket up about v6 dash board not loading and support tunnel open if you want to look. Ticket#7763637
{ "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "mjml", "hideSuccessfulItems": true }, "id": "1788347692784", "jobId": "bb0adfc7-93b9-45c9-a90f-7aa327f95a05", "jobName": "Tilton - Delta Backups", "message": "backup", "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df", "scheduleId": "249a8339-da2b-4031-b85d-dc3d9526efa5", "start": 1788347692784, "status": "failure", "end": 1788347752961, "result": { "message": "fetch failed", "name": "TypeError", "stack": "TypeError: fetch failed\n at node:internal/deps/undici/undici:14976:13\n at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n at processImmediate (node:internal/timers:484:21)\n at process.callbackTrampoline (node:internal/async_hooks:130:17)" } } -
@acebmxer Hi, I just checked your infra. It seems there was a connection error on the proxy around 2h ago: "Xen Orchestra Server could not be reached. Please check your connection."
Can you retry a backup whenever it is possible for you?
Edit: for this part, "error": "Having multiple authentication methods is not supported, please choose one"
It is because you are connecting with two connections in parallel (one with a token and one basic). -
v6 dahsboard is now connected and working.
Both Proxies are connected and vms are visible from the remote pools...
Backup job still fails. same error.
How and why does it have to many conections? To where XOA? to the Proxy? to the Backup remote?
I did recently have two seperate issues each different at both remote sites recenlty but the inital backup error seemed similar to the OP. And all backups worked until recent updates.
Edit -
This is soemthing new....
THis is still an active tranfer when the job reported failed....
Reran backup again. It does appear to be doing the backup but it stil fails after creating snapshots or around that point. Once the job fails the backup is still happening and health checks happening.
-
@acebmxer Hi am testing an hypothesis where removing an old library introduced a implicit compression and buffering between the proxy and the how.
Can I provide a branch to test, or can I deploy a fix through a support tunnel ?
-
@florent Im having the same issue on 6.8.1 as well. XO and Proxies both updated.
The backup fails with "Fetch Failed" and the following appears in the log:
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1788359113454", "jobId": "61c9c971-a2ca-453a-9ed6-934e7cf241b2", "jobName": "TestLab-14Day-VMVault03", "message": "backup", "proxyId": "3b26e911-1502-4686-ada4-7e4e33ad6222", "scheduleId": "23f4b532-feb3-4c23-8fcd-1fd318e9ce63", "start": 1788359113454, "status": "failure", "end": 1788359173436, "result": { "message": "fetch failed", "name": "TypeError", "stack": "TypeError: fetch failed\n at node:internal/deps/undici/undici:14976:13\n at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n at processImmediate (node:internal/timers:484:21)\n at process.callbackTrampoline (node:internal/async_hooks:130:17)" } }However the backup appears to carry on the in the background after i receive the email of failure / the backup shows as failed on the backups page in XO5.
-
@florent Yes you may deploy a patch thought the support tunnel.
-
@acebmxer can you give me the support tunnel number by chat ? patch is ready
-
fix is here https://github.com/vatesfr/xen-orchestra/pull/10346 , it works on @flakpyro proxies
the root cause is a timing condition between compression buffering data silently and timeout watching any stall connection
-
@florent I sent pm with support tunnel. Are you able to apply patch or will patch be pushed live soon?
-
Backups ran after patch applied...
1 pool had zero issues on backup 100% complete...
Other pool while it complete did have some errors...Started a second backup on same pool and giving same results.
{ "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "compactMjml", "hideSuccessfulItems": true }, "id": "1788428437610", "jobId": "c0e31730-7d45-4f3a-8733-8cea444b127d", "jobName": "Salem - Delta Backups", "message": "backup", "proxyId": "7840039e-a3cb-41ed-acee-3690f89a1426", "scheduleId": "ce3c801c-4384-4726-bda1-f6bae788e2f4", "start": 1788428437610, "status": "success", "tasks": [ { "id": "0mtlc477z-gdvpj68kva", "start": 1788428438927, "status": "success", "tasks": [ { "id": "0mtlc478f-guqsurb38qa", "start": 1788428438943, "status": "success", "end": 1788428440120, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mtlc48cx-c1y79k93vl", "start": 1788428440401, "status": "success", "end": 1788428442225, "result": "67677eec-494b-6ab6-0447-ac4dba8932a1", "message": "snapshot" }, { "id": "0mtlc49rm-ou0nslm2c7l", "start": 1788428442227, "status": "success", "tasks": [ { "id": "0mtlc724s-62ae1bzdupa", "start": 1788428572300, "status": "success", "end": 1788428932605, "result": { "size": 20385362944 }, "message": "transfer", "data": { "progress": 99 } }, { "id": "0mtlceupa-f476k7iq2u4", "start": 1788428935918, "status": "success", "tasks": [ { "id": "0mtlceutz-yztgqk0sv0c", "start": 1788428936087, "status": "success", "end": 1788429285986, "result": { "size": 20380123136, "id": "5fb5f6f1-709b-b149-4aa6-31514ef2c7f2" }, "message": "transfer" }, { "id": "0mtlcmctg-8z00dmmisds", "start": 1788429285988, "status": "success", "end": 1788429291289, "message": "vmstart" } ], "end": 1788429296722, "message": "health check" }, { "id": "0mtlcml5b-ywe7qa02f7q", "start": 1788429296783, "status": "success", "end": 1788429300723, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1788429300727, "message": "export", "data": { "id": "19270323-baaa-4a7b-bf1f-e152ae44f263", "isFull": false, "type": "remote" } } ], "warnings": [ { "message": "can't compute delta OpaqueRef:afcf0af9-64c9-845d-207c-789c9c5de78f from OpaqueRef:13ce290f-2194-280e-72c5-9e16271505df, fall back to a full" }, { "message": "can't connect through NBD, fall back to stream export" }, { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:afcf0af9-64c9-845d-207c-789c9c5de78f" }, "message": "Snapshot data has been deleted" } ], "end": 1788429300728, "message": "backup VM", "data": { "id": "0618e9fe-854e-7f01-1f22-6fb1667806d4", "type": "VM", "name_label": "Salem XO-CE", "progress": 0 } }, { "id": "0mtlc4783-s6m3pnf32pl", "start": 1788428438931, "status": "success", "tasks": [ { "id": "0mtlc478g-epd65nptnrn", "start": 1788428438944, "status": "success", "end": 1788428439927, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mtlc486t-mq755lvlxi", "start": 1788428440181, "status": "success", "end": 1788428443614, "result": "d52e8853-1102-8bcd-b588-adbe5b411a6b", "message": "snapshot" }, { "id": "0mtlc4au7-q4va6ntph6c", "start": 1788428443615, "status": "success", "tasks": [ { "id": "0mtlc9pax-xyje1fo25lf", "start": 1788428695641, "status": "success", "end": 1788429704445, "result": { "size": 44032329216 }, "message": "transfer", "data": { "progress": 100 } }, { "id": "0mtlcvdy1-pxmgi1rx7tk", "start": 1788429707353, "status": "success", "tasks": [ { "id": "0mtlcvdzc-uodtj65grrn", "start": 1788429707400, "status": "success", "end": 1788430481457, "result": { "size": 44021317632, "id": "ed4d4ce7-2685-a35a-b1e4-c54a1385364e" }, "message": "transfer" }, { "id": "0mtldbz96-dsndwbb93hv", "start": 1788430481466, "status": "success", "end": 1788430486658, "message": "vmstart" } ], "end": 1788430492516, "message": "health check" }, { "id": "0mtldc7vt-oopf5wcg74e", "start": 1788430492649, "status": "success", "end": 1788430501065, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1788430501070, "message": "export", "data": { "id": "19270323-baaa-4a7b-bf1f-e152ae44f263", "isFull": false, "type": "remote" } } ], "warnings": [ { "message": "can't compute delta OpaqueRef:82b55a23-b13e-df37-c9f7-11373a42c40b from OpaqueRef:9824e8f1-0199-885c-e69a-d9275cd62d2a, fall back to a full" }, { "message": "can't connect through NBD, fall back to stream export" }, { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:82b55a23-b13e-df37-c9f7-11373a42c40b" }, "message": "Snapshot data has been deleted" } ], "end": 1788430501071, "message": "backup VM", "data": { "id": "64476a1a-20f5-32a4-6352-47916b2162d8", "type": "VM", "name_label": "Salem UnifiOS", "progress": 0 } }, { "id": "0mtlcmo92-ezud8mmu26", "start": 1788429300806, "status": "success", "tasks": [ { "id": "0mtlcmoaf-ah3ohsdoy7r", "start": 1788429300855, "status": "success", "end": 1788429304315, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mtlcmrdk-xm266uwj1ni", "start": 1788429304856, "status": "success", "end": 1788429307794, "result": "dd0d0918-efdd-1a1d-ce94-85afbf0fd25c", "message": "snapshot" }, { "id": "0mtlcmtn7-clidkdhy2ht", "start": 1788429307795, "status": "success", "tasks": [ { "id": "0mtlcs7g8-3k59s9tshcf", "start": 1788429558968, "status": "success", "end": 1788430829213, "result": { "size": 43868711424 }, "message": "transfer", "data": { "progress": 100 } }, { "id": "0mtldjhbz-rfnh54dr3y", "start": 1788430831487, "status": "success", "tasks": [ { "id": "0mtldjhd9-g6rihy05ec7", "start": 1788430831533, "status": "success", "end": 1788431285549, "result": { "size": 43857739776, "id": "e3215b6b-9b5a-0d49-3392-491cc39c19e4" }, "message": "transfer" }, { "id": "0mtldt7ox-r81abwrax4", "start": 1788431285553, "status": "success", "end": 1788431290565, "message": "vmstart" } ], "end": 1788431296314, "message": "health check" }, { "id": "0mtldtg0h-8kvnbox5dg3", "start": 1788431296337, "status": "success", "end": 1788431297610, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1788431297611, "message": "export", "data": { "id": "19270323-baaa-4a7b-bf1f-e152ae44f263", "isFull": false, "type": "remote" } } ], "warnings": [ { "message": "can't compute delta OpaqueRef:4b3d9930-d795-03f8-c887-f8b7d2a95524 from OpaqueRef:894ead1a-dae6-f156-d8c7-c7f5ba2200ef, fall back to a full" }, { "message": "can't connect through NBD, fall back to stream export" }, { "message": "Backup fell back to a full" } ], "infos": [ { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:4b3d9930-d795-03f8-c887-f8b7d2a95524" }, "message": "Snapshot data has been deleted" } ], "end": 1788431297611, "message": "backup VM", "data": { "id": "ddf296c8-20a0-6aa9-0d74-ebc029d0d1b3", "type": "VM", "name_label": "Salem UnifiOS", "progress": 0 } } ], "end": 1788431297621, "infos": [ { "data": { "vms": [ "0618e9fe-854e-7f01-1f22-6fb1667806d4", "64476a1a-20f5-32a4-6352-47916b2162d8", "ddf296c8-20a0-6aa9-0d74-ebc029d0d1b3" ] }, "message": "vms" } ] } -
To sum up... It appears the patch has worked for me but has shed some light on miss configuration with the network design at this remote location, that I will need to take a real closer look at.
@florent thank you very much for your continue assistance with my issues.
-
@acebmxer we are far more explicit on the real usage of NBD, paing the way to mark a failed NBD connexion as an error in a few month
also working on a diagnostic tool to be more efficient on the checks
thank you all for your time
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login