XO error/warning: Clean VM directory. unhandled error while checking alias.
-
@julienXOvates Correct, latest XO branch is the problem. It was fine before...
{ "data": { "mode": "delta", "reportWhen": "never" }, "id": "1780192695536", "jobId": "d6c0a656-62c5-4c39-a57a-f246b39f1cef", "jobName": "minio-test", "message": "backup", "scheduleId": "bd4ef436-fd85-4f16-bf9e-71d1d0c8586f", "start": 1780192695536, "status": "success", "tasks": [ { "id": "0mpt4ron3-juzrilzvn0s", "start": 1780192697727, "status": "success", "tasks": [ { "id": "0mpt4ron9-pxah6gpi19", "start": 1780192697733, "status": "success", "end": 1780192698044, "result": { "merge": false, "size": 0 }, "message": "clean-vm" }, { "id": "0mpt4rpwm-ujzdf5a8ad", "start": 1780192699366, "status": "success", "end": 1780192700817, "result": "0d3f6e5e-fd58-7368-7d54-c892809927b6", "message": "snapshot" }, { "id": "0mpt4rr0x-sfehspx3ey", "start": 1780192700817, "status": "success", "tasks": [ { "id": "0mpt4rsdg-t05980lobz", "start": 1780192702564, "status": "success", "end": 1780192702622, "result": { "size": 43008 }, "message": "transfer" }, { "id": "0mpt4rts7-5p0b0d1r94y", "start": 1780192704391, "status": "success", "tasks": [ { "id": "0mpt4rtvx-0rtl320os8g", "start": 1780192704525, "status": "success", "end": 1780192704782, "message": "merge" } ], "warnings": [ { "data": { "alias": "/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260531T015641Z.alias.vhd", "err": { "$fault": "client", "$metadata": { "httpStatusCode": 404, "requestId": "18B483D548936DAB", "extendedRequestId": "a18350611414fa9366e109699727e695e309120afc9803fcaff78a199f9bc4d6", "attempts": 1, "totalRetryDelay": 0 }, "name": "NotFound", "message": "UnknownError" } }, "message": "unhandled error while checking alias" } ], "end": 1780192704964, "result": { "merge": true, "size": 8235471872 }, "message": "clean-vm" } ], "end": 1780192704964, "message": "export", "data": { "id": "9890e0c4-ba3a-4810-8245-a49fdf16b16e", "isFull": false, "type": "remote" } } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:50cca589-9c8c-478c-dac8-56debab464bb" }, "message": "Snapshot data has been deleted" } ], "end": 1780192704964, "message": "backup VM", "data": { "id": "a081f208-4e40-2ca5-c68e-86106af5a8ef", "type": "VM", "name_label": "Utest" } } ], "end": 1780192704964, "infos": [ { "data": { "vms": [ "a081f208-4e40-2ca5-c68e-86106af5a8ef" ] }, "message": "vms" } ] }May 30 21:58:24 xo2 xo-server[1180]: 2026-05-31T01:58:24.525Z xo:backups:MixinBackupWriter INFO Disk chain needs merging { count: 1 } May 30 21:58:24 xo2 xo-server[1180]: 2026-05-31T01:58:24.526Z xo:backups:MixinBackupWriter INFO merging disk chain { May 30 21:58:24 xo2 xo-server[1180]: chain: [ May 30 21:58:24 xo2 xo-server[1180]: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260531T015641Z.alias.vhd', May 30 21:58:24 xo2 xo-server[1180]: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260531T015759Z.alias.vhd' May 30 21:58:24 xo2 xo-server[1180]: ] May 30 21:58:24 xo2 xo-server[1180]: } May 30 21:58:24 xo2 xo-server[1180]: 2026-05-31T01:58:24.788Z xo:backups:MixinBackupWriter WARN unhandled error while checking alias { May 30 21:58:24 xo2 xo-server[1180]: alias: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260531T015641Z.alias.vhd', May 30 21:58:24 xo2 xo-server[1180]: err: NotFound: UnknownError May 30 21:58:24 xo2 xo-server[1180]: at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202605291150/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1850:27) May 30 21:58:24 xo2 xo-server[1180]: at process.processTicksAndRejections (node:internal/process/task_queues:104:5) May 30 21:58:24 xo2 xo-server[1180]: at async S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202605291150/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:424:13) May 30 21:58:24 xo2 xo-server[1180]: at async /opt/xo/xo-builds/xen-orchestra-202605291150/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:27:24 May 30 21:58:24 xo2 xo-server[1180]: at async /opt/xo/xo-builds/xen-orchestra-202605291150/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:387:20 May 30 21:58:24 xo2 xo-server[1180]: at async /opt/xo/xo-builds/xen-orchestra-202605291150/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50 May 30 21:58:24 xo2 xo-server[1180]: at async /opt/xo/xo-builds/xen-orchestra-202605291150/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:64:28 May 30 21:58:24 xo2 xo-server[1180]: at async /opt/xo/xo-builds/xen-orchestra-202605291150/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:91:20 May 30 21:58:24 xo2 xo-server[1180]: at async /opt/xo/xo-builds/xen-orchestra-202605291150/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:5:26 May 30 21:58:24 xo2 xo-server[1180]: at async S3Handler._getSize (/opt/xo/xo-builds/xen-orchestra-202605291150/@xen-orchestra/fs/dist/s3.js:304:20) { May 30 21:58:24 xo2 xo-server[1180]: '$fault': 'client', May 30 21:58:24 xo2 xo-server[1180]: '$retryable': undefined, May 30 21:58:24 xo2 xo-server[1180]: '$metadata': { May 30 21:58:24 xo2 xo-server[1180]: httpStatusCode: 404, May 30 21:58:24 xo2 xo-server[1180]: requestId: '18B483D548936DAB', May 30 21:58:24 xo2 xo-server[1180]: extendedRequestId: 'a18350611414fa9366e109699727e695e309120afc9803fcaff78a199f9bc4d6', May 30 21:58:24 xo2 xo-server[1180]: cfId: undefined, May 30 21:58:24 xo2 xo-server[1180]: attempts: 1, May 30 21:58:24 xo2 xo-server[1180]: totalRetryDelay: 0 May 30 21:58:24 xo2 xo-server[1180]: } May 30 21:58:24 xo2 xo-server[1180]: } May 30 21:58:24 xo2 xo-server[1180]: } May 30 21:58:24 xo2 xo-server[1180]: 2026-05-31T01:58:24.964Z xo:backups:worker INFO backup has ended -
@Andrew @fagnermoraes Hi,
A fix is ongoing here: https://github.com/vatesfr/xen-orchestra/pull/9925@fagnermoraes can you provide your logs please?
open fix: move try/catch inside initializations #9925
-
This is just a backup test that shows the same problem as in production.
The file is uploaded to the folder, then the verification error and file corruption alert are displayed.
Since the routine retains 3 copies, it ended up deleting the files that were working, and with the error I was left without backups.
I had to export the VMs to have at least one copy.
All packages are up to date and it is on the latest commit version.{ "data": { "mode": "full", "reportWhen": "never" }, "id": "1780237144366", "jobId": "90c1a9ac-be57-4084-a009-abea039c324e", "jobName": "Teste-BKP-RNP", "message": "backup", "scheduleId": "b603f95a-c230-4bb8-baf1-05d7b1906ae1", "start": 1780237144366, "status": "success", "tasks": [ { "id": "0mptv8soh-y67yux15fk", "start": 1780237166129, "status": "success", "tasks": [ { "id": "0mptv8u2z-z07gqebx7e8", "start": 1780237167947, "status": "success", "end": 1780237245751, "result": "7e51389c-fece-1b94-2a69-e097684f90d5", "message": "snapshot" }, { "id": "0mptvai62-dj05w24lbzp", "start": 1780237245818, "status": "success", "tasks": [ { "id": "0mptvai6g-wo1kemu62r", "start": 1780237245832, "status": "success", "end": 1780239711489, "result": { "size": 125955945472 }, "message": "transfer" } ], "end": 1780239711655, "message": "export", "data": { "id": "4e24de58-14a2-4cff-ae46-985c30831302", "type": "remote", "isFull": true } }, { "id": "0mptwroqq-oooeciwkcqg", "start": 1780239727106, "status": "success", "warnings": [ { "data": { "error": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": true, "expected": false, "operator": "strictEqual", "diff": "simple" } }, "message": "Issue while checking XVA" }, { "data": { "error": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": true, "expected": false, "operator": "strictEqual", "diff": "simple" } }, "message": "Checksum file not valid, not blocking" }, { "data": { "path": "/xo-vm-backups/695455b5-8e41-661f-d647-409620ee434e/20260501T140136Z.xva" }, "message": "XVA might be broken" }, { "data": { "error": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": true, "expected": false, "operator": "strictEqual", "diff": "simple" } }, "message": "Issue while checking XVA" }, { "data": { "error": { "generatedMessage": false, "code": "ERR_ASSERTION", "actual": true, "expected": false, "operator": "strictEqual", "diff": "simple" } }, "message": "Checksum file not valid, not blocking" }, { "data": { "path": "/xo-vm-backups/695455b5-8e41-661f-d647-409620ee434e/20260531T142045Z.xva" }, "message": "XVA might be broken" }, { "data": { "files": [ "20260501T140136Z.json", "20260501T140136Z.xva", "20260501T140136Z.xva.checksum" ] }, "message": "This files may be corrupted but not yet to be removed" }, { "data": { "files": [ "20260531T142045Z.json", "20260531T142045Z.xva", "20260531T142045Z.xva.checksum" ] }, "message": "This files may be corrupted but not yet to be removed" } ], "end": 1780239727137, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1780239727183, "message": "backup VM", "data": { "id": "695455b5-8e41-661f-d647-409620ee434e", "type": "VM", "name_label": "vm-giz-01" } } ], "end": 1780239727186, "infos": [ { "data": { "vms": [ "695455b5-8e41-661f-d647-409620ee434e" ] }, "message": "vms" } ] } -
@pierrebrunet
Production backup.
backup.txt -
@FagnerMoraes There is now a fix in latest channel now, but you will need to run backup again.
-
@pierrebrunet I updated and did a backup test with a smaller VM. It worked; it's no longer deleting the VMs backup. It still issues an alert about possible file corruption, but I restored it using XOCE and it restored normally.
Thank you for your help. -
@pierrebrunet I still see the same error....
Clean VM directory unhandled error while checking alias -
An issue was identified in backup jobs running in incremental with commit 100f113 or release 6.5.0. If one of your incremental backup chains was corrupted, it lead to a cleaning of vdis folder resulting into backup lose.
Please update your source at least to 4552fb9 or your XOA to 6.5.1.
If you encountered issues about a warning with aliases, you are concerned. You need to run your backup jobs again to have your backups back on track.
-
@Andrew The warning may still appear indeed but I think it is another issue or maybe a good warning to check your backups. Can you provide the new logs please?
-
@pierrebrunet Thanks for fixing this critical issue!
Do you have any plans regarding internal automated tests that would ensure that critical issues like these (deletion / corruption of backups etc.) do not get shipped?
Best regards
-
@pierrebrunet XO updated to master daefcab, same issue (delta backup S3, problem at merge).
I tried running backup several times to see if the error would clear (it did not). I tried deleting the backup and starting with a new full, error happened at the merge again.
Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.037Z xo:backups:MixinBackupWriter INFO Disk chain needs merging { count: 1 } Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.039Z xo:backups:MixinBackupWriter INFO merging disk chain { Jun 01 10:07:24 xo2 xo-server[9341]: chain: [ Jun 01 10:07:24 xo2 xo-server[9341]: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T134029Z.alias.vhd', Jun 01 10:07:24 xo2 xo-server[9341]: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T134306Z.alias.vhd' Jun 01 10:07:24 xo2 xo-server[9341]: ] Jun 01 10:07:24 xo2 xo-server[9341]: } Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.318Z xo:backups:MixinBackupWriter WARN unhandled error while checking alias { Jun 01 10:07:24 xo2 xo-server[9341]: alias: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T134029Z.alias.vhd', Jun 01 10:07:24 xo2 xo-server[9341]: err: NotFound: UnknownError Jun 01 10:07:24 xo2 xo-server[9341]: at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1850:27) Jun 01 10:07:24 xo2 xo-server[9341]: at process.processTicksAndRejections (node:internal/process/task_queues:104:5) Jun 01 10:07:24 xo2 xo-server[9341]: at async S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:424:13) Jun 01 10:07:24 xo2 xo-server[9341]: at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:27:24 Jun 01 10:07:24 xo2 xo-server[9341]: at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:387:20 Jun 01 10:07:24 xo2 xo-server[9341]: at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50 Jun 01 10:07:24 xo2 xo-server[9341]: at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:64:28 Jun 01 10:07:24 xo2 xo-server[9341]: at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:91:20 Jun 01 10:07:24 xo2 xo-server[9341]: at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:5:26 Jun 01 10:07:24 xo2 xo-server[9341]: at async S3Handler._getSize (/opt/xo/xo-builds/xen-orchestra-202606010843/@xen-orchestra/fs/dist/s3.js:304:20) { Jun 01 10:07:24 xo2 xo-server[9341]: '$fault': 'client', Jun 01 10:07:24 xo2 xo-server[9341]: '$retryable': undefined, Jun 01 10:07:24 xo2 xo-server[9341]: '$metadata': { Jun 01 10:07:24 xo2 xo-server[9341]: httpStatusCode: 404, Jun 01 10:07:24 xo2 xo-server[9341]: requestId: '18B4FA31C10C3A3E', Jun 01 10:07:24 xo2 xo-server[9341]: extendedRequestId: 'a18350611414fa9366e109699727e695e309120afc9803fcaff78a199f9bc4d6', Jun 01 10:07:24 xo2 xo-server[9341]: cfId: undefined, Jun 01 10:07:24 xo2 xo-server[9341]: attempts: 1, Jun 01 10:07:24 xo2 xo-server[9341]: totalRetryDelay: 0 Jun 01 10:07:24 xo2 xo-server[9341]: } Jun 01 10:07:24 xo2 xo-server[9341]: } Jun 01 10:07:24 xo2 xo-server[9341]: } Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.490Z xo:backups:worker INFO backup has ended -
@MajorP93 We are building an automated QA that we keep updating and running regularly (preferably nightly) to add any event possible. We missed this one because we did not took enough failures into account, so we added a regression test for this too.
-
@Andrew I think I found the issue, I am working on it
-
@Andrew
Your error happens after a successful merge in s3 which removes old alias not found (expected). However, the error given by s3 is not generic so it was not handled by alias checking. This issue was always there but we decided to show the warnings during backup and fix them with this new refactoring.Can you try this branch please?
https://github.com/vatesfr/xen-orchestra/tree/pbt_fix_warning_backup
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login