XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XO error/warning: Clean VM directory. unhandled error while checking alias.

    Scheduled Pinned Locked Moved Backup
    37 Posts 6 Posters 1.7k Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M Online
      MajorP93 @pierrebrunet
      last edited by

      @pierrebrunet Thanks for fixing this critical issue!

      Do you have any plans regarding internal automated tests that would ensure that critical issues like these (deletion / corruption of backups etc.) do not get shipped?

      Best regards

      P 1 Reply Last reply Reply Quote 0
      • A Offline
        Andrew Top contributor @pierrebrunet
        last edited by

        @pierrebrunet XO updated to master daefcab, same issue (delta backup S3, problem at merge).

        I tried running backup several times to see if the error would clear (it did not). I tried deleting the backup and starting with a new full, error happened at the merge again.

        Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.037Z xo:backups:MixinBackupWriter INFO Disk chain needs merging { count: 1 }
Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.039Z xo:backups:MixinBackupWriter INFO merging disk chain {
Jun 01 10:07:24 xo2 xo-server[9341]:   chain: [
Jun 01 10:07:24 xo2 xo-server[9341]:     '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T134029Z.alias.vhd',
Jun 01 10:07:24 xo2 xo-server[9341]:     '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T134306Z.alias.vhd'
Jun 01 10:07:24 xo2 xo-server[9341]:   ]
Jun 01 10:07:24 xo2 xo-server[9341]: }
Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.318Z xo:backups:MixinBackupWriter WARN unhandled error while checking alias {
Jun 01 10:07:24 xo2 xo-server[9341]:   alias: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T134029Z.alias.vhd',
Jun 01 10:07:24 xo2 xo-server[9341]:   err: NotFound: UnknownError
Jun 01 10:07:24 xo2 xo-server[9341]:       at S3RestXmlProtocol.handleError (/opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1850:27)
Jun 01 10:07:24 xo2 xo-server[9341]:       at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
Jun 01 10:07:24 xo2 xo-server[9341]:       at async S3RestXmlProtocol.deserializeResponse (/opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:424:13)
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:27:24
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:387:20
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@smithy/core/dist-cjs/submodules/retry/index.js:172:50
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:64:28
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:91:20
Jun 01 10:07:24 xo2 xo-server[9341]:       at async /opt/xo/xo-builds/xen-orchestra-202606010843/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:5:26
Jun 01 10:07:24 xo2 xo-server[9341]:       at async S3Handler._getSize (/opt/xo/xo-builds/xen-orchestra-202606010843/@xen-orchestra/fs/dist/s3.js:304:20) {
Jun 01 10:07:24 xo2 xo-server[9341]:     '$fault': 'client',
Jun 01 10:07:24 xo2 xo-server[9341]:     '$retryable': undefined,
Jun 01 10:07:24 xo2 xo-server[9341]:     '$metadata': {
Jun 01 10:07:24 xo2 xo-server[9341]:       httpStatusCode: 404,
Jun 01 10:07:24 xo2 xo-server[9341]:       requestId: '18B4FA31C10C3A3E',
Jun 01 10:07:24 xo2 xo-server[9341]:       extendedRequestId: 'a18350611414fa9366e109699727e695e309120afc9803fcaff78a199f9bc4d6',
Jun 01 10:07:24 xo2 xo-server[9341]:       cfId: undefined,
Jun 01 10:07:24 xo2 xo-server[9341]:       attempts: 1,
Jun 01 10:07:24 xo2 xo-server[9341]:       totalRetryDelay: 0
Jun 01 10:07:24 xo2 xo-server[9341]:     }
Jun 01 10:07:24 xo2 xo-server[9341]:   }
Jun 01 10:07:24 xo2 xo-server[9341]: }
Jun 01 10:07:24 xo2 xo-server[9341]: 2026-06-01T14:07:24.490Z xo:backups:worker INFO backup has ended
        P 2 Replies Last reply Reply Quote 0
        • P Offline
          pierrebrunet Vates 🪐 XO Team @MajorP93
          last edited by pierrebrunet

          @MajorP93 We are building an automated QA that we keep updating and running regularly (preferably nightly) to add any event possible. We missed this one because we did not took enough failures into account, so we added a regression test for this too.

          1 Reply Last reply Reply Quote 1
          • P Offline
            pierrebrunet Vates 🪐 XO Team @Andrew
            last edited by

            @Andrew I think I found the issue, I am working on it

            1 Reply Last reply Reply Quote 0
            • P Offline
              pierrebrunet Vates 🪐 XO Team @Andrew
              last edited by

              @Andrew
              Your error happens after a successful merge in s3 which removes old alias not found (expected). However, the error given by s3 is not generic so it was not handled by alias checking. This issue was always there but we decided to show the warnings during backup and fix them with this new refactoring.

              Can you try this branch please?
              https://github.com/vatesfr/xen-orchestra/tree/pbt_fix_warning_backup

              A 1 Reply Last reply Reply Quote 0
              • A Offline
                Andrew Top contributor @pierrebrunet
                last edited by

                @pierrebrunet The error happens using Wasabi and MinIO. The pbt_fix_warning_backup still has the error (but no 404 error):

                Clean VM directory 
 missing target of alias

                Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.571Z xo:backups:MixinBackupWriter INFO Disk chain needs merging { count: 1 }
Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.571Z xo:backups:MixinBackupWriter INFO merging disk chain {
Jun 01 11:23:22 xo2 xo-server[4216]:   chain: [
Jun 01 11:23:22 xo2 xo-server[4216]:     '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T152042Z.alias.vhd',
Jun 01 11:23:22 xo2 xo-server[4216]:     '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T152256Z.alias.vhd'
Jun 01 11:23:22 xo2 xo-server[4216]:   ]
Jun 01 11:23:22 xo2 xo-server[4216]: }
Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.741Z xo:backups:MixinBackupWriter WARN missing target of alias {
Jun 01 11:23:22 xo2 xo-server[4216]:   alias: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T152042Z.alias.vhd'
Jun 01 11:23:22 xo2 xo-server[4216]: }
Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.741Z xo:backups:MixinBackupWriter INFO removing alias with missing target {
Jun 01 11:23:22 xo2 xo-server[4216]:   alias: '/xo-vm-backups/a081f208-4e40-2ca5-c68e-86106af5a8ef/vdis/d6c0a656-62c5-4c39-a57a-f246b39f1cef/cbb7809e-8296-4dc6-8200-ed1f42e50c68/20260601T152042Z.alias.vhd'
Jun 01 11:23:22 xo2 xo-server[4216]: }
Jun 01 11:23:22 xo2 xo-server[4216]: 2026-06-01T15:23:22.866Z xo:backups:worker INFO backup has ended
                P 1 Reply Last reply Reply Quote 0
                • P Offline
                  pierrebrunet Vates 🪐 XO Team @Andrew
                  last edited by

                  @Andrew You mean it fails the whole job for this warning? Or this is just a warning inside a successful job?

                  A 1 Reply Last reply Reply Quote 0
                  • A Offline
                    Andrew Top contributor @pierrebrunet
                    last edited by

                    @pierrebrunet It's a warning. Backup job is successful.

                    P 1 Reply Last reply Reply Quote 0
                    • P Offline
                      pierrebrunet Vates 🪐 XO Team @Andrew
                      last edited by

                      @Andrew ok, thank you. This one indeed is not supposed to be triggered indeed.

                      1 Reply Last reply Reply Quote 0
                      • FagnerMoraesF Offline
                        FagnerMoraes
                        last edited by

                        Hello again,

                        When running the backup routine to the NAS, it is not saving the file anymore.

                        Please check the image showing:

                        the backup routine,
                        the path it points to on the NAS,
                        and the contents of the folder.

                        The .XVA file was not saved.

                        Erro xcp.png

                        P 2 Replies Last reply Reply Quote 0
                        • FagnerMoraesF Offline
                          FagnerMoraes
                          last edited by

                          On the S3 storage, the backup is being saved correctly and the old files are not being deleted unexpectedly.

                          Using the same backup configuration, the directory cleanup process is not executed twice there.
                          Erro xcp 02.png

                          1 Reply Last reply Reply Quote 0
                          • P Offline
                            pierrebrunet Vates 🪐 XO Team @FagnerMoraes
                            last edited by pierrebrunet

                            @FagnerMoraes Hi,
                            Can you upload your logs please? It is a bit difficult with only screenshots. You can download them from the very modal you are in this screenshot. You can remove any personal name, I just need the logs to see where this comes from but I suspect you have two broken XVA (indeed, you have more than two json metadata).
                            At first glance, I would advise you to use a new folder in your NAS to be sure everything goes well

                            P FagnerMoraesF 2 Replies Last reply Reply Quote 0
                            • P Offline
                              pierrebrunet Vates 🪐 XO Team @pierrebrunet
                              last edited by

                              Moreover, can you show me one metadata file that is supposed to work please?

                              FagnerMoraesF 2 Replies Last reply Reply Quote 0
                              • FagnerMoraesF Offline
                                FagnerMoraes @pierrebrunet
                                last edited by

                                This post is deleted!
                                1 Reply Last reply Reply Quote 0
                                • FagnerMoraesF Offline
                                  FagnerMoraes @pierrebrunet
                                  last edited by

                                  @pierrebrunet
                                  thats a metadata backup works fine, no alerts and no clean directory.
                                  Metadata works fine.txt

                                  P 1 Reply Last reply Reply Quote 0
                                  • FagnerMoraesF Offline
                                    FagnerMoraes @pierrebrunet
                                    last edited by

                                    @pierrebrunet thats a metadata backup works fine:
                                    Metadata works fine.txt

                                    1 Reply Last reply Reply Quote 0
                                    • P Offline
                                      pierrebrunet Vates 🪐 XO Team @FagnerMoraes
                                      last edited by

                                      @FagnerMoraes Is your backup encrypted?

                                      FagnerMoraesF 1 Reply Last reply Reply Quote 0
                                      • P Offline
                                        pierrebrunet Vates 🪐 XO Team @FagnerMoraes
                                        last edited by

                                        @FagnerMoraes I just realized you are not listing the same folder that what is backed up in your NAS backup job: on the left, you are listing incremental backups from a given folder, but in the logs I analysed the xva are in another folder

                                        1 Reply Last reply Reply Quote 0
                                        • FagnerMoraesF Offline
                                          FagnerMoraes @pierrebrunet
                                          last edited by

                                          @pierrebrunet yes is encrypted!

                                          1 Reply Last reply Reply Quote 0
                                          • FagnerMoraesF Offline
                                            FagnerMoraes
                                            last edited by

                                            I performed a full backup and a delta backup test.
                                            The full backup is working, but the delta backup is not.

                                            If you look at the image, I performed 2 full backups successfully, but the first backup, which is not encrypted and is a full backup, is not here — only the .json file that generated it is present.

                                            Where would the delta backup be located if not in the same folder as the others?
                                            According to the log, I only saw the same location being used.

                                            3f42504c-23cc-410d-b3f4-cfaba3b128a3-image.jpeg

                                            P 1 Reply Last reply Reply Quote 0

                                            Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                            Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                            With your input, this post could be even better 💗

                                            Register Login
                                            • First post
                                              Last post