XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XO Community edition backups dont work as of build 6b263

    Scheduled Pinned Locked Moved Backup
    61 Posts 17 Posters 1.3k Views 15 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      Check with fcefaae

      R 1 Reply Last reply Reply Quote 0
      • marcoiM Offline
        marcoi
        last edited by

        looks like Xen Orchestra, commit fcefa might have fixed the backup issue. I am running a backup test now and it showing export step running
        61668884-b733-4650-961d-1d171e83e2ce-image.png

        1 Reply Last reply Reply Quote 0
        • R Offline
          RobWhalley @olivierlambert
          last edited by

          @olivierlambert Sadly, it's not working for me - I've updated to the latest commit and tested a job that should take around 5 minutes; it's been stuck on "Started" for a couple of hours now.

          marcoiM 1 Reply Last reply Reply Quote 0
          • marcoiM Offline
            marcoi @RobWhalley
            last edited by

            @RobWhalley did you try to restart the service or reboot the server running XO? I have a nightly crontab job that updates the OS, updates XO and then reboots the server.

            R 1 Reply Last reply Reply Quote 0
            • R Offline
              RobWhalley @marcoi
              last edited by

              @marcoi Hi, the service restarted as part of the upgrade process, but I have done a restart of the VM running Orchestra as well, just to be on the safe side.

              Please see attached log.txt for additional log information.

              git log shows the expected commit:

              /opt/xo/xo-builds/xen-orchestra-202506301106# git log
              commit fcefaaea85651c3c0bb40bfba8199dd4e963211c (HEAD, origin/master, origin/HEAD)
              Author: Pierre Brunet <pierre.demonthelie@gmail.com>
              Date:   Mon Jun 30 10:43:41 2025 +0200
              
                  feat: technical release (#8767)
              
              marcoiM 1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                The rebuild maybe?

                1 Reply Last reply Reply Quote 0
                • marcoiM Offline
                  marcoi @RobWhalley
                  last edited by

                  @RobWhalley to see if it's behaving the same as what I saw on prior versions can you do the following steps?

                  1. open tasks in a new browser window so you can monitor it. If you can keep the backup window and task window side by side you can see if the export command comes and goes quickly.
                    1.a If you go between backup and tasks in the same window, ive noticed the prior task history clears out.

                  2. ssh into the VM running XO and monitor the syslog
                    tail -100f /var/log/syslog
                    #may need root then try sudo tail -100f /var/log/syslog

                  3. In primary window kick off a backup then watch the task window to see if export command comes and goes.
                    c7563916-882b-4077-87eb-7591b871b057-image.png

                  4. If the task disappears then click on previous tasks and see if you even had export command come up.

                  5. reply with the results of how task shows export step and any errors in the syslog file.

                  Hopefully that helps the devs figure out whats going on in your system.

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    Also maybe xo-server -h can give the info we need

                    R 1 Reply Last reply Reply Quote 0
                    • R Offline
                      RobWhalley @olivierlambert
                      last edited by RobWhalley

                      Thanks both for the help, it's much appreciated.

                      @olivierlambert - apologies, I'm having some difficulty in finding xo-server... the closest I can find is a mention to it in /opt/xo/xo-src/xen-orchestra/packages/xo-server/xo-server.service:

                      ExecStart=/usr/local/bin/xo-server
                      
                      # /usr/local/bin/xo-server
                      bash: /usr/local/bin/xo-server: No such file or directory
                      

                      However, when I check /etc/systemd/system/xo-server.service, I get something quite different:

                      ExecStart=/opt/xo/xo-server/dist/cli.mjs
                      

                      This file does exist, but running it with -h doesn't seem to provide a useful output:

                      # /opt/xo/xo-server/dist/cli.mjs -h
                      Usage: xo-server [--safe-mode]
                      
                      xo-server v5.180.1
                      

                      I can run the script in place of the service, but in terms of output it only gives me the same as the journalctl -u xo-server output (see previous attachment):

                      # systemctl stop xo-server
                      # /opt/xo/xo-server/dist/cli.mjs
                      
                      2025-06-30T15:00:42.871Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
                      2025-06-30T15:00:42.892Z xo:backups:worker WARN possibly unhandled rejection {
                        error: RequestAbortedError [AbortError]: Request aborted
                            at BodyReadable.destroy (/opt/xo/xo-builds/xen-orchestra-202506301106/node_modules/undici/lib/api/readable.js:51:13)
                            at XapiVhdStreamSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:160:22)
                            at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:34:29)
                            at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiStreamNbd.mjs:81:19)
                            at #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:92:21)
                            at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
                            at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:198:18)
                            at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
                            at async file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/backups/_incrementalVm.mjs:67:5
                            at async Promise.all (index 0) {
                          code: 'UND_ERR_ABORTED'
                        }
                      }
                      

                      @marcoi - I didn't see anything appear other than this:

                      da6d74fd-5972-4f6c-9fd7-6b10dba8f77e-image.png

                      Edited to correct screenshot

                      marcoiM 1 Reply Last reply Reply Quote 0
                      • marcoiM Offline
                        marcoi @RobWhalley
                        last edited by

                        @RobWhalley I see in the error it mentioning NDB, do your backups have that enabled?
                        6930a2fe-8e9c-48f9-973e-32d3d5f4b690-image.png

                        can you try disabling it and testing backup if you do have it on? I think i saw someone else post issue with NDB code in the forums.

                        R 1 Reply Last reply Reply Quote 0
                        • R Offline
                          RobWhalley @marcoi
                          last edited by

                          Thanks @marcoi - I don't believe so, though I have no experience of the Network Block Device protocol. The VMs are backed up to a NFS share on a Synology NAS, so pretty low tech in comparison to what a lot of folks here will be using. Based on this page, I'm guessing NBD isn't natively supported on these devices?

                          This is a sample of the settings, let me know if you need any further information:

                          83c0102e-d3f3-4588-99e9-7ec30b7c9319-image.png

                          marcoiM 1 Reply Last reply Reply Quote 0
                          • marcoiM Offline
                            marcoi @RobWhalley
                            last edited by

                            @RobWhalley Okay you have similar setup to me then. Can you confirm NFS share for backup is working and enabled?
                            it under settings -->Remote
                            43439813-609e-4425-b968-d074a45c9a05-image.png

                            Make sure the enabled is green and try clicking on the test icon in far right.

                            R 1 Reply Last reply Reply Quote 0
                            • R Offline
                              RobWhalley @marcoi
                              last edited by

                              @marcoi Good thinking, here we go - seems OK:

                              64b68740-1be4-4cbf-bd52-68fda4630ca1-image.png

                              cb88e5d3-8b34-4e13-8daf-6ec2d9052aa5-image.png

                              I think I neglected to mention that while backups using the Delta Backup mode failed, other modes have been working:

                              64cb7c1d-bd61-429a-b8ca-64999fa94bb7-image.png

                              marcoiM 1 Reply Last reply Reply Quote 0
                              • marcoiM Offline
                                marcoi @RobWhalley
                                last edited by

                                @RobWhalley lol i forgot to test deltas - let me see on that.

                                marcoiM 1 Reply Last reply Reply Quote 0
                                • marcoiM Offline
                                  marcoi @marcoi
                                  last edited by

                                  scratch that prior comment. My backups are all setup as delta backup with schedule 2nd run that forces full backup. So i believe both are working correctly.

                                  354b8995-e51b-4e02-9b0e-9de5fff0f06e-image.png
                                  8b8b764b-698a-4538-9d6b-c9b8484ea3c5-image.png
                                  8410c029-8e0e-463a-9065-d3128482d6a0-image.png

                                  if you can also try deleting all the backup snapshots of the VMs in the backup and running full backup then the delta backup jobs.

                                  R 2 Replies Last reply Reply Quote 1
                                  • R Offline
                                    RobWhalley @marcoi
                                    last edited by

                                    Thanks again @marcoi - I wasn't able to get a full backup to work using the Delta Backup with Force full backup option selected. However, using the Backup method was successful, so I have backups for now. I'll try to take another look at it today and do some further testing.

                                    1 Reply Last reply Reply Quote 0
                                    • B Offline
                                      bufanda
                                      last edited by

                                      Hi,

                                      I tested build 48bf9c9 and backups whether they using NBD or not seem not to start transfers at all.

                                      This is the log output without NBD:

                                      2025-07-01T05:09:04.436Z xo:backups:worker INFO starting backup
                                      2025-07-01T05:09:06.481Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
                                      }
                                      2025-07-01T05:09:06.490Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
                                      }
                                      2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/20250701T050331Z.vhd'
                                      }
                                      2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
                                      }
                                      2025-07-01T05:09:08.029Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
                                      }
                                      2025-07-01T05:09:52.631Z xo:backups:worker WARN possibly unhandled rejection {
                                        error: TypeError: this.source.readBlock is not a function
                                            at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
                                            at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
                                            at buildDiskBlockGenerator.next (<anonymous>)
                                            at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
                                            at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                                            at async Promise.all (index 0)
                                      }
                                      2025-07-01T05:09:52.633Z xo:backups:worker WARN possibly unhandled rejection {
                                        error: TypeError: this.source.readBlock is not a function
                                            at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
                                            at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
                                            at buildDiskBlockGenerator.next (<anonymous>)
                                            at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
                                            at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                                            at async Promise.all (index 0)
                                      }
                                      2025-07-01T05:09:52.634Z xo:backups:worker WARN possibly unhandled rejection {
                                        error: TypeError: this.source.readBlock is not a function
                                            at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
                                            at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
                                            at buildDiskBlockGenerator.next (<anonymous>)
                                            at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
                                            at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                                            at async Promise.all (index 0)
                                      }
                                      

                                      And smae VM when trying to use NBD

                                      2025-07-01T05:02:49.119Z xo:backups:worker INFO starting backup
                                      2025-07-01T05:02:51.256Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
                                      }
                                      2025-07-01T05:02:51.268Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
                                      }
                                      2025-07-01T05:02:51.270Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
                                      }
                                      2025-07-01T05:02:51.627Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
                                      }
                                      2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
                                      }
                                      2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                        path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
                                      }
                                      2025-07-01T05:06:04.924Z xo:backups:worker INFO starting backup
                                      2025-07-01T05:06:09.185Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                        path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
                                      }
                                      2025-07-01T05:06:09.852Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                        path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
                                      }
                                      2025-07-01T05:06:18.711Z xo:xapi:vdi WARN invalid HTTP header in response body {
                                        body: 'HTTP/1.1 500 Internal Error\r\n' +
                                          'content-length: 346\r\n' +
                                          'content-type: text/html\r\n' +
                                          'connection: close\r\n' +
                                          'cache-control: no-cache, no-store\r\n' +
                                          '\r\n' +
                                          '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>Db_exn.Read_missing_uuid(&quot;VDI&quot;, &quot;&quot;, &quot;OpaqueRef:9805714f-7a36-6c2c-c194-9f08baf519df&quot;)</body></html>'
                                      }
                                      
                                      1 Reply Last reply Reply Quote 0
                                      • R Offline
                                        RobWhalley @marcoi
                                        last edited by

                                        @marcoi @olivierlambert OK, I think I've found the problem, and a fix... at least for the error I was seeing...

                                        • If you create a new backup with the method Delta Backup, it will fail with the error posted previously.
                                        • Activate the "Use NBD to transfer disk if available" switch
                                        • Save the backup job
                                        • Deactivate the "Use NBD to transfer disk if available" switch
                                        • Save the backup job
                                        • Run the backup job again

                                        This time, journalctl doesn't log any errors, and the export task can be seen:

                                        026d4457-9d0b-4d1d-90a6-91e199c99938-image.png

                                        My guess is, that even though by default the "Use NBD to transfer disk if available" switch is shown as inactive when creating a new backup job...

                                        244a3cce-27d8-44ab-abe4-c34f196efd09-image.png

                                        ...in reality NBD is switched on by default. Activating and deactivating the switch presumably creates an entry in the job that specifies NBD shouldn't be used.

                                        It would be interesting to know if anyone else can replicate this behaviour, or if it's just something odd in my setup?

                                        florentF siemsenitS marcoiM itservicesI M 5 Replies Last reply Reply Quote 2
                                        • florentF Offline
                                          florent Vates 🪐 XO Team @RobWhalley
                                          last edited by

                                          @RobWhalley nice work, I am testing it right now

                                          R 1 Reply Last reply Reply Quote 2
                                          • siemsenitS Offline
                                            siemsenit @RobWhalley
                                            last edited by

                                            @RobWhalley Thanks! It looks like that fixed it for me.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post