XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XO Community edition backups dont work as of build 6b263

    Scheduled Pinned Locked Moved Backup
    64 Posts 17 Posters 1.6k Views 15 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • itservicesI Offline
      itservices
      last edited by itservices

      Hi.

      I am on commit 2effd and have the same problem.

      Snapshots are being created but no transfer is taking place.

      Regards,
      Marc

      marcoiM 1 Reply Last reply Reply Quote 0
      • marcoiM Offline
        marcoi @itservices
        last edited by

        @itservices
        also the same with Xen Orchestra, commit 2effd

        1 Reply Last reply Reply Quote 0
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          Check with fcefaae

          R 1 Reply Last reply Reply Quote 0
          • marcoiM Offline
            marcoi
            last edited by

            looks like Xen Orchestra, commit fcefa might have fixed the backup issue. I am running a backup test now and it showing export step running
            61668884-b733-4650-961d-1d171e83e2ce-image.png

            1 Reply Last reply Reply Quote 0
            • R Offline
              RobWhalley @olivierlambert
              last edited by

              @olivierlambert Sadly, it's not working for me - I've updated to the latest commit and tested a job that should take around 5 minutes; it's been stuck on "Started" for a couple of hours now.

              marcoiM 1 Reply Last reply Reply Quote 0
              • marcoiM Offline
                marcoi @RobWhalley
                last edited by

                @RobWhalley did you try to restart the service or reboot the server running XO? I have a nightly crontab job that updates the OS, updates XO and then reboots the server.

                R 1 Reply Last reply Reply Quote 0
                • R Offline
                  RobWhalley @marcoi
                  last edited by

                  @marcoi Hi, the service restarted as part of the upgrade process, but I have done a restart of the VM running Orchestra as well, just to be on the safe side.

                  Please see attached log.txt for additional log information.

                  git log shows the expected commit:

                  /opt/xo/xo-builds/xen-orchestra-202506301106# git log
                  commit fcefaaea85651c3c0bb40bfba8199dd4e963211c (HEAD, origin/master, origin/HEAD)
                  Author: Pierre Brunet <pierre.demonthelie@gmail.com>
                  Date:   Mon Jun 30 10:43:41 2025 +0200
                  
                      feat: technical release (#8767)
                  
                  marcoiM 1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    The rebuild maybe?

                    1 Reply Last reply Reply Quote 0
                    • marcoiM Offline
                      marcoi @RobWhalley
                      last edited by

                      @RobWhalley to see if it's behaving the same as what I saw on prior versions can you do the following steps?

                      1. open tasks in a new browser window so you can monitor it. If you can keep the backup window and task window side by side you can see if the export command comes and goes quickly.
                        1.a If you go between backup and tasks in the same window, ive noticed the prior task history clears out.

                      2. ssh into the VM running XO and monitor the syslog
                        tail -100f /var/log/syslog
                        #may need root then try sudo tail -100f /var/log/syslog

                      3. In primary window kick off a backup then watch the task window to see if export command comes and goes.
                        c7563916-882b-4077-87eb-7591b871b057-image.png

                      4. If the task disappears then click on previous tasks and see if you even had export command come up.

                      5. reply with the results of how task shows export step and any errors in the syslog file.

                      Hopefully that helps the devs figure out whats going on in your system.

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by

                        Also maybe xo-server -h can give the info we need

                        R 1 Reply Last reply Reply Quote 0
                        • R Offline
                          RobWhalley @olivierlambert
                          last edited by RobWhalley

                          Thanks both for the help, it's much appreciated.

                          @olivierlambert - apologies, I'm having some difficulty in finding xo-server... the closest I can find is a mention to it in /opt/xo/xo-src/xen-orchestra/packages/xo-server/xo-server.service:

                          ExecStart=/usr/local/bin/xo-server
                          
                          # /usr/local/bin/xo-server
                          bash: /usr/local/bin/xo-server: No such file or directory
                          

                          However, when I check /etc/systemd/system/xo-server.service, I get something quite different:

                          ExecStart=/opt/xo/xo-server/dist/cli.mjs
                          

                          This file does exist, but running it with -h doesn't seem to provide a useful output:

                          # /opt/xo/xo-server/dist/cli.mjs -h
                          Usage: xo-server [--safe-mode]
                          
                          xo-server v5.180.1
                          

                          I can run the script in place of the service, but in terms of output it only gives me the same as the journalctl -u xo-server output (see previous attachment):

                          # systemctl stop xo-server
                          # /opt/xo/xo-server/dist/cli.mjs
                          
                          2025-06-30T15:00:42.871Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
                          2025-06-30T15:00:42.892Z xo:backups:worker WARN possibly unhandled rejection {
                            error: RequestAbortedError [AbortError]: Request aborted
                                at BodyReadable.destroy (/opt/xo/xo-builds/xen-orchestra-202506301106/node_modules/undici/lib/api/readable.js:51:13)
                                at XapiVhdStreamSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:160:22)
                                at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:34:29)
                                at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiStreamNbd.mjs:81:19)
                                at #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:92:21)
                                at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
                                at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:198:18)
                                at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
                                at async file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/backups/_incrementalVm.mjs:67:5
                                at async Promise.all (index 0) {
                              code: 'UND_ERR_ABORTED'
                            }
                          }
                          

                          @marcoi - I didn't see anything appear other than this:

                          da6d74fd-5972-4f6c-9fd7-6b10dba8f77e-image.png

                          Edited to correct screenshot

                          marcoiM 1 Reply Last reply Reply Quote 0
                          • marcoiM Offline
                            marcoi @RobWhalley
                            last edited by

                            @RobWhalley I see in the error it mentioning NDB, do your backups have that enabled?
                            6930a2fe-8e9c-48f9-973e-32d3d5f4b690-image.png

                            can you try disabling it and testing backup if you do have it on? I think i saw someone else post issue with NDB code in the forums.

                            R 1 Reply Last reply Reply Quote 0
                            • R Offline
                              RobWhalley @marcoi
                              last edited by

                              Thanks @marcoi - I don't believe so, though I have no experience of the Network Block Device protocol. The VMs are backed up to a NFS share on a Synology NAS, so pretty low tech in comparison to what a lot of folks here will be using. Based on this page, I'm guessing NBD isn't natively supported on these devices?

                              This is a sample of the settings, let me know if you need any further information:

                              83c0102e-d3f3-4588-99e9-7ec30b7c9319-image.png

                              marcoiM 1 Reply Last reply Reply Quote 0
                              • marcoiM Offline
                                marcoi @RobWhalley
                                last edited by

                                @RobWhalley Okay you have similar setup to me then. Can you confirm NFS share for backup is working and enabled?
                                it under settings -->Remote
                                43439813-609e-4425-b968-d074a45c9a05-image.png

                                Make sure the enabled is green and try clicking on the test icon in far right.

                                R 1 Reply Last reply Reply Quote 0
                                • R Offline
                                  RobWhalley @marcoi
                                  last edited by

                                  @marcoi Good thinking, here we go - seems OK:

                                  64b68740-1be4-4cbf-bd52-68fda4630ca1-image.png

                                  cb88e5d3-8b34-4e13-8daf-6ec2d9052aa5-image.png

                                  I think I neglected to mention that while backups using the Delta Backup mode failed, other modes have been working:

                                  64cb7c1d-bd61-429a-b8ca-64999fa94bb7-image.png

                                  marcoiM 1 Reply Last reply Reply Quote 0
                                  • marcoiM Offline
                                    marcoi @RobWhalley
                                    last edited by

                                    @RobWhalley lol i forgot to test deltas - let me see on that.

                                    marcoiM 1 Reply Last reply Reply Quote 0
                                    • marcoiM Offline
                                      marcoi @marcoi
                                      last edited by

                                      scratch that prior comment. My backups are all setup as delta backup with schedule 2nd run that forces full backup. So i believe both are working correctly.

                                      354b8995-e51b-4e02-9b0e-9de5fff0f06e-image.png
                                      8b8b764b-698a-4538-9d6b-c9b8484ea3c5-image.png
                                      8410c029-8e0e-463a-9065-d3128482d6a0-image.png

                                      if you can also try deleting all the backup snapshots of the VMs in the backup and running full backup then the delta backup jobs.

                                      R 2 Replies Last reply Reply Quote 1
                                      • R Offline
                                        RobWhalley @marcoi
                                        last edited by

                                        Thanks again @marcoi - I wasn't able to get a full backup to work using the Delta Backup with Force full backup option selected. However, using the Backup method was successful, so I have backups for now. I'll try to take another look at it today and do some further testing.

                                        1 Reply Last reply Reply Quote 0
                                        • B Offline
                                          bufanda
                                          last edited by

                                          Hi,

                                          I tested build 48bf9c9 and backups whether they using NBD or not seem not to start transfers at all.

                                          This is the log output without NBD:

                                          2025-07-01T05:09:04.436Z xo:backups:worker INFO starting backup
                                          2025-07-01T05:09:06.481Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
                                          }
                                          2025-07-01T05:09:06.490Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
                                          }
                                          2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/20250701T050331Z.vhd'
                                          }
                                          2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
                                          }
                                          2025-07-01T05:09:08.029Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
                                          }
                                          2025-07-01T05:09:52.631Z xo:backups:worker WARN possibly unhandled rejection {
                                            error: TypeError: this.source.readBlock is not a function
                                                at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
                                                at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
                                                at buildDiskBlockGenerator.next (<anonymous>)
                                                at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
                                                at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                                                at async Promise.all (index 0)
                                          }
                                          2025-07-01T05:09:52.633Z xo:backups:worker WARN possibly unhandled rejection {
                                            error: TypeError: this.source.readBlock is not a function
                                                at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
                                                at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
                                                at buildDiskBlockGenerator.next (<anonymous>)
                                                at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
                                                at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                                                at async Promise.all (index 0)
                                          }
                                          2025-07-01T05:09:52.634Z xo:backups:worker WARN possibly unhandled rejection {
                                            error: TypeError: this.source.readBlock is not a function
                                                at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
                                                at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
                                                at buildDiskBlockGenerator.next (<anonymous>)
                                                at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
                                                at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                                                at async Promise.all (index 0)
                                          }
                                          

                                          And smae VM when trying to use NBD

                                          2025-07-01T05:02:49.119Z xo:backups:worker INFO starting backup
                                          2025-07-01T05:02:51.256Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
                                          }
                                          2025-07-01T05:02:51.268Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
                                          }
                                          2025-07-01T05:02:51.270Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
                                          }
                                          2025-07-01T05:02:51.627Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
                                          }
                                          2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
                                          }
                                          2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                            path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
                                          }
                                          2025-07-01T05:06:04.924Z xo:backups:worker INFO starting backup
                                          2025-07-01T05:06:09.185Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                                            path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
                                          }
                                          2025-07-01T05:06:09.852Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                                            path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
                                          }
                                          2025-07-01T05:06:18.711Z xo:xapi:vdi WARN invalid HTTP header in response body {
                                            body: 'HTTP/1.1 500 Internal Error\r\n' +
                                              'content-length: 346\r\n' +
                                              'content-type: text/html\r\n' +
                                              'connection: close\r\n' +
                                              'cache-control: no-cache, no-store\r\n' +
                                              '\r\n' +
                                              '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>Db_exn.Read_missing_uuid(&quot;VDI&quot;, &quot;&quot;, &quot;OpaqueRef:9805714f-7a36-6c2c-c194-9f08baf519df&quot;)</body></html>'
                                          }
                                          
                                          1 Reply Last reply Reply Quote 0
                                          • R Offline
                                            RobWhalley @marcoi
                                            last edited by

                                            @marcoi @olivierlambert OK, I think I've found the problem, and a fix... at least for the error I was seeing...

                                            • If you create a new backup with the method Delta Backup, it will fail with the error posted previously.
                                            • Activate the "Use NBD to transfer disk if available" switch
                                            • Save the backup job
                                            • Deactivate the "Use NBD to transfer disk if available" switch
                                            • Save the backup job
                                            • Run the backup job again

                                            This time, journalctl doesn't log any errors, and the export task can be seen:

                                            026d4457-9d0b-4d1d-90a6-91e199c99938-image.png

                                            My guess is, that even though by default the "Use NBD to transfer disk if available" switch is shown as inactive when creating a new backup job...

                                            244a3cce-27d8-44ab-abe4-c34f196efd09-image.png

                                            ...in reality NBD is switched on by default. Activating and deactivating the switch presumably creates an entry in the job that specifies NBD shouldn't be used.

                                            It would be interesting to know if anyone else can replicate this behaviour, or if it's just something odd in my setup?

                                            florentF siemsenitS marcoiM itservicesI M 5 Replies Last reply Reply Quote 2
                                            • First post
                                              Last post