XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XO Community edition backups dont work as of build 6b263

    Scheduled Pinned Locked Moved Backup
    64 Posts 17 Posters 1.7k Views 15 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      Also maybe xo-server -h can give the info we need

      R 1 Reply Last reply Reply Quote 0
      • R Offline
        RobWhalley @olivierlambert
        last edited by RobWhalley

        Thanks both for the help, it's much appreciated.

        @olivierlambert - apologies, I'm having some difficulty in finding xo-server... the closest I can find is a mention to it in /opt/xo/xo-src/xen-orchestra/packages/xo-server/xo-server.service:

        ExecStart=/usr/local/bin/xo-server
        
        # /usr/local/bin/xo-server
        bash: /usr/local/bin/xo-server: No such file or directory
        

        However, when I check /etc/systemd/system/xo-server.service, I get something quite different:

        ExecStart=/opt/xo/xo-server/dist/cli.mjs
        

        This file does exist, but running it with -h doesn't seem to provide a useful output:

        # /opt/xo/xo-server/dist/cli.mjs -h
        Usage: xo-server [--safe-mode]
        
        xo-server v5.180.1
        

        I can run the script in place of the service, but in terms of output it only gives me the same as the journalctl -u xo-server output (see previous attachment):

        # systemctl stop xo-server
        # /opt/xo/xo-server/dist/cli.mjs
        
        2025-06-30T15:00:42.871Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
        2025-06-30T15:00:42.892Z xo:backups:worker WARN possibly unhandled rejection {
          error: RequestAbortedError [AbortError]: Request aborted
              at BodyReadable.destroy (/opt/xo/xo-builds/xen-orchestra-202506301106/node_modules/undici/lib/api/readable.js:51:13)
              at XapiVhdStreamSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:160:22)
              at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:34:29)
              at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiStreamNbd.mjs:81:19)
              at #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:92:21)
              at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
              at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:198:18)
              at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
              at async file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/backups/_incrementalVm.mjs:67:5
              at async Promise.all (index 0) {
            code: 'UND_ERR_ABORTED'
          }
        }
        

        @marcoi - I didn't see anything appear other than this:

        da6d74fd-5972-4f6c-9fd7-6b10dba8f77e-image.png

        Edited to correct screenshot

        marcoiM 1 Reply Last reply Reply Quote 0
        • marcoiM Offline
          marcoi @RobWhalley
          last edited by

          @RobWhalley I see in the error it mentioning NDB, do your backups have that enabled?
          6930a2fe-8e9c-48f9-973e-32d3d5f4b690-image.png

          can you try disabling it and testing backup if you do have it on? I think i saw someone else post issue with NDB code in the forums.

          R 1 Reply Last reply Reply Quote 0
          • R Offline
            RobWhalley @marcoi
            last edited by

            Thanks @marcoi - I don't believe so, though I have no experience of the Network Block Device protocol. The VMs are backed up to a NFS share on a Synology NAS, so pretty low tech in comparison to what a lot of folks here will be using. Based on this page, I'm guessing NBD isn't natively supported on these devices?

            This is a sample of the settings, let me know if you need any further information:

            83c0102e-d3f3-4588-99e9-7ec30b7c9319-image.png

            marcoiM 1 Reply Last reply Reply Quote 0
            • marcoiM Offline
              marcoi @RobWhalley
              last edited by

              @RobWhalley Okay you have similar setup to me then. Can you confirm NFS share for backup is working and enabled?
              it under settings -->Remote
              43439813-609e-4425-b968-d074a45c9a05-image.png

              Make sure the enabled is green and try clicking on the test icon in far right.

              R 1 Reply Last reply Reply Quote 0
              • R Offline
                RobWhalley @marcoi
                last edited by

                @marcoi Good thinking, here we go - seems OK:

                64b68740-1be4-4cbf-bd52-68fda4630ca1-image.png

                cb88e5d3-8b34-4e13-8daf-6ec2d9052aa5-image.png

                I think I neglected to mention that while backups using the Delta Backup mode failed, other modes have been working:

                64cb7c1d-bd61-429a-b8ca-64999fa94bb7-image.png

                marcoiM 1 Reply Last reply Reply Quote 0
                • marcoiM Offline
                  marcoi @RobWhalley
                  last edited by

                  @RobWhalley lol i forgot to test deltas - let me see on that.

                  marcoiM 1 Reply Last reply Reply Quote 0
                  • marcoiM Offline
                    marcoi @marcoi
                    last edited by

                    scratch that prior comment. My backups are all setup as delta backup with schedule 2nd run that forces full backup. So i believe both are working correctly.

                    354b8995-e51b-4e02-9b0e-9de5fff0f06e-image.png
                    8b8b764b-698a-4538-9d6b-c9b8484ea3c5-image.png
                    8410c029-8e0e-463a-9065-d3128482d6a0-image.png

                    if you can also try deleting all the backup snapshots of the VMs in the backup and running full backup then the delta backup jobs.

                    R 2 Replies Last reply Reply Quote 1
                    • R Offline
                      RobWhalley @marcoi
                      last edited by

                      Thanks again @marcoi - I wasn't able to get a full backup to work using the Delta Backup with Force full backup option selected. However, using the Backup method was successful, so I have backups for now. I'll try to take another look at it today and do some further testing.

                      1 Reply Last reply Reply Quote 0
                      • B Offline
                        bufanda
                        last edited by

                        Hi,

                        I tested build 48bf9c9 and backups whether they using NBD or not seem not to start transfers at all.

                        This is the log output without NBD:

                        2025-07-01T05:09:04.436Z xo:backups:worker INFO starting backup
                        2025-07-01T05:09:06.481Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
                        }
                        2025-07-01T05:09:06.490Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
                        }
                        2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/20250701T050331Z.vhd'
                        }
                        2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
                        }
                        2025-07-01T05:09:08.029Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
                        }
                        2025-07-01T05:09:52.631Z xo:backups:worker WARN possibly unhandled rejection {
                          error: TypeError: this.source.readBlock is not a function
                              at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
                              at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
                              at buildDiskBlockGenerator.next (<anonymous>)
                              at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
                              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                              at async Promise.all (index 0)
                        }
                        2025-07-01T05:09:52.633Z xo:backups:worker WARN possibly unhandled rejection {
                          error: TypeError: this.source.readBlock is not a function
                              at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
                              at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
                              at buildDiskBlockGenerator.next (<anonymous>)
                              at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
                              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                              at async Promise.all (index 0)
                        }
                        2025-07-01T05:09:52.634Z xo:backups:worker WARN possibly unhandled rejection {
                          error: TypeError: this.source.readBlock is not a function
                              at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
                              at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
                              at buildDiskBlockGenerator.next (<anonymous>)
                              at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
                              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
                              at async Promise.all (index 0)
                        }
                        

                        And smae VM when trying to use NBD

                        2025-07-01T05:02:49.119Z xo:backups:worker INFO starting backup
                        2025-07-01T05:02:51.256Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
                        }
                        2025-07-01T05:02:51.268Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
                        }
                        2025-07-01T05:02:51.270Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
                        }
                        2025-07-01T05:02:51.627Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
                        }
                        2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
                        }
                        2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
                        }
                        2025-07-01T05:06:04.924Z xo:backups:worker INFO starting backup
                        2025-07-01T05:06:09.185Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
                          path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
                        }
                        2025-07-01T05:06:09.852Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
                          path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
                        }
                        2025-07-01T05:06:18.711Z xo:xapi:vdi WARN invalid HTTP header in response body {
                          body: 'HTTP/1.1 500 Internal Error\r\n' +
                            'content-length: 346\r\n' +
                            'content-type: text/html\r\n' +
                            'connection: close\r\n' +
                            'cache-control: no-cache, no-store\r\n' +
                            '\r\n' +
                            '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>Db_exn.Read_missing_uuid(&quot;VDI&quot;, &quot;&quot;, &quot;OpaqueRef:9805714f-7a36-6c2c-c194-9f08baf519df&quot;)</body></html>'
                        }
                        
                        1 Reply Last reply Reply Quote 0
                        • R Offline
                          RobWhalley @marcoi
                          last edited by

                          @marcoi @olivierlambert OK, I think I've found the problem, and a fix... at least for the error I was seeing...

                          • If you create a new backup with the method Delta Backup, it will fail with the error posted previously.
                          • Activate the "Use NBD to transfer disk if available" switch
                          • Save the backup job
                          • Deactivate the "Use NBD to transfer disk if available" switch
                          • Save the backup job
                          • Run the backup job again

                          This time, journalctl doesn't log any errors, and the export task can be seen:

                          026d4457-9d0b-4d1d-90a6-91e199c99938-image.png

                          My guess is, that even though by default the "Use NBD to transfer disk if available" switch is shown as inactive when creating a new backup job...

                          244a3cce-27d8-44ab-abe4-c34f196efd09-image.png

                          ...in reality NBD is switched on by default. Activating and deactivating the switch presumably creates an entry in the job that specifies NBD shouldn't be used.

                          It would be interesting to know if anyone else can replicate this behaviour, or if it's just something odd in my setup?

                          florentF siemsenitS marcoiM itservicesI M 5 Replies Last reply Reply Quote 2
                          • florentF Offline
                            florent Vates 🪐 XO Team @RobWhalley
                            last edited by

                            @RobWhalley nice work, I am testing it right now

                            R 1 Reply Last reply Reply Quote 2
                            • siemsenitS Offline
                              siemsenit @RobWhalley
                              last edited by

                              @RobWhalley Thanks! It looks like that fixed it for me.

                              1 Reply Last reply Reply Quote 1
                              • R Offline
                                RobWhalley @florent
                                last edited by

                                @florent Thanks for taking a look, much appreciated!
                                @siemsenit Glad to help! I can't claim credit though, @marcoi pointed out the mention of NBD in the logs 👍

                                1 Reply Last reply Reply Quote 1
                                • T Offline
                                  torso70
                                  last edited by

                                  @RobWhalley
                                  It works for me too.

                                  1 Reply Last reply Reply Quote 1
                                  • florentF Offline
                                    florent Vates 🪐 XO Team
                                    last edited by

                                    you're right, preferNbd default value is not the same on the front and the back, we'll fix it

                                    Most of the backup issue are solved by either removing the snapshot, or having it fa enough so thaht the next backup will be a base one, with all the data

                                    I am still not reproducing the nackup stuck issue after this commit https://github.com/vatesfr/xen-orchestra/commit/3c8efb4dc3e1cb3955521063047227849cfb6ed3

                                    0 fbeauchamp committed to vatesfr/xen-orchestra
                                    fix(backups): non NBD backups stalls (#8766)
                                    1 Reply Last reply Reply Quote 1
                                    • marcoiM Offline
                                      marcoi @RobWhalley
                                      last edited by

                                      @RobWhalley I think in the beginning i had that option enabled in my backup jobs, but found it was a hassle to move VMs between hosts with it on so i turned it off for all backups. Might be why my jobs worked because i had already done that switch in the past.

                                      1 Reply Last reply Reply Quote 2
                                      • itservicesI Offline
                                        itservices @RobWhalley
                                        last edited by

                                        @RobWhalley
                                        That worked for me as well.

                                        I created a new XenOrchestra VM with commit 3c543.

                                        Enabling and disabling NBD on the backup job did the trick.

                                        Thank you and everyone else for your efforts and digging into the matter.

                                        Regards
                                        Marc

                                        1 Reply Last reply Reply Quote 2
                                        • M Offline
                                          manoli @RobWhalley
                                          last edited by

                                          @RobWhalley Thank you... Works for me as well, good job!

                                          Have a great day everyone.

                                          1 Reply Last reply Reply Quote 1
                                          • florentF Offline
                                            florent Vates 🪐 XO Team
                                            last edited by

                                            @RobWhalley saved the day
                                            We are still working on it to better understand why it fixes and improve the situation

                                            florentF 1 Reply Last reply Reply Quote 2
                                            • First post
                                              Last post