XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XO Community edition backups dont work as of build 6b263

    Scheduled Pinned Locked Moved Backup
    64 Posts 17 Posters 1.5k Views 15 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R Offline
      RobWhalley @marcoi
      last edited by

      Thanks again @marcoi - I wasn't able to get a full backup to work using the Delta Backup with Force full backup option selected. However, using the Backup method was successful, so I have backups for now. I'll try to take another look at it today and do some further testing.

      1 Reply Last reply Reply Quote 0
      • B Offline
        bufanda
        last edited by

        Hi,

        I tested build 48bf9c9 and backups whether they using NBD or not seem not to start transfers at all.

        This is the log output without NBD:

        2025-07-01T05:09:04.436Z xo:backups:worker INFO starting backup
        2025-07-01T05:09:06.481Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
        }
        2025-07-01T05:09:06.490Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
        }
        2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/20250701T050331Z.vhd'
        }
        2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
        }
        2025-07-01T05:09:08.029Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
        }
        2025-07-01T05:09:52.631Z xo:backups:worker WARN possibly unhandled rejection {
          error: TypeError: this.source.readBlock is not a function
              at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
              at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
              at buildDiskBlockGenerator.next (<anonymous>)
              at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async Promise.all (index 0)
        }
        2025-07-01T05:09:52.633Z xo:backups:worker WARN possibly unhandled rejection {
          error: TypeError: this.source.readBlock is not a function
              at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
              at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
              at buildDiskBlockGenerator.next (<anonymous>)
              at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async Promise.all (index 0)
        }
        2025-07-01T05:09:52.634Z xo:backups:worker WARN possibly unhandled rejection {
          error: TypeError: this.source.readBlock is not a function
              at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
              at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
              at buildDiskBlockGenerator.next (<anonymous>)
              at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
              at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
              at async Promise.all (index 0)
        }
        

        And smae VM when trying to use NBD

        2025-07-01T05:02:49.119Z xo:backups:worker INFO starting backup
        2025-07-01T05:02:51.256Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
        }
        2025-07-01T05:02:51.268Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
        }
        2025-07-01T05:02:51.270Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
        }
        2025-07-01T05:02:51.627Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
        }
        2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
        }
        2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
          path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
        }
        2025-07-01T05:06:04.924Z xo:backups:worker INFO starting backup
        2025-07-01T05:06:09.185Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
          path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
        }
        2025-07-01T05:06:09.852Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
          path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
        }
        2025-07-01T05:06:18.711Z xo:xapi:vdi WARN invalid HTTP header in response body {
          body: 'HTTP/1.1 500 Internal Error\r\n' +
            'content-length: 346\r\n' +
            'content-type: text/html\r\n' +
            'connection: close\r\n' +
            'cache-control: no-cache, no-store\r\n' +
            '\r\n' +
            '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>Db_exn.Read_missing_uuid(&quot;VDI&quot;, &quot;&quot;, &quot;OpaqueRef:9805714f-7a36-6c2c-c194-9f08baf519df&quot;)</body></html>'
        }
        
        1 Reply Last reply Reply Quote 0
        • R Offline
          RobWhalley @marcoi
          last edited by

          @marcoi @olivierlambert OK, I think I've found the problem, and a fix... at least for the error I was seeing...

          • If you create a new backup with the method Delta Backup, it will fail with the error posted previously.
          • Activate the "Use NBD to transfer disk if available" switch
          • Save the backup job
          • Deactivate the "Use NBD to transfer disk if available" switch
          • Save the backup job
          • Run the backup job again

          This time, journalctl doesn't log any errors, and the export task can be seen:

          026d4457-9d0b-4d1d-90a6-91e199c99938-image.png

          My guess is, that even though by default the "Use NBD to transfer disk if available" switch is shown as inactive when creating a new backup job...

          244a3cce-27d8-44ab-abe4-c34f196efd09-image.png

          ...in reality NBD is switched on by default. Activating and deactivating the switch presumably creates an entry in the job that specifies NBD shouldn't be used.

          It would be interesting to know if anyone else can replicate this behaviour, or if it's just something odd in my setup?

          florentF siemsenitS marcoiM itservicesI M 5 Replies Last reply Reply Quote 2
          • florentF Offline
            florent Vates 🪐 XO Team @RobWhalley
            last edited by

            @RobWhalley nice work, I am testing it right now

            R 1 Reply Last reply Reply Quote 2
            • siemsenitS Offline
              siemsenit @RobWhalley
              last edited by

              @RobWhalley Thanks! It looks like that fixed it for me.

              1 Reply Last reply Reply Quote 1
              • R Offline
                RobWhalley @florent
                last edited by

                @florent Thanks for taking a look, much appreciated!
                @siemsenit Glad to help! I can't claim credit though, @marcoi pointed out the mention of NBD in the logs 👍

                1 Reply Last reply Reply Quote 1
                • T Offline
                  torso70
                  last edited by

                  @RobWhalley
                  It works for me too.

                  1 Reply Last reply Reply Quote 1
                  • florentF Offline
                    florent Vates 🪐 XO Team
                    last edited by

                    you're right, preferNbd default value is not the same on the front and the back, we'll fix it

                    Most of the backup issue are solved by either removing the snapshot, or having it fa enough so thaht the next backup will be a base one, with all the data

                    I am still not reproducing the nackup stuck issue after this commit https://github.com/vatesfr/xen-orchestra/commit/3c8efb4dc3e1cb3955521063047227849cfb6ed3

                    0 fbeauchamp committed to vatesfr/xen-orchestra
                    fix(backups): non NBD backups stalls (#8766)
                    1 Reply Last reply Reply Quote 1
                    • marcoiM Offline
                      marcoi @RobWhalley
                      last edited by

                      @RobWhalley I think in the beginning i had that option enabled in my backup jobs, but found it was a hassle to move VMs between hosts with it on so i turned it off for all backups. Might be why my jobs worked because i had already done that switch in the past.

                      1 Reply Last reply Reply Quote 2
                      • itservicesI Offline
                        itservices @RobWhalley
                        last edited by

                        @RobWhalley
                        That worked for me as well.

                        I created a new XenOrchestra VM with commit 3c543.

                        Enabling and disabling NBD on the backup job did the trick.

                        Thank you and everyone else for your efforts and digging into the matter.

                        Regards
                        Marc

                        1 Reply Last reply Reply Quote 2
                        • M Offline
                          manoli @RobWhalley
                          last edited by

                          @RobWhalley Thank you... Works for me as well, good job!

                          Have a great day everyone.

                          1 Reply Last reply Reply Quote 1
                          • florentF Offline
                            florent Vates 🪐 XO Team
                            last edited by

                            @RobWhalley saved the day
                            We are still working on it to better understand why it fixes and improve the situation

                            florentF 1 Reply Last reply Reply Quote 2
                            • florentF Offline
                              florent Vates 🪐 XO Team @florent
                              last edited by

                              found it , there was not one, but 2 bugs
                              albeit small they were putting the xo backup in a incorrect state
                              the fix is here, and will be merged as soon as possible and then released as a patch for xoa users
                              https://github.com/vatesfr/xen-orchestra/pull/8780

                              really thank you everyone for the information

                              fbeauchamp opened this pull request in vatesfr/xen-orchestra

                              open fix(xapi): respect the config file setting for preferNbd #8780

                              R 1 Reply Last reply Reply Quote 2
                              • R Offline
                                RobWhalley @florent
                                last edited by

                                @florent Great work, thanks for the fixes 👍

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post