XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    XO VM Backup Started and never completed

    Scheduled Pinned Locked Moved Backup
    7 Posts 4 Posters 80 Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M Offline
      manoli
      last edited by

      My XO VM Backups were working just fine till last update that I did last night.

      From today the XO VM Delta Backup says "Started" but never "Successful" (it is never completed).

      • I'm on XO (built from source) and trying to do Delta Backup of my XO VM to NFS share.
      • I don't see any recently unfinished Tasks.
      • Other VM backups are "Successful".
      • XO config and Pool Metadata are "Successful".
      • NFS Remote on latest version of TrueNAS scale Test Passed OK.
      • Local Storage (where XO VM resides), XCP-ng host and NFS share all have plenty of free storage space available.
      • I did already restart of everything, TrueNAS, XCP-ng host and XO VM it self.

      Nothing helped so far.

      I'm on latest XO version from 30/06/2025 - commit no. 48bf9c9

      Any idea where to start troubleshooting steps? Should I try to rollback previous snapshot that is still available?

      Thank you in advance for your kind help, any information appreciated.

      Have a great day, regards,

      Vladislav

      OhSoNoobO 1 Reply Last reply Reply Quote 1
      • OhSoNoobO Offline
        OhSoNoob @manoli
        last edited by

        @manoli
        I have the exact same issue, backup starts (delta) snapshots are made, but then no export job for the VDI starts running and it hangs indefinitely. This is for me on 3 different sites and setups.

        1 Reply Last reply Reply Quote 0
        • B Offline
          bogikornel
          last edited by bogikornel

          Same issue, here is xo-server logs:

          Jul 01 04:00:07 xoa xo-server[15851]: }
Jul 01 04:00:07 xoa xo-server[15851]:   }
Jul 01 04:00:07 xoa xo-server[15851]:     code: 'UND_ERR_ABORTED'
Jul 01 04:00:07 xoa xo-server[15851]:       at async Promise.all (index 1) {
Jul 01 04:00:07 xoa xo-server[15851]:       at async file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/backups/_incrementalVm.mjs:67:5
Jul 01 04:00:07 xoa xo-server[15851]:       at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Jul 01 04:00:07 xoa xo-server[15851]:       at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/xapi/disks/Xapi.mjs:194:18)
Jul 01 04:00:07 xoa xo-server[15851]:       at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Jul 01 04:00:07 xoa xo-server[15851]:       at #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/xapi/disks/Xapi.mjs:92:21)
Jul 01 04:00:07 xoa xo-server[15851]:       at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/xapi/disks/XapiStreamNbd.mjs:81:19)
Jul 01 04:00:07 xoa xo-server[15851]:       at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:34:29)
Jul 01 04:00:07 xoa xo-server[15851]:       at XapiVhdStreamSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301858/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:160:22)
Jul 01 04:00:07 xoa xo-server[15851]:       at BodyReadable.destroy (/opt/xo/xo-builds/xen-orchestra-202506301858/node_modules/undici/lib/api/readable.js:51:13)
Jul 01 04:00:07 xoa xo-server[15851]:   error: RequestAbortedError [AbortError]: Request aborted
Jul 01 04:00:07 xoa xo-server[15851]: 2025-07-01T02:00:07.508Z xo:backups:worker WARN possibly unhandled rejection {
Jul 01 04:00:07 xoa xo-server[15851]: 2025-07-01T02:00:07.484Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
Jul 01 04:00:06 xoa xo-server[15851]: }
Jul 01 04:00:06 xoa xo-server[15851]:   }
Jul 01 04:00:06 xoa xo-server[15851]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 141))((process xapi)(f>
Jul 01 04:00:06 xoa xo-server[15851]:     subtasks: [],
Jul 01 04:00:06 xoa xo-server[15851]:     subtask_of: 'OpaqueRef:NULL',
Jul 01 04:00:06 xoa xo-server[15851]:     other_config: {},
Jul 01 04:00:06 xoa xo-server[15851]:     ],
Jul 01 04:00:06 xoa xo-server[15851]:       ''
Jul 01 04:00:06 xoa xo-server[15851]:       'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Jul 01 04:00:06 xoa xo-server[15851]:       '',
Jul 01 04:00:06 xoa xo-server[15851]:       'SR_BACKEND_FAILURE_460',
Jul 01 04:00:06 xoa xo-server[15851]:     error_info: [
Jul 01 04:00:06 xoa xo-server[15851]:     result: '',
Jul 01 04:00:06 xoa xo-server[15851]:     type: '<none/>',
Jul 01 04:00:06 xoa xo-server[15851]:     progress: 1,
Jul 01 04:00:06 xoa xo-server[15851]:     resident_on: 'OpaqueRef:a08170fe-0f8d-4a9d-b890-b65bfa0f5aeb',
Jul 01 04:00:06 xoa xo-server[15851]:     status: 'failure',
Jul 01 04:00:06 xoa xo-server[15851]:     finished: '20250701T02:00:06Z',
Jul 01 04:00:06 xoa xo-server[15851]:     created: '20250701T02:00:05Z',
Jul 01 04:00:06 xoa xo-server[15851]:     current_operations: {},
Jul 01 04:00:06 xoa xo-server[15851]:     allowed_operations: [],
Jul 01 04:00:06 xoa xo-server[15851]:     name_description: '',
Jul 01 04:00:06 xoa xo-server[15851]:     name_label: 'Async.VDI.list_changed_blocks',
Jul 01 04:00:06 xoa xo-server[15851]:     uuid: 'f6981d73-138e-3fac-c944-1218e9f54934',
Jul 01 04:00:06 xoa xo-server[15851]:   task: task {
Jul 01 04:00:06 xoa xo-server[15851]:   url: undefined,
Jul 01 04:00:06 xoa xo-server[15851]:   call: undefined,
Jul 01 04:00:06 xoa xo-server[15851]:   ],
Jul 01 04:00:06 xoa xo-server[15851]:     ''
Jul 01 04:00:06 xoa xo-server[15851]:     'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Jul 01 04:00:06 xoa xo-server[15851]:     '',
Jul 01 04:00:06 xoa xo-server[15851]:   params: [
Jul 01 04:00:06 xoa xo-server[15851]:   code: 'SR_BACKEND_FAILURE_460',
Jul 01 04:00:06 xoa xo-server[15851]:     at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
Jul 01 04:00:06 xoa xo-server[15851]:     at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/index.mjs:1270:14)
Jul 01 04:00:06 xoa xo-server[15851]:     at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/index.mjs:1097:12)
Jul 01 04:00:06 xoa xo-server[15851]:     at Array.forEach (<anonymous>)
Jul 01 04:00:06 xoa xo-server[15851]:     at file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/index.mjs:1107:14
Jul 01 04:00:06 xoa xo-server[15851]:     at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/index.mjs:1073:24)
Jul 01 04:00:06 xoa xo-server[15851]:     at default (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/_getTaskResult.mjs:13:29)
Jul 01 04:00:06 xoa xo-server[15851]:     at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202506301858/packages/xen-api/_XapiError.mjs:16:12)
Jul 01 04:00:06 xoa xo-server[15851]: 2025-07-01T02:00:06.151Z @xen-orchestra/xapi/disks/Xapi INFO openNbdCBT XapiError: SR_BACKEND_FAILURE_460(, Failed to calculate changed blocks for give>
          florentF 1 Reply Last reply Reply Quote 0
          • florentF Offline
            florent Vates 🪐 XO Team @bogikornel
            last edited by

            @bogikornel this look like the most recent backup job did not use CBT , but the previous was using it
            is it intendend ?

            florentF B 2 Replies Last reply Reply Quote 0
            • florentF Offline
              florent Vates 🪐 XO Team @florent
              last edited by

              @OhSoNoob @manoli
              do you have anything in the Xo logs ?

              1 Reply Last reply Reply Quote 0
              • B Offline
                bogikornel @florent
                last edited by

                @florent said in XO VM Backup Started and never completed:

                this look like the most recent backup job did not use CBT , but the previous was using it
                is it intendend ?

                Válasz

                This backup has been running like this for a long time, until yesterday's XO update. It has not been configured since then.

                1 Reply Last reply Reply Quote 0
                • M Offline
                  manoli
                  last edited by

                  Workaround found here https://xcp-ng.org/forum/post/94626

                  Seems like turning "Use NDB to transfer disk if available" switch ON and save backup job,
                  then turn the switch OFF and save backup job again, will fix the issue.

                  Thanks to @RobWhalley and have a nice day everyone.

                  1 Reply Last reply Reply Quote 3
                  • First post
                    Last post