Posts made by katapaltes | XCP-ng and XO forum

katapaltes

Hello @florent , the results below between 8:04:10 and 8:04:36 seem to align with one of the three failed VM backups. NB: The error did not return when I did another Delta backup (everything backed up successfully). I am now doing a Force Full Backup on all VMs just to be safe. As of now I am still running 1a7b5 and have not yet updated to 7994f.

Jun 23 08:01:14 sxoap31 xo-server[19663]: }
Jun 23 08:04:10 sxoap31 xo-server[19663]: 2025-06-23T13:04:10.703Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
Jun 23 08:04:10 sxoap31 xo-server[19663]:   path: '/xo-vm-backups/bca06bdf-3e1f-40a4-570e-7b584b356e22/20250623T130122Z.json',
Jun 23 08:04:10 sxoap31 xo-server[19663]:   actual: 12283019264,
Jun 23 08:04:10 sxoap31 xo-server[19663]:   expected: 12288363520
Jun 23 08:04:10 sxoap31 xo-server[19663]: }
Jun 23 08:04:31 sxoap31 xo-server[19663]: 2025-06-23T13:04:31.411Z @xen-orchestra/xapi/disks/Xapi WARN openNbdCBT XapiError: SR_BACKEND_FAILURE_460(, Failed to calculate changed blocks for given VDIs. [opterr=Source and target V>
Jun 23 08:04:31 sxoap31 xo-server[19663]:     at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202506230729/packages/xen-api/_XapiError.mjs:16:12)
Jun 23 08:04:31 sxoap31 xo-server[19663]:     at default (file:///opt/xo/xo-builds/xen-orchestra-202506230729/packages/xen-api/_getTaskResult.mjs:13:29)
Jun 23 08:04:31 sxoap31 xo-server[19663]:     at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202506230729/packages/xen-api/index.mjs:1073:24)
Jun 23 08:04:31 sxoap31 xo-server[19663]:     at file:///opt/xo/xo-builds/xen-orchestra-202506230729/packages/xen-api/index.mjs:1107:14
Jun 23 08:04:31 sxoap31 xo-server[19663]:     at Array.forEach (<anonymous>)
Jun 23 08:04:31 sxoap31 xo-server[19663]:     at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202506230729/packages/xen-api/index.mjs:1097:12)
Jun 23 08:04:31 sxoap31 xo-server[19663]:     at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202506230729/packages/xen-api/index.mjs:1270:14)
Jun 23 08:04:31 sxoap31 xo-server[19663]:     at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
Jun 23 08:04:31 sxoap31 xo-server[19663]:   code: 'SR_BACKEND_FAILURE_460',
Jun 23 08:04:31 sxoap31 xo-server[19663]:   params: [
Jun 23 08:04:31 sxoap31 xo-server[19663]:     '',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     ''
Jun 23 08:04:31 sxoap31 xo-server[19663]:   ],
Jun 23 08:04:31 sxoap31 xo-server[19663]:   call: undefined,
Jun 23 08:04:31 sxoap31 xo-server[19663]:   url: undefined,
Jun 23 08:04:31 sxoap31 xo-server[19663]:   task: task {
Jun 23 08:04:31 sxoap31 xo-server[19663]:     uuid: 'aeac5fce-c3c8-d3d5-eff0-f1aa3b7dc4dc',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     name_label: 'Async.VDI.list_changed_blocks',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     name_description: '',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     allowed_operations: [],
Jun 23 08:04:31 sxoap31 xo-server[19663]:     current_operations: {},
Jun 23 08:04:31 sxoap31 xo-server[19663]:     created: '20250623T13:04:30Z',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     finished: '20250623T13:04:30Z',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     status: 'failure',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     resident_on: 'OpaqueRef:e9735ec6-f9a5-da8a-0891-262ae26db402',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     progress: 1,
Jun 23 08:04:31 sxoap31 xo-server[19663]:     type: '<none/>',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     result: '',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     error_info: [
Jun 23 08:04:31 sxoap31 xo-server[19663]:       'SR_BACKEND_FAILURE_460',
Jun 23 08:04:31 sxoap31 xo-server[19663]:       '',
Jun 23 08:04:31 sxoap31 xo-server[19663]:       'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Jun 23 08:04:31 sxoap31 xo-server[19663]:       ''
Jun 23 08:04:31 sxoap31 xo-server[19663]:     ],
Jun 23 08:04:31 sxoap31 xo-server[19663]:     other_config: {},
Jun 23 08:04:31 sxoap31 xo-server[19663]:     subtask_of: 'OpaqueRef:NULL',
Jun 23 08:04:31 sxoap31 xo-server[19663]:     subtasks: [],
Jun 23 08:04:31 sxoap31 xo-server[19663]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 141))((process xapi)(filename ocaml/xapi/message_forwardi>
Jun 23 08:04:31 sxoap31 xo-server[19663]:   }
Jun 23 08:04:31 sxoap31 xo-server[19663]: }
Jun 23 08:04:36 sxoap31 xo-server[19663]: 2025-06-23T13:04:36.914Z xo:xapi:vdi WARN invalid HTTP header in response body {
Jun 23 08:04:36 sxoap31 xo-server[19663]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Jun 23 08:04:36 sxoap31 xo-server[19663]:     'content-length: 318\r\n' +
Jun 23 08:04:36 sxoap31 xo-server[19663]:     'content-type: text/html\r\n' +
Jun 23 08:04:36 sxoap31 xo-server[19663]:     'connection: close\r\n' +
Jun 23 08:04:36 sxoap31 xo-server[19663]:     'cache-control: no-cache, no-store\r\n' +
Jun 23 08:04:36 sxoap31 xo-server[19663]:     '\r\n' +
Jun 23 08:04:36 sxoap31 xo-server[19663]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representati>
Jun 23 08:04:36 sxoap31 xo-server[19663]: }
Jun 23 08:05:14 sxoap31 systemd-timesyncd[412]: Timed out waiting for reply from 23.155.40.38:123 (0.debian.pool.ntp.org).

katapaltes

Yes, you will always see those "cleanVm: incorrect backup size in metadata" warnings, and Vates has said that it's OK to ignore them. You won't see them if you "Store backup as multiple data blocks instead of a whole VHD file," but your storage will need to support the required 500-1000 files per backed up GB (and of course you'll need to evaluate whether you wish to use multiple data blocks instead of VHD files in the first place).

I re-ran my backups and the three VMs that failed backup previously have now succeeded.

katapaltes

I've updated to the latest master as well (1a7b5) and my metadata/config backups completed successfully, as did my smaller test backup. However, only 16 of 19 VMs in my main backup job completed. I am re-running the main backup now and will report back if the error recurs.

Update: The job is still running, but here is the error from the three VMs that failed previously: "Error: invalid HTTP header in response body"

katapaltes

@escape222 Secure Boot became enabled on every single UEFI VM I migrated from VMware using Xen Orchestra's Import function - well, except for the BIOS-configured VMs. I too had to disable Secure Boot after the import to get them to boot. None of my VMs ever had Secure Boot enabled while they were on VMware, so I'm unsure where this came from unless it's a default somehow.

katapaltes

@florent Hi there, and thank you for your reply. I did a file compare of the .VMX and .VMSD files and found only one difference between when the VM was configured for Native Snapshots and not configured for Native Snapshots. Note that snapshot.alwaysAllowNative = "FALSE" is the default afaik, so a .VMX file without this line in it has Native Snapshots disabled.

C:\Downloads>fc stem2p19_files-snap-false\stem2p19.vmx stem2p19_files-snap-true\stem2p19.vmx
Comparing files STEM2P19_FILES-SNAP-FALSE\stem2p19.vmx and STEM2P19_FILES-SNAP-TRUE\STEM2P19.VMX
***** STEM2P19_FILES-SNAP-FALSE\stem2p19.vmx
usb_xhci:4.parent = "-1"
snapshot.alwaysAllowNative = "FALSE"
***** STEM2P19_FILES-SNAP-TRUE\STEM2P19.VMX
usb_xhci:4.parent = "-1"
snapshot.alwaysAllowNative = "TRUE"

C:\Downloads>fc stem2p19_files-snap-false\stem2p19.vmsd stem2p19_files-snap-true\stem2p19.vmsd
Comparing files STEM2P19_FILES-SNAP-FALSE\stem2p19.vmsd and STEM2P19_FILES-SNAP-TRUE\STEM2P19.VMSD
FC: no differences encountered

I tested several scenarios for fun:

Windows VM with Native Snapshots enabled:
With VM powered down and no snapshot taken, can I still import it successfully?
Yes.
What about powered down and with a (native) snapshot taken?
Yes, with or without "Stop the Source VM" enabled.
What about with VM powered on, snapshot taken, and Stop the Source VM enabled, can I import VM successfully?
No, at the end of the import, both imported disks and the imported VM disappear.
What about with VM powered on, native snapshot enabled, no snapshot taken, and Stop the Source VM enabled, can I import VM successfully?
Yes.

Windows VM with Native Snapshots not enabled (just one single test scenario):
What about with VM powered on, snapshot taken, and Stop the Source VM enabled, can I import VM successfully?
Yes

So the problem seems to be when the VM is powered on and there's a native snapshot present.

The warm migration capability is very cool. I likely won't use it because a reboot would be required to remove VMware Tools anyway. Also, my VMs are not huge and should migrate fast.

I see what you mean by Fast Clone. That is very fast.. I'll try to get that to work with Citrix MCS. I think it may only be useful for our single-sessions hosts, not our RDSH-type hosts, but that would still be helpful.

One last thing: NetApp offers a very good ONTAP simulator for free. Now that ESXi is also free once again (LOL), you may be able to test the above scenario on your end if you like.

Please let me know if you have any questions. Cheers!

katapaltes

Greetings. Don't know if Vates is still monitoring this thread, but I've noticed an interesting edge case with the Migration tool where imports fail: When "Native Snapshots" are enabled. (Forgive me if this has been discussed above.)

We are investigating moving our Citrix servers from vSphere 8.0/ESXi 7.0 to XO/XCP-NG 8.3, and both products work very well with Citrix. In our current VMware environment, we take advantage of the VMware VAAI API which provides instant clones and "Native NetApp Snapshots," both of which are pretty awesome. (That is one thing I'll truly miss in any transition from VMware to XCP-NG.) I noticed that when Native Snapshots are enabled on a VM (by installing a NetApp-supplied .VIB on the ESXi hosts, setting some variables on the NetApp, and setting snapshot.alwaysAllowNative=TRUE on the VMs) the import from VMware to XCP-NG fails. When I disable Native Snapshots, the import succeeds.

Note: By "succeeds," I mean that the entire import process runs and the VM appears in XO/XCP-NG Center. By "fail," I mean that the process runs and no VM appears in XO/XCP-NG and the disk images that were being created in the target Storage Repository simply disappear. We're running XO d7e64 at the moment.