@dinhngtu Same results with the updated qemu-4.2.1-5.2.12.2~fixvwc2.1.xcpng8.3.x86_64. The Win2025 VM (without Xen tools) fails to boot correctly using stock XCP 8.3 but starts correctly with the fixvwc patch. (no actual performance or other testing done).

@dinhngtu On a quick test, the ISO boots and installs. When the VM boots from the HD it crashes (and on reboots).

Updating XCP 8.3 with qemu-4.2.1-5.2.12.2~fixvwc1.1.xcpng8.3.x86_64 solves the issue and the installed windows image boots correctly without any additional changes.

@olivierlambert @McHenry I have a similar setup... 5 host main pool (N+2) with TrueNAS NFS and a single on-site CR host (stand alone with local RAID storage) that is updated hourly from the main pool. There is also an on-site backup (for quick restore) and an off-site backup to wasabi S3 (nightly).

While CR is not a traditional backup it does allow for a quick restart of a damaged or lost VM. As my CR host is on-site it can share the same network and I can just start a VM immediately and then copy it back to the main pool. The CR also acts a secondary pool where some import redundant VMs run (ie DNS, another XO, etc). These secondary VMs are CR backed up to the main pool. If the main pool were to totally fail then VMs could be started on the CR host (with restricted resources).

S3 backups allow for a long term incremental historical storage but it takes longer to restore a large VM (but you can pick a point in time to restore from).

CR off-site is great DR option, but remember CR is not a true backup, it's just a recent copy...

@flakpyro I see the same issue.

@dinhngtu Win10 on XCP 8.3 with E5-2600v2 works just fine for me...

@florent It seems the Type: delta display during Backup fell back to a full may just be a display issue as it does seem to run a full copy.

@Pilow You could mount an iSCSI LUN on the XO VM and use a local backup remote.... If your XO VM is local to your SAN. This should offer a little more performance as it's directly attached to the XO VM doing the backup.

Using a VM running MinIO with an iSCSI LUN to act as a S3 remote is good too. It does add an intermediate step but gives you some options and flexibility.

@florent

Sep 15 10:00:01 xo1 xo-server[5582]: 2025-09-15T14:00:01.026Z xo:backups:worker INFO starting backup
Sep 15 10:00:32 xo1 xo-server[5582]: 2025-09-15T14:00:32.794Z xo:xapi:xapi-disks INFO Error in openNbdCBT XapiError: SR_BACKEND_FAILURE_460(, Failed to calculate changed blocks for given VD
Is. [opterr=Source and target VDI are unrelated], )
Sep 15 10:00:32 xo1 xo-server[5582]:     at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/_XapiError.mjs:16:12)
Sep 15 10:00:32 xo1 xo-server[5582]:     at default (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/_getTaskResult.mjs:13:29)
Sep 15 10:00:32 xo1 xo-server[5582]:     at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1073:24)
Sep 15 10:00:32 xo1 xo-server[5582]:     at file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1107:14
Sep 15 10:00:32 xo1 xo-server[5582]:     at Array.forEach (<anonymous>)
Sep 15 10:00:32 xo1 xo-server[5582]:     at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1097:12)
Sep 15 10:00:32 xo1 xo-server[5582]:     at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1270:14)
Sep 15 10:00:32 xo1 xo-server[5582]:     at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
Sep 15 10:00:32 xo1 xo-server[5582]:   code: 'SR_BACKEND_FAILURE_460',
Sep 15 10:00:32 xo1 xo-server[5582]:   params: [
Sep 15 10:00:32 xo1 xo-server[5582]:     '',
Sep 15 10:00:32 xo1 xo-server[5582]:     'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Sep 15 10:00:32 xo1 xo-server[5582]:     ''
Sep 15 10:00:32 xo1 xo-server[5582]:   ],
Sep 15 10:00:32 xo1 xo-server[5582]:   call: undefined,
Sep 15 10:00:32 xo1 xo-server[5582]:   url: undefined,
Sep 15 10:00:32 xo1 xo-server[5582]:   task: task {
Sep 15 10:00:32 xo1 xo-server[5582]:     uuid: '6265a41c-29c3-f4d9-5f0c-e52db6e0fcc9',
Sep 15 10:00:32 xo1 xo-server[5582]:     name_label: 'Async.VDI.list_changed_blocks',
Sep 15 10:00:32 xo1 xo-server[5582]:     name_description: '',
Sep 15 10:00:32 xo1 xo-server[5582]:     allowed_operations: [],
Sep 15 10:00:32 xo1 xo-server[5582]:     current_operations: {},
Sep 15 10:00:32 xo1 xo-server[5582]:     created: '20250915T14:00:32Z',
Sep 15 10:00:32 xo1 xo-server[5582]:     finished: '20250915T14:00:32Z',
Sep 15 10:00:32 xo1 xo-server[5582]:     status: 'failure',
Sep 15 10:00:32 xo1 xo-server[5582]:     resident_on: 'OpaqueRef:65b7a047-094b-4c7a-a503-2823e92b9fe4',
Sep 15 10:00:32 xo1 xo-server[5582]:     progress: 1,
Sep 15 10:00:32 xo1 xo-server[5582]:     type: '<none/>',
Sep 15 10:00:32 xo1 xo-server[5582]:     result: '',
Sep 15 10:00:32 xo1 xo-server[5582]:     error_info: [
Sep 15 10:00:32 xo1 xo-server[5582]:       'SR_BACKEND_FAILURE_460',
Sep 15 10:00:32 xo1 xo-server[5582]:       '',
Sep 15 10:00:32 xo1 xo-server[5582]:       'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Sep 15 10:00:32 xo1 xo-server[5582]:       ''
Sep 15 10:00:32 xo1 xo-server[5582]:     ],
Sep 15 10:00:32 xo1 xo-server[5582]:     other_config: {},
Sep 15 10:00:32 xo1 xo-server[5582]:     subtask_of: 'OpaqueRef:NULL',
Sep 15 10:00:32 xo1 xo-server[5582]:     subtasks: [],
Sep 15 10:00:32 xo1 xo-server[5582]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 141))((process xapi)(fi
lename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/x
api-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 188))((process xapi)(filename ocaml/xapi/rbac.ml)(line 197))((process xapi)
(filename ocaml/xapi/server_helpers.ml)(line 77)))'
Sep 15 10:00:32 xo1 xo-server[5582]:   }
Sep 15 10:00:32 xo1 xo-server[5582]: }
Sep 15 10:00:35 xo1 xo-server[5582]: 2025-09-15T14:00:35.240Z xo:xapi:vdi WARN invalid HTTP header in response body {
Sep 15 10:00:35 xo1 xo-server[5582]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'content-length: 318\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'content-type: text/html\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'connection: close\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'cache-control: no-cache, no-store\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     '\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please
 contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:ec54186c-0ab5-0390-d244-64414e9d8887; CBT metadata ]</body></html>'
Sep 15 10:00:35 xo1 xo-server[5582]: }
Sep 15 10:00:35 xo1 xo-server[5582]: 2025-09-15T14:00:35.249Z xo:xapi:xapi-disks WARN can't compute delta OpaqueRef:888ccaa8-07c6-e6f0-3bfb-3e6b5ac4da2a from OpaqueRef:ec54186c-0ab5-0390-d2
44-64414e9d8887, fallBack to a full {
Sep 15 10:00:35 xo1 xo-server[5582]:   error: Error: invalid HTTP header in response body
Sep 15 10:00:35 xo1 xo-server[5582]:       at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/vdi.mjs:36:19)
Sep 15 10:00:35 xo1 xo-server[5582]:       at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/vdi.mjs:244:5)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async XapiVhdStreamSource.init (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async #openExportStream (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/Xapi.mjs:128:7)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/Xapi.mjs:81:22)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/Xapi.mjs:191:18)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/backups/_incrementalVm.mjs:67:5
Sep 15 10:00:35 xo1 xo-server[5582]: }

@olivierlambert @florent Looks like 1471ab0c7c79fa6dca9a1598e7be2a141753ba91 (in current master) has fixed the issue.

But, during testing I found a new related issue (no error messages). Running current XO master b16d5...

During the normal CR delta backup job, two VMs shows the warning message: Backup fell back to a full but it did a delta (not full) backup.

I caused this by running a different CR backup job (for testing) that used the two same VMs but to a different SR. The new test job did a full backup and delta backup correctly. Since there is only one CBT snapshot the normal backup job did recognize that the snapshot was not for the original normal backup job and should have done a full (and started a new CBT snapshot), but it only did a short delta.

As the VMs were off, I guess it could correctly use the other CBT snapshot as no blocks had changed... Just odd that CR backup said it was going to do a full and then did a delta.

@olivierlambert fix_replication works for some... but most of my VMs now have an error:
the writer IncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot read properties of undefined (reading 'managed').

      "result": {
        "message": "Cannot read properties of undefined (reading 'managed')",
        "name": "TypeError",
        "stack": "TypeError: Cannot read properties of undefined (reading 'managed')\n    at file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:29:15\n    at Array.filter (<anonymous>)\n    at IncrementalXapiWriter.checkBaseVdis (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:26:8)\n    at file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:159:54\n    at callWriter (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:33:15)\n    at IncrementalXapiVmBackupRunner._callWriters (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:52:14)\n    at IncrementalXapiVmBackupRunner._selectBaseVm (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:158:16)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:378:5)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
      }

@olivierlambert Correct. Running commit 4944ea902ff19f172b1b86ec96ad989e322bec2c works.

@olivierlambert This happens to me too with a459015ca91c159123bb682f16237b4371a312a6.

I did open an issue https://github.com/vatesfr/xen-orchestra/issues/8969

andrew64k created this issue in vatesfr/xen-orchestra

open a459015 causes delta replication to always be full #8969

@planedrop I don't think there is a true active/standby XO setup. You can always replicate to a different pool and start/recover it there if you have a failure. You can also just have XO run on a different pool and not have it do anything (ie. backups) but still be able to access pools and restore backups.

As a last ditch total disaster recovery option, I have XO running on a laptop (Windows/VirtualBox/Linux). It is not a true duplicate/backup but does have connectivity (from the office or a VPN) to the needed hosts so I can fix/rebuild/restore. I can start over with new location, new hardware, and recover from off-site backups.

You can always deploy new XCP and new XO in minutes then restore your config... You have backups, right?

@gduperrey 8.3 Pools updated and running. RPU worked 99%...failed with a host out of memory error on the last migrations (pool is N+2, so no reason). Single hosts are updated as usual.

@halvor Yes, kind of... The pool master must be updated and rebooted first. If you move the master to a different host then you are no longer rebooting the pool master first. XOA is just a very smart client and needs the pool master to provide all of the pool information. Secondary machines will just redirect to the master, so if you try to connect XOA to one as a backup it won't work.

You just need to wait for the master to reboot and return to a functioning state and wait for XOA to automatically reconnect. The everything will start happening again.

If your master host is destroyed (ie. software corruption, hardware failure) then an existing host can be forced to take over the pool as master, but this is not a good thing to do unless the actual pool master is unrecoverable.

@halvor This happens when the Master host is unreachable (first to be rebooted). XOA should connect again after the Master host reboots and is available (which could take a while depending on the host reboot time).

@PC_123 This may sound odd, but try Debian 11. It's older but I have less issues with 11 than Debian 12 or 13.

@antalig As Ubuntu also fails it could be some problem with the card hardware design.

Yes, the i225 chips have better drivers.

@antalig There's no firmware update for the RTL812x cards. It's loaded by the driver and Realtek as not released any newer updates (from what XCP already has).

I have had issues with some multi-port cards in XCP 8.2. I have not tested them again with XCP 8.3

Does it work (or even show up) without the updated alt driver?
Try a static IPv4 assignment and see if it works.
Try booting newer Linux (like Debian 13) and see if it works.
Post a lspci

@TrapoSAMA AlmaLinux 10 does have a x64 version that still supports Xeon v2 systems. ISOs here.

When you UEFI boot normal Alma 10 on a v2, it just stops, there's no additional error.