@stormi Microcode updated on affected Gen11 i7. Running normally.

@gduperrey I jumped in all the way by mistake... I updated a wrong host, so I just did them all. Older AMD, Intel E3/E5, NUC11, etc. So far, so good. Add/migrate/backup/etc VMs are working as usual. Good for guest tools too, but mine are mostly Debian 7-11. Stuff is as usual so far.

I agree and I'll say it again, Thanks! It's not just Linux/Xen stuff. It is XCP-ng and XO that make everything work as a cohesive vertical open-source solution (some nice buzz words). Thanks to the Vates team and community that have built and support it. I look forward to ongoing continuous improvement and innovation!

@bleader Installed and running.

@bleader Updates running on several old and new intel machines (including microcode update). Working fine so far. Rolling Pool Reboot is a helpful feature.

@bleader I installed it on a bunch of busy hosts. All are fine, but none used PCI passthrough. The Rolling Pool Reboot in XO was very helpful.

@dinhngtu On a quick test, the ISO boots and installs. When the VM boots from the HD it crashes (and on reboots).

Updating XCP 8.3 with qemu-4.2.1-5.2.12.2~fixvwc1.1.xcpng8.3.x86_64 solves the issue and the installed windows image boots correctly without any additional changes.

@abudef @olivierlambert @thomas-dkmt I agree. I read block the same way... how about disable or prevent. (French? empêcher)

@julien-f @olivierlambert

Thanks for the ability to delete XO task logs feature! (XO commit f6e6e)

@gduperrey Installed and running on Intel systems, and Zen3 system that sees the microcode update.

@dinhngtu Same results with the updated qemu-4.2.1-5.2.12.2~fixvwc2.1.xcpng8.3.x86_64. The Win2025 VM (without Xen tools) fails to boot correctly using stock XCP 8.3 but starts correctly with the fixvwc patch. (no actual performance or other testing done).

@dinhngtu On a quick test, the ISO boots and installs. When the VM boots from the HD it crashes (and on reboots).

Updating XCP 8.3 with qemu-4.2.1-5.2.12.2~fixvwc1.1.xcpng8.3.x86_64 solves the issue and the installed windows image boots correctly without any additional changes.

@olivierlambert @McHenry I have a similar setup... 5 host main pool (N+2) with TrueNAS NFS and a single on-site CR host (stand alone with local RAID storage) that is updated hourly from the main pool. There is also an on-site backup (for quick restore) and an off-site backup to wasabi S3 (nightly).

While CR is not a traditional backup it does allow for a quick restart of a damaged or lost VM. As my CR host is on-site it can share the same network and I can just start a VM immediately and then copy it back to the main pool. The CR also acts a secondary pool where some import redundant VMs run (ie DNS, another XO, etc). These secondary VMs are CR backed up to the main pool. If the main pool were to totally fail then VMs could be started on the CR host (with restricted resources).

S3 backups allow for a long term incremental historical storage but it takes longer to restore a large VM (but you can pick a point in time to restore from).

CR off-site is great DR option, but remember CR is not a true backup, it's just a recent copy...

@flakpyro I see the same issue.

@dinhngtu Win10 on XCP 8.3 with E5-2600v2 works just fine for me...

@florent It seems the Type: delta display during Backup fell back to a full may just be a display issue as it does seem to run a full copy.

@Pilow You could mount an iSCSI LUN on the XO VM and use a local backup remote.... If your XO VM is local to your SAN. This should offer a little more performance as it's directly attached to the XO VM doing the backup.

Using a VM running MinIO with an iSCSI LUN to act as a S3 remote is good too. It does add an intermediate step but gives you some options and flexibility.

@florent

Sep 15 10:00:01 xo1 xo-server[5582]: 2025-09-15T14:00:01.026Z xo:backups:worker INFO starting backup
Sep 15 10:00:32 xo1 xo-server[5582]: 2025-09-15T14:00:32.794Z xo:xapi:xapi-disks INFO Error in openNbdCBT XapiError: SR_BACKEND_FAILURE_460(, Failed to calculate changed blocks for given VD
Is. [opterr=Source and target VDI are unrelated], )
Sep 15 10:00:32 xo1 xo-server[5582]:     at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/_XapiError.mjs:16:12)
Sep 15 10:00:32 xo1 xo-server[5582]:     at default (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/_getTaskResult.mjs:13:29)
Sep 15 10:00:32 xo1 xo-server[5582]:     at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1073:24)
Sep 15 10:00:32 xo1 xo-server[5582]:     at file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1107:14
Sep 15 10:00:32 xo1 xo-server[5582]:     at Array.forEach (<anonymous>)
Sep 15 10:00:32 xo1 xo-server[5582]:     at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1097:12)
Sep 15 10:00:32 xo1 xo-server[5582]:     at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1270:14)
Sep 15 10:00:32 xo1 xo-server[5582]:     at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
Sep 15 10:00:32 xo1 xo-server[5582]:   code: 'SR_BACKEND_FAILURE_460',
Sep 15 10:00:32 xo1 xo-server[5582]:   params: [
Sep 15 10:00:32 xo1 xo-server[5582]:     '',
Sep 15 10:00:32 xo1 xo-server[5582]:     'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Sep 15 10:00:32 xo1 xo-server[5582]:     ''
Sep 15 10:00:32 xo1 xo-server[5582]:   ],
Sep 15 10:00:32 xo1 xo-server[5582]:   call: undefined,
Sep 15 10:00:32 xo1 xo-server[5582]:   url: undefined,
Sep 15 10:00:32 xo1 xo-server[5582]:   task: task {
Sep 15 10:00:32 xo1 xo-server[5582]:     uuid: '6265a41c-29c3-f4d9-5f0c-e52db6e0fcc9',
Sep 15 10:00:32 xo1 xo-server[5582]:     name_label: 'Async.VDI.list_changed_blocks',
Sep 15 10:00:32 xo1 xo-server[5582]:     name_description: '',
Sep 15 10:00:32 xo1 xo-server[5582]:     allowed_operations: [],
Sep 15 10:00:32 xo1 xo-server[5582]:     current_operations: {},
Sep 15 10:00:32 xo1 xo-server[5582]:     created: '20250915T14:00:32Z',
Sep 15 10:00:32 xo1 xo-server[5582]:     finished: '20250915T14:00:32Z',
Sep 15 10:00:32 xo1 xo-server[5582]:     status: 'failure',
Sep 15 10:00:32 xo1 xo-server[5582]:     resident_on: 'OpaqueRef:65b7a047-094b-4c7a-a503-2823e92b9fe4',
Sep 15 10:00:32 xo1 xo-server[5582]:     progress: 1,
Sep 15 10:00:32 xo1 xo-server[5582]:     type: '<none/>',
Sep 15 10:00:32 xo1 xo-server[5582]:     result: '',
Sep 15 10:00:32 xo1 xo-server[5582]:     error_info: [
Sep 15 10:00:32 xo1 xo-server[5582]:       'SR_BACKEND_FAILURE_460',
Sep 15 10:00:32 xo1 xo-server[5582]:       '',
Sep 15 10:00:32 xo1 xo-server[5582]:       'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Sep 15 10:00:32 xo1 xo-server[5582]:       ''
Sep 15 10:00:32 xo1 xo-server[5582]:     ],
Sep 15 10:00:32 xo1 xo-server[5582]:     other_config: {},
Sep 15 10:00:32 xo1 xo-server[5582]:     subtask_of: 'OpaqueRef:NULL',
Sep 15 10:00:32 xo1 xo-server[5582]:     subtasks: [],
Sep 15 10:00:32 xo1 xo-server[5582]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 141))((process xapi)(fi
lename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/x
api-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 188))((process xapi)(filename ocaml/xapi/rbac.ml)(line 197))((process xapi)
(filename ocaml/xapi/server_helpers.ml)(line 77)))'
Sep 15 10:00:32 xo1 xo-server[5582]:   }
Sep 15 10:00:32 xo1 xo-server[5582]: }
Sep 15 10:00:35 xo1 xo-server[5582]: 2025-09-15T14:00:35.240Z xo:xapi:vdi WARN invalid HTTP header in response body {
Sep 15 10:00:35 xo1 xo-server[5582]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'content-length: 318\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'content-type: text/html\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'connection: close\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'cache-control: no-cache, no-store\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     '\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please
 contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:ec54186c-0ab5-0390-d244-64414e9d8887; CBT metadata ]</body></html>'
Sep 15 10:00:35 xo1 xo-server[5582]: }
Sep 15 10:00:35 xo1 xo-server[5582]: 2025-09-15T14:00:35.249Z xo:xapi:xapi-disks WARN can't compute delta OpaqueRef:888ccaa8-07c6-e6f0-3bfb-3e6b5ac4da2a from OpaqueRef:ec54186c-0ab5-0390-d2
44-64414e9d8887, fallBack to a full {
Sep 15 10:00:35 xo1 xo-server[5582]:   error: Error: invalid HTTP header in response body
Sep 15 10:00:35 xo1 xo-server[5582]:       at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/vdi.mjs:36:19)
Sep 15 10:00:35 xo1 xo-server[5582]:       at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/vdi.mjs:244:5)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async XapiVhdStreamSource.init (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async #openExportStream (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/Xapi.mjs:128:7)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/Xapi.mjs:81:22)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/Xapi.mjs:191:18)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/backups/_incrementalVm.mjs:67:5
Sep 15 10:00:35 xo1 xo-server[5582]: }

@olivierlambert @florent Looks like 1471ab0c7c79fa6dca9a1598e7be2a141753ba91 (in current master) has fixed the issue.

But, during testing I found a new related issue (no error messages). Running current XO master b16d5...

During the normal CR delta backup job, two VMs shows the warning message: Backup fell back to a full but it did a delta (not full) backup.

I caused this by running a different CR backup job (for testing) that used the two same VMs but to a different SR. The new test job did a full backup and delta backup correctly. Since there is only one CBT snapshot the normal backup job did recognize that the snapshot was not for the original normal backup job and should have done a full (and started a new CBT snapshot), but it only did a short delta.

As the VMs were off, I guess it could correctly use the other CBT snapshot as no blocks had changed... Just odd that CR backup said it was going to do a full and then did a delta.

@olivierlambert fix_replication works for some... but most of my VMs now have an error:
the writer IncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot read properties of undefined (reading 'managed').

      "result": {
        "message": "Cannot read properties of undefined (reading 'managed')",
        "name": "TypeError",
        "stack": "TypeError: Cannot read properties of undefined (reading 'managed')\n    at file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:29:15\n    at Array.filter (<anonymous>)\n    at IncrementalXapiWriter.checkBaseVdis (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:26:8)\n    at file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:159:54\n    at callWriter (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:33:15)\n    at IncrementalXapiVmBackupRunner._callWriters (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:52:14)\n    at IncrementalXapiVmBackupRunner._selectBaseVm (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:158:16)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:378:5)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
      }