@florent

Sep 15 10:00:01 xo1 xo-server[5582]: 2025-09-15T14:00:01.026Z xo:backups:worker INFO starting backup
Sep 15 10:00:32 xo1 xo-server[5582]: 2025-09-15T14:00:32.794Z xo:xapi:xapi-disks INFO Error in openNbdCBT XapiError: SR_BACKEND_FAILURE_460(, Failed to calculate changed blocks for given VD
Is. [opterr=Source and target VDI are unrelated], )
Sep 15 10:00:32 xo1 xo-server[5582]:     at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/_XapiError.mjs:16:12)
Sep 15 10:00:32 xo1 xo-server[5582]:     at default (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/_getTaskResult.mjs:13:29)
Sep 15 10:00:32 xo1 xo-server[5582]:     at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1073:24)
Sep 15 10:00:32 xo1 xo-server[5582]:     at file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1107:14
Sep 15 10:00:32 xo1 xo-server[5582]:     at Array.forEach (<anonymous>)
Sep 15 10:00:32 xo1 xo-server[5582]:     at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1097:12)
Sep 15 10:00:32 xo1 xo-server[5582]:     at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202509150942/packages/xen-api/index.mjs:1270:14)
Sep 15 10:00:32 xo1 xo-server[5582]:     at process.processTicksAndRejections (node:internal/process/task_queues:105:5) {
Sep 15 10:00:32 xo1 xo-server[5582]:   code: 'SR_BACKEND_FAILURE_460',
Sep 15 10:00:32 xo1 xo-server[5582]:   params: [
Sep 15 10:00:32 xo1 xo-server[5582]:     '',
Sep 15 10:00:32 xo1 xo-server[5582]:     'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Sep 15 10:00:32 xo1 xo-server[5582]:     ''
Sep 15 10:00:32 xo1 xo-server[5582]:   ],
Sep 15 10:00:32 xo1 xo-server[5582]:   call: undefined,
Sep 15 10:00:32 xo1 xo-server[5582]:   url: undefined,
Sep 15 10:00:32 xo1 xo-server[5582]:   task: task {
Sep 15 10:00:32 xo1 xo-server[5582]:     uuid: '6265a41c-29c3-f4d9-5f0c-e52db6e0fcc9',
Sep 15 10:00:32 xo1 xo-server[5582]:     name_label: 'Async.VDI.list_changed_blocks',
Sep 15 10:00:32 xo1 xo-server[5582]:     name_description: '',
Sep 15 10:00:32 xo1 xo-server[5582]:     allowed_operations: [],
Sep 15 10:00:32 xo1 xo-server[5582]:     current_operations: {},
Sep 15 10:00:32 xo1 xo-server[5582]:     created: '20250915T14:00:32Z',
Sep 15 10:00:32 xo1 xo-server[5582]:     finished: '20250915T14:00:32Z',
Sep 15 10:00:32 xo1 xo-server[5582]:     status: 'failure',
Sep 15 10:00:32 xo1 xo-server[5582]:     resident_on: 'OpaqueRef:65b7a047-094b-4c7a-a503-2823e92b9fe4',
Sep 15 10:00:32 xo1 xo-server[5582]:     progress: 1,
Sep 15 10:00:32 xo1 xo-server[5582]:     type: '<none/>',
Sep 15 10:00:32 xo1 xo-server[5582]:     result: '',
Sep 15 10:00:32 xo1 xo-server[5582]:     error_info: [
Sep 15 10:00:32 xo1 xo-server[5582]:       'SR_BACKEND_FAILURE_460',
Sep 15 10:00:32 xo1 xo-server[5582]:       '',
Sep 15 10:00:32 xo1 xo-server[5582]:       'Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]',
Sep 15 10:00:32 xo1 xo-server[5582]:       ''
Sep 15 10:00:32 xo1 xo-server[5582]:     ],
Sep 15 10:00:32 xo1 xo-server[5582]:     other_config: {},
Sep 15 10:00:32 xo1 xo-server[5582]:     subtask_of: 'OpaqueRef:NULL',
Sep 15 10:00:32 xo1 xo-server[5582]:     subtasks: [],
Sep 15 10:00:32 xo1 xo-server[5582]:     backtrace: '(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 141))((process xapi)(fi
lename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/x
api-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 188))((process xapi)(filename ocaml/xapi/rbac.ml)(line 197))((process xapi)
(filename ocaml/xapi/server_helpers.ml)(line 77)))'
Sep 15 10:00:32 xo1 xo-server[5582]:   }
Sep 15 10:00:32 xo1 xo-server[5582]: }
Sep 15 10:00:35 xo1 xo-server[5582]: 2025-09-15T14:00:35.240Z xo:xapi:vdi WARN invalid HTTP header in response body {
Sep 15 10:00:35 xo1 xo-server[5582]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'content-length: 318\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'content-type: text/html\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'connection: close\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     'cache-control: no-cache, no-store\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     '\r\n' +
Sep 15 10:00:35 xo1 xo-server[5582]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please
 contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:ec54186c-0ab5-0390-d244-64414e9d8887; CBT metadata ]</body></html>'
Sep 15 10:00:35 xo1 xo-server[5582]: }
Sep 15 10:00:35 xo1 xo-server[5582]: 2025-09-15T14:00:35.249Z xo:xapi:xapi-disks WARN can't compute delta OpaqueRef:888ccaa8-07c6-e6f0-3bfb-3e6b5ac4da2a from OpaqueRef:ec54186c-0ab5-0390-d2
44-64414e9d8887, fallBack to a full {
Sep 15 10:00:35 xo1 xo-server[5582]:   error: Error: invalid HTTP header in response body
Sep 15 10:00:35 xo1 xo-server[5582]:       at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/vdi.mjs:36:19)
Sep 15 10:00:35 xo1 xo-server[5582]:       at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/vdi.mjs:244:5)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async XapiVhdStreamSource.init (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async #openExportStream (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/Xapi.mjs:128:7)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/Xapi.mjs:81:22)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/xapi/disks/Xapi.mjs:191:18)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Sep 15 10:00:35 xo1 xo-server[5582]:       at async file:///opt/xo/xo-builds/xen-orchestra-202509150942/@xen-orchestra/backups/_incrementalVm.mjs:67:5
Sep 15 10:00:35 xo1 xo-server[5582]: }

@olivierlambert @florent Looks like 1471ab0c7c79fa6dca9a1598e7be2a141753ba91 (in current master) has fixed the issue.

But, during testing I found a new related issue (no error messages). Running current XO master b16d5...

During the normal CR delta backup job, two VMs shows the warning message: Backup fell back to a full but it did a delta (not full) backup.

I caused this by running a different CR backup job (for testing) that used the two same VMs but to a different SR. The new test job did a full backup and delta backup correctly. Since there is only one CBT snapshot the normal backup job did recognize that the snapshot was not for the original normal backup job and should have done a full (and started a new CBT snapshot), but it only did a short delta.

As the VMs were off, I guess it could correctly use the other CBT snapshot as no blocks had changed... Just odd that CR backup said it was going to do a full and then did a delta.

@olivierlambert fix_replication works for some... but most of my VMs now have an error:
the writer IncrementalXapiWriter has failed the step writer.checkBaseVdis() with error Cannot read properties of undefined (reading 'managed').

      "result": {
        "message": "Cannot read properties of undefined (reading 'managed')",
        "name": "TypeError",
        "stack": "TypeError: Cannot read properties of undefined (reading 'managed')\n    at file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:29:15\n    at Array.filter (<anonymous>)\n    at IncrementalXapiWriter.checkBaseVdis (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_writers/IncrementalXapiWriter.mjs:26:8)\n    at file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:159:54\n    at callWriter (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:33:15)\n    at IncrementalXapiVmBackupRunner._callWriters (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:52:14)\n    at IncrementalXapiVmBackupRunner._selectBaseVm (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:158:16)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:378:5)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202509150025/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
      }

@olivierlambert Correct. Running commit 4944ea902ff19f172b1b86ec96ad989e322bec2c works.

@olivierlambert This happens to me too with a459015ca91c159123bb682f16237b4371a312a6.

I did open an issue https://github.com/vatesfr/xen-orchestra/issues/8969

andrew64k created this issue in vatesfr/xen-orchestra

open a459015 causes delta replication to always be full #8969

@planedrop I don't think there is a true active/standby XO setup. You can always replicate to a different pool and start/recover it there if you have a failure. You can also just have XO run on a different pool and not have it do anything (ie. backups) but still be able to access pools and restore backups.

As a last ditch total disaster recovery option, I have XO running on a laptop (Windows/VirtualBox/Linux). It is not a true duplicate/backup but does have connectivity (from the office or a VPN) to the needed hosts so I can fix/rebuild/restore. I can start over with new location, new hardware, and recover from off-site backups.

You can always deploy new XCP and new XO in minutes then restore your config... You have backups, right?

@gduperrey 8.3 Pools updated and running. RPU worked 99%...failed with a host out of memory error on the last migrations (pool is N+2, so no reason). Single hosts are updated as usual.

@halvor Yes, kind of... The pool master must be updated and rebooted first. If you move the master to a different host then you are no longer rebooting the pool master first. XOA is just a very smart client and needs the pool master to provide all of the pool information. Secondary machines will just redirect to the master, so if you try to connect XOA to one as a backup it won't work.

You just need to wait for the master to reboot and return to a functioning state and wait for XOA to automatically reconnect. The everything will start happening again.

If your master host is destroyed (ie. software corruption, hardware failure) then an existing host can be forced to take over the pool as master, but this is not a good thing to do unless the actual pool master is unrecoverable.

@halvor This happens when the Master host is unreachable (first to be rebooted). XOA should connect again after the Master host reboots and is available (which could take a while depending on the host reboot time).

@PC_123 This may sound odd, but try Debian 11. It's older but I have less issues with 11 than Debian 12 or 13.

@antalig As Ubuntu also fails it could be some problem with the card hardware design.

Yes, the i225 chips have better drivers.

@antalig There's no firmware update for the RTL812x cards. It's loaded by the driver and Realtek as not released any newer updates (from what XCP already has).

I have had issues with some multi-port cards in XCP 8.2. I have not tested them again with XCP 8.3

Does it work (or even show up) without the updated alt driver?
Try a static IPv4 assignment and see if it works.
Try booting newer Linux (like Debian 13) and see if it works.
Post a lspci

@TrapoSAMA AlmaLinux 10 does have a x64 version that still supports Xeon v2 systems. ISOs here.

When you UEFI boot normal Alma 10 on a v2, it just stops, there's no additional error.

@dnikola almost 300 reports of Temperature above threshold. Lots of segfault and general protection errors in Dom0.

It's looking bad for the physical environment and hardware in use. Overheating, CPU issues, Memory issues, USB issues, etc... They can all be related to form an unstable system, very bad for a VM server.

@dnikola That machine is a hot mess.

@olivierlambert No IP record... Using Debian 11 with Management agent 1.0.0-proto-0.4.0. It's a non-standard interface setup with the IPv4/IPv6 assigned to the bridge interface. The agent does not report any addresses up to XO.

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP group default qlen 1000
    link/ether 32:a9:24:28:18:56 brd ff:ff:ff:ff:ff:ff
3: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:31:46:59:66:a1 brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.33/24 brd 192.168.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 2000::5031:46ff:fe59:66a1/64 scope global dynamic mngtmpaddr
       valid_lft 2591833sec preferred_lft 604633sec
    inet6 fe80::5031:46ff:fe59:66a1/64 scope link
       valid_lft forever preferred_lft forever

@gduperrey I updated my little AMD Ryzen 5 5600U (Zen3) and it's running great!

As for the important VM to VM network performance, using Debian 13 and iperf3 single thread, before (update not enabled) is about 7.2-8Gb/sec. After the update (xen-platform-pci-bar-uc=false) I get about 10.1-13Gb/sec. So that's about a 40-60% improvement. This brings it in line with similar small Intel systems.

I did not see any change (or problem) setting it on my small test Intel system (getting about 10.1-10.8Gb/sec).

FYI: Remember, after the config change and xe-toolstack-restart, restart the VMs!

@gduperrey Updated and running. Single hosts were fine. No AMD testing.

Upgrading a busy pool seems to have had some odd issues with VM migration, but all seems to be running fine now. I had upgraded the pool from 8.2.1 to 8.3 last month and everything has been fine. This time (after rebooting pool master) while trying to migrate guests so I could reboot hosts got a few XO errors like:

xo:api WARN admin | vm.migrate(...) [1s] =!> XapiError: INTERNAL_ERROR(Object with type VM and id bd38ee46-2701-3022-ec61-03bf3ffbdcc9/config does not exist in xenopsd)
xo:api WARN admin | vm.migrate(...) [2s] =!> XapiError: INTERNAL_ERROR(Object with type VM and id 235de1d7-832e-f1a7-fa1c-a45877aab8f6/config does not exist in xenopsd)

It was only a few on one host. I can NOT confirm this is related to the updates as I'm not having problems now. After manually rebooting the host and stuck guests, things were ok again.

@stormi Unless new Dom0 is Kernel 6.15 or newer (with updates), this will be an ongoing problem to support the newest hardware as they seem to keep updating the chips (you can't stop progress).

The current r8125 driver in 8.2/8.3 supports the 8125 and some 8126 chips, but not 8127. Other than supporting the new hardware versions of the chips, I have not seen any real complaints posted about the current r8125 driver (may be there are some Vates tickets?)

While the risk on the new driver is low, it is different (not just an update). I agree that it would be best to have r8125-module-alt for the new code and then r8126-module and r8127-module.

Note, that the new r8126-module driver should not be installed without the r8125-module-alt as the existing r8125-module would conflict with the new r8126-module driver.

The work is already done (compiled and running)... I'm just waiting for Realtek to fix some issues with the new code before it's worth fully testing the drivers (issues already acknowledged by Realtek).

@cairoti The data that is sent and stored on the remote is locally encrypted. The volume as a whole is not encrypted.