@MathieuRA sudo journalctl -u xo-server -n 50 Mar 27 19:27:10 xo-ce xo-server[55210]: 2026-03-27T19:27:10.657Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:27:10 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:27:10 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:27:10 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:27:10 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:27:10 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:27:10 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:27:10 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:27:10 xo-ce xo-server[55210]: } Mar 27 19:27:10 xo-ce xo-server[55210]: 2026-03-27T19:27:10.657Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> Mar 27 19:27:24 xo-ce xo-server[55210]: 2026-03-27T19:27:24.056Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:27:24 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:27:24 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:27:24 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:27:24 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:27:24 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:27:24 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:27:24 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:27:24 xo-ce xo-server[55210]: } Mar 27 19:27:24 xo-ce xo-server[55210]: 2026-03-27T19:27:24.057Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> Mar 27 19:27:41 xo-ce xo-server[55210]: 2026-03-27T19:27:41.663Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:27:41 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:27:41 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:27:41 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:27:41 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:27:41 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:27:41 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:27:41 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:27:41 xo-ce xo-server[55210]: } Mar 27 19:27:41 xo-ce xo-server[55210]: 2026-03-27T19:27:41.663Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> Mar 27 19:27:55 xo-ce xo-server[55210]: 2026-03-27T19:27:55.046Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:27:55 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:27:55 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:27:55 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:27:55 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:27:55 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:27:55 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:27:55 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:27:55 xo-ce xo-server[55210]: } Mar 27 19:27:55 xo-ce xo-server[55210]: 2026-03-27T19:27:55.046Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> Mar 27 19:28:12 xo-ce xo-server[55210]: 2026-03-27T19:28:12.633Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:28:12 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:28:12 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:28:12 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:28:12 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:28:12 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:28:12 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:28:12 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:28:12 xo-ce xo-server[55210]: } Mar 27 19:28:12 xo-ce xo-server[55210]: 2026-03-27T19:28:12.633Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> lines 1-50/50 (END)

@JSylvia007 Sorry, I'm really late to this thread, but note that backups can become problematic if the SR is something like 90% or more full. There needs to be some buffer for storage as part of the process. The fact you could copy/clone VMs means your SR is working OK, but backups are a different situation. If need be, you can always migrate VMs to other storage which is evidently what you ended up doing, which frees up extra disk space. Also backups are pretty intensive so make sure you have both enough CPU capacity and memory to handle the load. Finally. a defective SR will definitely cause issues if there are I/O errors, so watch your /var/log/SMlog for any such entries.

@florent Since both jobs were test jobs that i have been running manually, i do have an unnamed and disabled schedule on both that does look identical, so i unintentionally did have multiple jobs on the same schedule. I have since named the schedule within my test job so to each is unique. Updating to "f445b" shows improvement: I was able to replicate from pool A to Pool B, then run a backup job which was incremental. I then ran the replication job again which was incremental and did not create a new VM! Unfortunately though after this i ran the backup job again which resulted in a full backup from the replica rather than a delta, not sure why. The snapshot from the first backup job run was also not removed, leaving 2 snapshots behind, one from each backup run. [image: 1774618641909-e38c23b0-4259-4d16-9ed3-fa44d8490541-image-resized.jpeg] I then tried the process again. Ran the CR job, which was a delta (this part seemed fixed!) then ran the backup job after, same behavior, a full ran instead of a delta and the previous backup snapshot was left behind leaving the VM looking like: [image: 1774618909029-7edf1dba-e729-4b52-abcc-ffde76d8a0bb-image-resized.jpeg] So it seems one problem solved but another remains.

And this is the result [image: 1774531386302-screenshot-2026-03-21-at-09-31-45-backup.png]

@Kraken89 I will look into the transition from the previous system to the new one maybe that's the key

@ph7 that is a good news thank you for your patience and help

Great News!!

@florent I rebuilt my XCP hosting environment (everything is faster and bigger stuffed into one rack).... and this issue is now worse. The main changes to this new setup are 2x40Gb networking, faster NFS NVMe NAS, faster pool servers, more memory, much faster CR destination machine with ZFS. Running XCP 8.3 (March 2026 updates) and XO (master a2e33). Replication is leaving many attached to control domain every day with NBD connection set to 2. Changing it to 1 seems to resolve the issue (no more stuck to control domain).

@Pilow We tried that as well, same problem Also tried with a VM on the same newtork, just another VLAN, and we're seeing the same thing. At first we figured it was because one of the xcp-ng's was on a remote site which is connected through an IPSEC VPN, but that wasn't the case.

@Dezerd you just have to start-copy the Replica VM it permits the original job to keep replicating on the VM. there is no failover/failback mecanism AFAIK if you work on replica started VM, you will have to put up in place a replica going to original hosts

png @florent

@MajorP93 said: -disable HA on pool level -disable load balancer plugin -upgrade master -upgrade all other nodes -restart toolstack on master -restart toolstack on all other nodes -live migrate all VMs running on master to other node(s) -reboot master -reboot next node (live migrate all VMs running on that particular node away before doing so) -repeat until all nodes have been rebooted (one node at a time) -re-enable HA on pool level -re-enable load balancer plugin Never had any issues with that. No downtime for none of the VMs. update time again. and same issue I followed these steps -upgrade master -upgrade all other nodes -restart toolstack on master -restart toolstack on all other nodes -live migrate all VMs running on master to other node(s) -reboot master now cant migrate anything else. live migration : NOT_SUPPORTED_DURING_UPGRADE warm migration: fails and VM shuts down immediately and needs to be forced back to life CR backup to another server : NOT_SUPPORTED_DURING_UPGRADE

@olivierlambert done.

@Pilow While that could be true only 1 vm was shuffled to another SR. But all VMs now being backup with CBT enabled. Well... i just check again at backups and after the last scheduled backup at 1am all vms fell back to full backup again.

@florent XO also can't migrate a VM to a new pool with a CD in the drive... It does generate an error, but the error is unclear.

@florent Thanks for the reply. I will rebuild XO and test with current master.

@florent One last update. I reverted to Master branch (6699b) yesterday evening and the backup ran without issues overnight.