@MathieuRA sudo journalctl -u xo-server -n 50 Mar 27 19:27:10 xo-ce xo-server[55210]: 2026-03-27T19:27:10.657Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:27:10 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:27:10 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:27:10 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:27:10 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:27:10 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:27:10 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:27:10 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:27:10 xo-ce xo-server[55210]: } Mar 27 19:27:10 xo-ce xo-server[55210]: 2026-03-27T19:27:10.657Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> Mar 27 19:27:24 xo-ce xo-server[55210]: 2026-03-27T19:27:24.056Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:27:24 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:27:24 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:27:24 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:27:24 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:27:24 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:27:24 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:27:24 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:27:24 xo-ce xo-server[55210]: } Mar 27 19:27:24 xo-ce xo-server[55210]: 2026-03-27T19:27:24.057Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> Mar 27 19:27:41 xo-ce xo-server[55210]: 2026-03-27T19:27:41.663Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:27:41 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:27:41 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:27:41 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:27:41 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:27:41 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:27:41 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:27:41 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:27:41 xo-ce xo-server[55210]: } Mar 27 19:27:41 xo-ce xo-server[55210]: 2026-03-27T19:27:41.663Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> Mar 27 19:27:55 xo-ce xo-server[55210]: 2026-03-27T19:27:55.046Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:27:55 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:27:55 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:27:55 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:27:55 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:27:55 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:27:55 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:27:55 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:27:55 xo-ce xo-server[55210]: } Mar 27 19:27:55 xo-ce xo-server[55210]: 2026-03-27T19:27:55.046Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> Mar 27 19:28:12 xo-ce xo-server[55210]: 2026-03-27T19:28:12.633Z xo:rest-api:error-handler ERROR Cannot read properties of undefined (reading 'vms'> Mar 27 19:28:12 xo-ce xo-server[55210]: error: TypeError: Cannot read properties of undefined (reading 'vms') Mar 27 19:28:12 xo-ce xo-server[55210]: at BackupLogService.isVmInBackupLog (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/backup-lo> Mar 27 19:28:12 xo-ce xo-server[55210]: at filter (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.mjs:228:40) Mar 27 19:28:12 xo-ce xo-server[55210]: at Array.filter (<anonymous>) Mar 27 19:28:12 xo-ce xo-server[55210]: at Xo.getBackupNgLogsSorted (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/backups-ng-lo> Mar 27 19:28:12 xo-ce xo-server[55210]: at VmService.#getBackupsInfo (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/vms/vm.service.m> Mar 27 19:28:12 xo-ce xo-server[55210]: at promiseWriteInStream (file:///opt/xen-orchestra/@xen-orchestra/rest-api/dist/helpers/utils.helper.> Mar 27 19:28:12 xo-ce xo-server[55210]: } Mar 27 19:28:12 xo-ce xo-server[55210]: 2026-03-27T19:28:12.633Z xo:rest-api:error-handler INFO [GET] /vms/138538a8-ef52-4d0a-4433-5ebb31d7e152/das> lines 1-50/50 (END)

@carloum70 are the scopes email and uid provided by the source (are they delivered to xo by surfnet)? By default only the openid scope is provided, without email or uid https://servicedesk.surf.nl/wiki/spaces/IAM/pages/74226151/Connect+using+OpenID+Connect

@MajorP93 This was also my thought, the disconnection method you mentioned, I was just wanting some feedback about that. I do have to be a bit careful because the migration must happen and then the VM on VMware must go back online while I work with the vendor to migrate all the rest of the data. As for the bigger than 2TB issue, that won't be an issue for me. This VM is HUGE but it's because it's thick provisioned and no longer needs to be, the new VM will have like 8 disks but none of them will be over 2TB (most will be less than 100GB). So I am thinking using V2V and just powering off the VM on VMware, disconnect the unneeded VDIs, then migrate, then reconnect those VDIs and power the VM back on.

I can confirm that when using Citrix/Xenserver guest utilities version 8.4 (https://github.com/xenserver/xe-guest-utilities/releases/tag/v8.4.0) memory ballooning / DMC is working fine. After live migration the RAM of the linux guest is expanded to dynamic_max again. So this issue was in fact caused by Rust based xen-guest-agent. For now I'll keep using Citrix/Xenserver guest utilities on my Linux guests until the feature is implemented in Vates rust-based guest utilities. Best regards

@JSylvia007 Sorry, I'm really late to this thread, but note that backups can become problematic if the SR is something like 90% or more full. There needs to be some buffer for storage as part of the process. The fact you could copy/clone VMs means your SR is working OK, but backups are a different situation. If need be, you can always migrate VMs to other storage which is evidently what you ended up doing, which frees up extra disk space. Also backups are pretty intensive so make sure you have both enough CPU capacity and memory to handle the load. Finally. a defective SR will definitely cause issues if there are I/O errors, so watch your /var/log/SMlog for any such entries.

@florent Since both jobs were test jobs that i have been running manually, i do have an unnamed and disabled schedule on both that does look identical, so i unintentionally did have multiple jobs on the same schedule. I have since named the schedule within my test job so to each is unique. Updating to "f445b" shows improvement: I was able to replicate from pool A to Pool B, then run a backup job which was incremental. I then ran the replication job again which was incremental and did not create a new VM! Unfortunately though after this i ran the backup job again which resulted in a full backup from the replica rather than a delta, not sure why. The snapshot from the first backup job run was also not removed, leaving 2 snapshots behind, one from each backup run. [image: 1774618641909-e38c23b0-4259-4d16-9ed3-fa44d8490541-image-resized.jpeg] I then tried the process again. Ran the CR job, which was a delta (this part seemed fixed!) then ran the backup job after, same behavior, a full ran instead of a delta and the previous backup snapshot was left behind leaving the VM looking like: [image: 1774618909029-7edf1dba-e729-4b52-abcc-ffde76d8a0bb-image-resized.jpeg] So it seems one problem solved but another remains.

Thanks for the quick answer @mathieura

@TeddyAstie I hadn't seen anything special was needed in a VM other that a kernel new enough for support which 6.17 has, is there more to the story with adding things to Ubuntu?

I have two PE R630 and one PE R730xd. The two R630s show the IPMI data in XO but the R730xd does not. If I run ipmitool sensor from an ssh session, I get very similar data back on the 730 and 630. This isn't critical to me, just letting you know that there are cases that functioning ipmitool on a Dell does not seem to be enough to show it. There are some differences in the labels. The R730 has "Fan1 RPM", "Fan2 RPM" while the R630 has "Fan1A", "Fan2A", "Fan1B". Most of the other interesting fields seem to have the same names but the order is a bit different and I didn't do an exhaustive comparison. It's very cool that you are surfacing this data now.

Ping @Team-XO-Backend

And this is the result [image: 1774531386302-screenshot-2026-03-21-at-09-31-45-backup.png]

@hoh This is SOOO long in the tooth........... Always was annoying. Thx for findinf a fix, looking forward to it.

@TeddyAstie Attached is the output you requested xen-cpuid -p.txt

@acebmxer I haven't tried this yet, but liking the menu you just showed!

UPDATE: its been 2 days and as I suspected the VDI is still waiting to coalesce at a length of 1, The GC does not seem to be cleaning it up. Does anyone have suggestions for how to remove this orphaned VDI and base copy. I'm not interested in saving the VM just cleaning up the remnants of the botched migration and removal.

@Kraken89 I will look into the transition from the previous system to the new one maybe that's the key

I am trying to setup the same VxLAN Private Network between two pools. I have been running XO from sources (commit 813514821) for a while now and encountered the same error "No PIF found in center". Therefore I updated to the latest commit 60ba5070c (still using XO v5) and now I was able to create a VxLAN private network that spans two pools without an error. I checked the networks for both pools and noticed that two networks were created per pool, so I was able to create it despite the error from the older XO version. However, using either VxLAN to connect two VMs in different pools doesn't work. Pinging a VM over that VxLAN in the same pool and host works, but pinging a VM from another Pool doesn't work. This is also the case when using GRE, Looking at the ARP Table, it seems that the VMs from different pool do show up, but as incomplete: [image: 1774432891674-screenshot-from-2026-03-25-10-59-34.png] 172.30.0.10 is the VM in the same pool as work (172.30.0.30), 172.30.0.20 is the VM from a different pool. I manually set the MTU size in each VM to 1450 just to be sure. I have two pools with one server each. This is due to CPU feature mismatch. Each host is connected via a bond of two SFP+ Ports (which have default MTU 1500) to a switch that allows all VLANs. Using normal VLAN Networks does work with multiple pools.

@acebmxer Yes, 22 or 24 should work correctly. Some early versions of 24 had issues, but not currently. I'm running 24 and it is working.