@stormi installed updates. Did some backups and VM migrations between various storages (bad and qcow2) looks good so far.

@stormi said:

@bufanda I'm being told it's expected to have it the first time after the update, but in theory the next ones should be not be fulls. Can you try?

Just checked and the VM I was testing with was part of two backups and it seems that when one runs and the second starts that it will fall back. I removed the VM now from one backup and with being only member of one backup job it looks good then. Will keep an eye on it.

@acebmxer purge snapshots is active since I created the backup job over a year ago. I always enable purge snapshots on backup jobs.

Installed on my usual lab setup works fine so far although I have the "Fell back to full backup" message for qcow2 VMs

Upgraded my usual pool. VM migrations during reboot worked without issues. So far everything works

@stormi Installed on two node Pool. Both nodes are HP EliteDesk 800 G3 Miini. 10 Linux VMs with Secure Boot enabled and 1 Windows Server 2019 with SecureBoot and vTPM enabled. No issues. Migration between nodes worked flawlessly too.

@gduperrey Installed on my Lab-Pool with teo HP EliteDesk 800 G3's no issues during and after upgrade. migration, creating, and deleteing of VMs wihtout issues too.

@sid I made that request a while ago, and tried to look into myself too, but the current API of XenOrchestra just doesn't support it, there are many pieces missing from what I could see. I hope with XenOrchestra 6 the API wil support it though.

Installed on a 2 Node pool consisting of 2
HP EliteDesk 300 G3 Mini with

• 1x i7 6700T & 1x i5 6500T
• 32GB RAM each
• NFS VM SR
• iSCSI VM SR
• various SMB ISO SR

VMs migrated during reboot withput issues, also no issues with migration after updates completed with all nodes rebooted. Alos no issues found with VMs. Primarily AlmaLinux 9 and FreeBSD 13.

@noritaka971 You sure it does always a full backup. I have this isuue for ages and looking at the backup protocols it's only doing the deltas, never a full backup when not supposed to.

@dthenot Only have 2 Test VMs currently not running with qcow2. Nothing production and nothing crucial. It's a lab pool. As I said just to understand the behavior when I test the new updates. But thanks for the details. Really appreciated

@dthenot No worries, just wanted to make sure I understand the expected behavior when I had the qcow2 patches installed.

Did install on my lab pool

2 HP EliteDesk 800 G3 mini

One with i7 6700T, one with i5 6500T
Both 32GB RAM
iSCSI SR, NFS SR and local NVMe SR. VMs from all SR run. Migration during update worked. Migration of VMs after update worked. Backups (exporting VMs) is working
Didn't see any issue so far.

One thing the pool had the qcow2 patches installed since XAPI and sm packages gto updated I guess it won't support qcow2 at the moment right?

Hi,

I tested build 48bf9c9 and backups whether they using NBD or not seem not to start transfers at all.

This is the log output without NBD:

2025-07-01T05:09:04.436Z xo:backups:worker INFO starting backup
2025-07-01T05:09:06.481Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
}
2025-07-01T05:09:06.490Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
}
2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/20250701T050331Z.vhd'
}
2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
}
2025-07-01T05:09:08.029Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
}
2025-07-01T05:09:52.631Z xo:backups:worker WARN possibly unhandled rejection {
  error: TypeError: this.source.readBlock is not a function
      at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
      at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
      at buildDiskBlockGenerator.next (<anonymous>)
      at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Promise.all (index 0)
}
2025-07-01T05:09:52.633Z xo:backups:worker WARN possibly unhandled rejection {
  error: TypeError: this.source.readBlock is not a function
      at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
      at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
      at buildDiskBlockGenerator.next (<anonymous>)
      at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Promise.all (index 0)
}
2025-07-01T05:09:52.634Z xo:backups:worker WARN possibly unhandled rejection {
  error: TypeError: this.source.readBlock is not a function
      at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
      at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
      at buildDiskBlockGenerator.next (<anonymous>)
      at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Promise.all (index 0)
}

And smae VM when trying to use NBD

2025-07-01T05:02:49.119Z xo:backups:worker INFO starting backup
2025-07-01T05:02:51.256Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.268Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.270Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.627Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
}
2025-07-01T05:06:04.924Z xo:backups:worker INFO starting backup
2025-07-01T05:06:09.185Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
}
2025-07-01T05:06:09.852Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
}
2025-07-01T05:06:18.711Z xo:xapi:vdi WARN invalid HTTP header in response body {
  body: 'HTTP/1.1 500 Internal Error\r\n' +
    'content-length: 346\r\n' +
    'content-type: text/html\r\n' +
    'connection: close\r\n' +
    'cache-control: no-cache, no-store\r\n' +
    '\r\n' +
    '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>Db_exn.Read_missing_uuid(&quot;VDI&quot;, &quot;&quot;, &quot;OpaqueRef:9805714f-7a36-6c2c-c194-9f08baf519df&quot;)</body></html>'
}

@stormi Oh Damn, bad timing I shouldn't have updated my 8.2 pool last sunday

Updated my Lab pool consisting of 2 Nodes this morning
Rebooted master first, then slave. No issues VMs migrated between reboots.
Couldn't see any issue til now.

Installed the patches on my lab pool and both hosts are up and running and no issues so far.

@andriy.sultanov
I ran it for a minute.

rrd2csv.txt

@andriy.sultanov
Same issue after another reboot. The stats on host just flatline. No CPU usage or anything.
See screenshot

I attached log files
daemon.log.txt xcp-rrdd-plugins.log.txt xensource.log.txt

Installed on my Test pool with 2 HP EliteDesk 800 G3 mini. Except the no stats in XenOrchestra. Did some VM migragration between hosts, reboot testes (only Linux VMs though) and so far no issues.