@dthenot Only have 2 Test VMs currently not running with qcow2. Nothing production and nothing crucial. It's a lab pool. As I said just to understand the behavior when I test the new updates. But thanks for the details. Really appreciated

@dthenot No worries, just wanted to make sure I understand the expected behavior when I had the qcow2 patches installed.

Did install on my lab pool

2 HP EliteDesk 800 G3 mini

One with i7 6700T, one with i5 6500T
Both 32GB RAM
iSCSI SR, NFS SR and local NVMe SR. VMs from all SR run. Migration during update worked. Migration of VMs after update worked. Backups (exporting VMs) is working
Didn't see any issue so far.

One thing the pool had the qcow2 patches installed since XAPI and sm packages gto updated I guess it won't support qcow2 at the moment right?

Hi,

I tested build 48bf9c9 and backups whether they using NBD or not seem not to start transfers at all.

This is the log output without NBD:

2025-07-01T05:09:04.436Z xo:backups:worker INFO starting backup
2025-07-01T05:09:06.481Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
}
2025-07-01T05:09:06.490Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
}
2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/20250701T050331Z.vhd'
}
2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
}
2025-07-01T05:09:08.029Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
}
2025-07-01T05:09:52.631Z xo:backups:worker WARN possibly unhandled rejection {
  error: TypeError: this.source.readBlock is not a function
      at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
      at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
      at buildDiskBlockGenerator.next (<anonymous>)
      at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Promise.all (index 0)
}
2025-07-01T05:09:52.633Z xo:backups:worker WARN possibly unhandled rejection {
  error: TypeError: this.source.readBlock is not a function
      at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
      at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
      at buildDiskBlockGenerator.next (<anonymous>)
      at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Promise.all (index 0)
}
2025-07-01T05:09:52.634Z xo:backups:worker WARN possibly unhandled rejection {
  error: TypeError: this.source.readBlock is not a function
      at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
      at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
      at buildDiskBlockGenerator.next (<anonymous>)
      at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Promise.all (index 0)
}

And smae VM when trying to use NBD

2025-07-01T05:02:49.119Z xo:backups:worker INFO starting backup
2025-07-01T05:02:51.256Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.268Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.270Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.627Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
}
2025-07-01T05:06:04.924Z xo:backups:worker INFO starting backup
2025-07-01T05:06:09.185Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
}
2025-07-01T05:06:09.852Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
}
2025-07-01T05:06:18.711Z xo:xapi:vdi WARN invalid HTTP header in response body {
  body: 'HTTP/1.1 500 Internal Error\r\n' +
    'content-length: 346\r\n' +
    'content-type: text/html\r\n' +
    'connection: close\r\n' +
    'cache-control: no-cache, no-store\r\n' +
    '\r\n' +
    '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>Db_exn.Read_missing_uuid(&quot;VDI&quot;, &quot;&quot;, &quot;OpaqueRef:9805714f-7a36-6c2c-c194-9f08baf519df&quot;)</body></html>'
}

@stormi Oh Damn, bad timing I shouldn't have updated my 8.2 pool last sunday

Updated my Lab pool consisting of 2 Nodes this morning
Rebooted master first, then slave. No issues VMs migrated between reboots.
Couldn't see any issue til now.

Installed the patches on my lab pool and both hosts are up and running and no issues so far.

@andriy.sultanov
I ran it for a minute.

rrd2csv.txt

@andriy.sultanov
Same issue after another reboot. The stats on host just flatline. No CPU usage or anything.
See screenshot

I attached log files
daemon.log.txt xcp-rrdd-plugins.log.txt xensource.log.txt

Installed on my Test pool with 2 HP EliteDesk 800 G3 mini. Except the no stats in XenOrchestra. Did some VM migragration between hosts, reboot testes (only Linux VMs though) and so far no issues.

@Greg_E according to the Knowledge base of Redhawks it is den Witz Intel CPUs. So you AMD isn‘t affected

@bberndt you Rocky/Alma 9 will be fine since it's a bug in the kernel for 8 only.

@Greg_E They are all running for a while but none where upgrades from RHEL 7 to RHEL 8. I don't do LEAPPs that gives me more often errors than it works. What I did though on one is replacing the System VDI with a newer one, but that was Alma 8 to Alma 9 and that one has no issues.

@Greg_E said in XCP vm fails to boot after newest kernel applied.:

@bberndt
I just installed a fresh Alma 8, did yum update to see what would happen, and it's still working. Gave out the same kernel as above.

This was installed UEFI on XCP-ng 8.3 which was a fresh install a few days ago from a nightly (near release?) ISO. It was installed to an NFS share, 2 cores and 4GB with an Intel i1000 interface. Xenserver tools 8.4.0-1 installed.

There are no extra packages installed yet, could this be a package conflict.

Anything else I can check to see why mine works and others are failing?

Hmm....I did the same and that VM died just like any other with the broken kernel.
Although I used the Cloud Image so not completly new install from scratch.
Maybe I'll check building a new cloud image what will happen then.

@Murf There is a blog post about using XenOrchestra for a dynamic inventory.
https://xen-orchestra.com/blog/virtops3-ansible-with-xen-orchestra/

Creating VMs is usually a job done by terraform and not ansible. I know there is an ansible collection for VMWare which can do that but as far as I know there is no collection that is able to manage VMs and sadly Satellite's virt-who doesn't seem to have XenOrchestra support at the moment and since RedHat is pushing OpenShift I don't think it will ever come.

But there was a wish for a colletion on the DevOps Megathread
https://xcp-ng.org/forum/post/88993

Upgraded my dev Pool

2 HP EliteDesk 800 G3 Mini
One with i5 6500T and one i7 6700T
Both 32GB RAM, SATA and NVMe Storage
1x Intel NIC
1x RTL8139 NIC

Updated Master first and then the second node. No issues so far.

@nathanael-h said in DevOps Megathread: what you need and how we can help!:

@bufanda I think we'll be able to add backup support to Terraform when 1. the provider will use the new Rest API, and 2. when this API will offer endpoints for backups management. I took note. (This won't be done in minutes )
About Ansible, it'll depends also if/when we start work on it.

No worries it's not a critical issue for me. It's jsut been on my wish list and I already was about to look myself into it by either writing some ansible module for it or finally have a reason to learn go but as you said the current API has some limitations in that regard.

Backup management with the Terraform provider would be a great feature. Maybe also for an upcoming ansible module. I always struggle to find the right backup for a VM since I grouped them in logical groups. So one Backup may handle multiple VMs and Sometimes it would be just easier to edit some IaC then the GUI, especially when I destroy a VM, I always forget to check if Backups exists.

In the Tasks menu on the left it should show all active tasks including the VM creation or it's sub tasks like VDI cloning.

@gb-123 said in XCP-ng 8.3 betas and RCs feedback :

@stormi @olivierlambert

I have also successfully run a test install of RC1 on my lab. Everything seems to be working fine. The TLS issue also seems to go away (will also put a separate post on the topic concerned).

Good Job guys !

One thing I noticed :

I seem to get an error message

no crontab for root

while running crontab -e the first time.

Is that normal? Is it ok to run a cron job for root ?

Could be very well happen if you have a fresh install and no cronjobs in the crontab. Most distributions would put corn jobs in /etc/cron.d anyways and so does xcp-ng too. And the users crontab then will be created on the first edit.