I finally got my lab updated and the two vms started, so far so good. Stats data is present, the first time I've seen any of my hosts pull 100% CPU. Hosts are little HP T740 thin clients with AMD V1756B processors and 64GB of ram, in a 3 host pool.
Sorry I can't bring more detail, not really doing work on my XCP lab right now, and lots of shake up around tech. space at work.
I decided to burn my lab down because of some trouble and wanted to start fresh to isolate any problems I may have created in the past. Wiped the disks, built a new NFS share ready for work, installed the XCP-ng OS and configured networking for the management interface on the local machines, all normal stuff. Went back to my desk and logged into XO Lite on each machine, using console --> xsconsole I was able to get all the hosts into a pool, and add an ISO and NFS SR, all seemed to work.
Went to the New VM to create a simple Debian vm on which I intended to host XO-CE from sources. Typed the info in, chose the Debian image from my ISO SR, the rest of the config as normal. Uncheck the boot after creation box, and create.
Click on the VM to bring the console up, then click power on... UEFI loader comes up, and ... Can't find the boot disk. Tried another iso that I knew was capable of EFI boot, same.
Realized I was trying to start in EFI, made a new one in BIOS mode, this one worked but now my screen shots are messed up so I'll probably have to go through and delete stuff with a different XO and go back to grabbing images of the process to write up "for the next guy".
Other than this small issue, it's working and doing it all from XO Lite. Yes still a bunch of missing features, but I know you are working on this. I just wanted to try things this way on this fresh build and see what I could get done from local console and XO Lite.
HP T740 with 64g of ram, 256g sata for OS, optional nvme for faster storage, optional 10gbe networking through low profile card. Works great for my lab.
Ran vSphere8 on three of them, running Harvester on 3 of them now. XCP-ng and vSphere8 work great, Harvester is a bit more resource hungry. Only the bottom three have nvme and currently have Harvester 1.7.0 running while I try to find time to learn more about this system.
My production pool upgraded fine as far as I can tell, but not so good with the windows drivers, I'll make a thread on the drivers.
Nothing really to add, my 3 host Intel production pool updated just fine. The load balancer is always a little weird, but I'm sure it is calculated based on CPU and RAM assigned to each VM, where I split things up based on workload.
It's a small system, and the real workload is handled by 3 Windows VMs so I tend to split them up onto one of the three hosts.
I may get to my lab in the next couple of days, but it isn't doing work so testing is kind of pointless right now. The only thing "doing work" is a VM with XO from sources.
Did my production this morning... Not a smooth upgrade this time.
The master took so long that the process timed out, I can't tell too much more because I was doing other work and was doing this remotely.
I handled the next host manually, and that took so much time I gathered my hearing protection and went over to the rack. It was stuck in a reboot phase but hadn't shut down for about 20 minutes. Held my finger on the power button. booted back up with a couple reboots in the process and it finally was ready.
Moved the VMs off the third host, manually triggered the update, and sat and watched looking at the VGA output to see what was happening. The reboot phase saw the XCP-ng animation progress all the way to an empty bar, but sat there another 5-10 minutes until I again held my finger on the power button to shut it off. Power up and after a bit of time it was ready again. All in all, updating three hosts took me around 2 hours today.
Here's the one condition that I think could contribute to this issue:
I have two NAS normally connected, an ISO and NFS connection on each. One of the servers is powered down for construction, but I did not disconnect it from the hosts. Could this severed connection be the reason why my updates took so long, something around not being able to purge or drain the state before the reboot?
I've disconnected those SR from the hosts, and I'll probably do a rolling pool reboot later today or next week and see if things go better.
And all that said, my XO sources is not happy with this update. XO6 isn't grabbing the data so dashboards are blank or take a long time with spinning wheels gathering the data. XO5 is immediate. One of my XO sources updated this morning, the other was yesterday so they should be pretty close to current.
BTW, the ram increase made zero difference in the VDI migration times.
I may still change the ram for each host in production, but I only have like 12 VMs right now and probably not many more to create in my little system. Might be useful if I ever get down to a single host because of a catastrophe. Currently running what it chose for default at 7.41 MiB, roughly doubled that in my lab to see what might happen.
Just got my VMware lab started so all this other stuff is going to back burner for a while. I will say I like the basic ESXi interface that they give you, I can see why smaller systems may not bother buying VCenter. I hope XO-lite gives us the same amount of function when it gets done (or more).
I was looking for a way to mark this solved, can't find it.
I haven't moved things, but after migrating my big lab to my mini-lab, I'm confident that the warm migration is the way to go. It was fast and seamless as long as you have the right network adapters set up. I had to fool with one of my networks to make a VM function, but that was certainly something I overlooked while setting up the mini-lab. A little testing before moving the VMs should make this go easily if using the old servers is the option for this project.
Thanks. I'm still going to wait until next week, my schedule just doesn't give me the time right now. But nice to know that when I do update, it will be fixed and my laziness prevented a problem for me 
I think I'll get my pool updated today, only been waiting a week at this point.
@coolsport00 You actually referenced my post, so I referenced yours as backup to the condition still being present. I think we just need to wait for them to finish making XO-lite what they seem to intend it to be, I think it's going to be good and I think they are doing a great job so far.
That's what I thought, I have it disconnected now and I'll try a rolling reboot when I'm back at work.
Did my production this morning... Not a smooth upgrade this time.
The master took so long that the process timed out, I can't tell too much more because I was doing other work and was doing this remotely.
I handled the next host manually, and that took so much time I gathered my hearing protection and went over to the rack. It was stuck in a reboot phase but hadn't shut down for about 20 minutes. Held my finger on the power button. booted back up with a couple reboots in the process and it finally was ready.
Moved the VMs off the third host, manually triggered the update, and sat and watched looking at the VGA output to see what was happening. The reboot phase saw the XCP-ng animation progress all the way to an empty bar, but sat there another 5-10 minutes until I again held my finger on the power button to shut it off. Power up and after a bit of time it was ready again. All in all, updating three hosts took me around 2 hours today.
Here's the one condition that I think could contribute to this issue:
I have two NAS normally connected, an ISO and NFS connection on each. One of the servers is powered down for construction, but I did not disconnect it from the hosts. Could this severed connection be the reason why my updates took so long, something around not being able to purge or drain the state before the reboot?
I've disconnected those SR from the hosts, and I'll probably do a rolling pool reboot later today or next week and see if things go better.
And all that said, my XO sources is not happy with this update. XO6 isn't grabbing the data so dashboards are blank or take a long time with spinning wheels gathering the data. XO5 is immediate. One of my XO sources updated this morning, the other was yesterday so they should be pretty close to current.
These have 9.4.2-xxxx and I need to go through and update everything now that I see how far behind I am.
the second VM completed fine after its reboot, something was just stuck in a dirty state and got in the way of the import or health check.
I didn't wait long enough for XO to catch up, the manual backup I did after rebooting the VM worked, I'll hit the other VM before I start some other work and see what I see. If I don't respond back toady, then it was a "dirty" VM because of some odd reboot condition. I've just never seen that before and odd that it happened 8 days after installing MS updates.
Both VMs had a reboot condition that they needed, but even after reboot I have a fail.
I watched this one, snapshot and transfer all went fine, during the import stage it only got to around 30% and then failed out and all the IO dropped off.
These are the only two VMs that I can't backup with compression due to the large amount of empty space on the disk. It's an old problem and I'm not sure it has been fixed. I may try setting zstd after lunch and trying again. If that doesn't work, it's a tomorrow problem.
Yes, I'm doing a health check on each backup job. I updated XO sources but it looks like the manual backup I started is going to fail, it is almost at 30 minutes for a 22 minute process. I'll reboot the VMs next and see if there is just something dirty in the state that needs to be cleaned up.
I have 2 VMs that are now failing and have been for the last two days, the three day old worked. Just sorting through it now.
Both VMs are "bigger" than the rest, both have a disk that are 500gb and more than a lot of empty space in them. Compression is off, no updates have been performed to the XCP hosts for a while, and MS updates were applied 8 days ago. Running Xenserver 9.4.2 agent and drivers, same as the VMs that are still working. There are no snapshots for any of the VMs.
XO sources with Xen Orchestra, commit c5b7e and Master, commit e3a58 currently 31 commits behind so I guess I'll be updating it to see if something got changed.
The only error listed is a timeout error. Going to reboot and try manually running them again to see what happens. Including the the error report
{
"data": {
"mode": "full",
"reportWhen": "failure"
},
"id": "1779369775885",
"jobId": "05f70792-9624-4150-9597-5917c045bea6",
"jobName": "DC2-backup",
"message": "backup",
"scheduleId": "5ec0bd0c-b537-4da9-afff-3f1eb0fbd3e9",
"start": 1779369775885,
"status": "failure",
"tasks": [
{
"id": "0mpfitnx5-n8d213qn7md",
"start": 1779369778265,
"status": "failure",
"tasks": [
{
"id": "0mpfitogj-nllocq70f6q",
"start": 1779369778963,
"status": "success",
"end": 1779369780410,
"result": "4353adcc-9de6-1ff1-8172-f600c7e0cb4c",
"message": "snapshot"
},
{
"id": "0mpfitpl7-j4loeu6sjhj",
"start": 1779369780427,
"status": "success",
"tasks": [
{
"id": "0mpfitplr-condtv8ouhw",
"start": 1779369780447,
"status": "success",
"end": 1779370618949,
"result": {
"size": 151409389568
},
"message": "transfer"
}
],
"end": 1779370618976,
"message": "export",
"data": {
"id": "0c6ed546-4ff5-49b8-b28e-80bbd3513e52",
"type": "remote",
"isFull": true
}
},
{
"id": "0mpfjbp3d-wpp55k5n2b",
"start": 1779370619593,
"status": "failure",
"tasks": [
{
"id": "0mpfjbp3i-c4eg55kbkjn",
"start": 1779370619598,
"status": "success",
"end": 1779371188852,
"result": {
"size": 151409389568,
"id": "e6c3c4c5-f272-a475-8f7a-d9715c257557"
},
"message": "transfer"
},
{
"id": "0mpfjnwc5-sqxh8e4sfsr",
"start": 1779371188853,
"status": "failure",
"end": 1779371789013,
"result": {
"message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
"name": "Error",
"stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
},
"message": "vmstart"
}
],
"end": 1779371792327,
"result": {
"message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
"name": "Error",
"stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
},
"message": "health check"
},
{
"id": "0mpfk0tzk-vg8hq80to3g",
"start": 1779371792336,
"status": "success",
"end": 1779371792896,
"result": {
"merge": false,
"size": 0
},
"message": "clean-vm"
}
],
"end": 1779371792897,
"result": {
"message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
"name": "Error",
"stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
},
"message": "backup VM",
"data": {
"id": "1c163183-d863-f8cc-159a-5c5281aafee3",
"type": "VM",
"name_label": "BMC-DC2"
}
}
],
"end": 1779371792898,
"infos": [
{
"data": {
"vms": [
"1c163183-d863-f8cc-159a-5c5281aafee3"
]
},
"message": "vms"
}
]
}
The above suggests the drivers are a problem, Xenserver has 9.6.0 available which requires removing current, then install new. I'll have to plan that upgrade to make sure the AD VMs stay functional, probably a job for tomorrow.
Thanks, my updates and my RPU are normally at different times. I also don't backup multiple times per day like I probably should.
Why did you need to disable backups before applying this?
Before the LTS release UEFI worked extremely well, something kind of went sideways when it hit release. I only have a single UEFI VM, it was created under version 8.2 and does not give me trouble.
I guess I need to get my lab back up, had a UPS fail and shut everything off a few days ago. Then I need to test UEFI VMs and see what I can see. But back when it went LTS I reinstalled from scratch and tried to do everything through XO-lite to get the first VM running and found that Debian 13 or Windows Server 2022 would not boot if I created them with UEFI. Since then I haven't tested this again.
I'm also tempted to just move my whole lab to version 9, waiting for the next ISO to land and I may do this. I can run real testing workloads on Harvester for the time being and maybe play around migrating back and forth between both systems.