I finally got my lab updated and the two vms started, so far so good. Stats data is present, the first time I've seen any of my hosts pull 100% CPU. Hosts are little HP T740 thin clients with AMD V1756B processors and 64GB of ram, in a 3 host pool.
Sorry I can't bring more detail, not really doing work on my XCP lab right now, and lots of shake up around tech. space at work.
I decided to burn my lab down because of some trouble and wanted to start fresh to isolate any problems I may have created in the past. Wiped the disks, built a new NFS share ready for work, installed the XCP-ng OS and configured networking for the management interface on the local machines, all normal stuff. Went back to my desk and logged into XO Lite on each machine, using console --> xsconsole I was able to get all the hosts into a pool, and add an ISO and NFS SR, all seemed to work.
Went to the New VM to create a simple Debian vm on which I intended to host XO-CE from sources. Typed the info in, chose the Debian image from my ISO SR, the rest of the config as normal. Uncheck the boot after creation box, and create.
Click on the VM to bring the console up, then click power on... UEFI loader comes up, and ... Can't find the boot disk. Tried another iso that I knew was capable of EFI boot, same.
Realized I was trying to start in EFI, made a new one in BIOS mode, this one worked but now my screen shots are messed up so I'll probably have to go through and delete stuff with a different XO and go back to grabbing images of the process to write up "for the next guy".
Other than this small issue, it's working and doing it all from XO Lite. Yes still a bunch of missing features, but I know you are working on this. I just wanted to try things this way on this fresh build and see what I could get done from local console and XO Lite.
HP T740 with 64g of ram, 256g sata for OS, optional nvme for faster storage, optional 10gbe networking through low profile card. Works great for my lab.
Ran vSphere8 on three of them, running Harvester on 3 of them now. XCP-ng and vSphere8 work great, Harvester is a bit more resource hungry. Only the bottom three have nvme and currently have Harvester 1.7.0 running while I try to find time to learn more about this system.
My production pool upgraded fine as far as I can tell, but not so good with the windows drivers, I'll make a thread on the drivers.
Nothing really to add, my 3 host Intel production pool updated just fine. The load balancer is always a little weird, but I'm sure it is calculated based on CPU and RAM assigned to each VM, where I split things up based on workload.
It's a small system, and the real workload is handled by 3 Windows VMs so I tend to split them up onto one of the three hosts.
I may get to my lab in the next couple of days, but it isn't doing work so testing is kind of pointless right now. The only thing "doing work" is a VM with XO from sources.
Did my production this morning... Not a smooth upgrade this time.
The master took so long that the process timed out, I can't tell too much more because I was doing other work and was doing this remotely.
I handled the next host manually, and that took so much time I gathered my hearing protection and went over to the rack. It was stuck in a reboot phase but hadn't shut down for about 20 minutes. Held my finger on the power button. booted back up with a couple reboots in the process and it finally was ready.
Moved the VMs off the third host, manually triggered the update, and sat and watched looking at the VGA output to see what was happening. The reboot phase saw the XCP-ng animation progress all the way to an empty bar, but sat there another 5-10 minutes until I again held my finger on the power button to shut it off. Power up and after a bit of time it was ready again. All in all, updating three hosts took me around 2 hours today.
Here's the one condition that I think could contribute to this issue:
I have two NAS normally connected, an ISO and NFS connection on each. One of the servers is powered down for construction, but I did not disconnect it from the hosts. Could this severed connection be the reason why my updates took so long, something around not being able to purge or drain the state before the reboot?
I've disconnected those SR from the hosts, and I'll probably do a rolling pool reboot later today or next week and see if things go better.
And all that said, my XO sources is not happy with this update. XO6 isn't grabbing the data so dashboards are blank or take a long time with spinning wheels gathering the data. XO5 is immediate. One of my XO sources updated this morning, the other was yesterday so they should be pretty close to current.
BTW, the ram increase made zero difference in the VDI migration times.
I may still change the ram for each host in production, but I only have like 12 VMs right now and probably not many more to create in my little system. Might be useful if I ever get down to a single host because of a catastrophe. Currently running what it chose for default at 7.41 MiB, roughly doubled that in my lab to see what might happen.
Just got my VMware lab started so all this other stuff is going to back burner for a while. I will say I like the basic ESXi interface that they give you, I can see why smaller systems may not bother buying VCenter. I hope XO-lite gives us the same amount of function when it gets done (or more).
I was looking for a way to mark this solved, can't find it.
I haven't moved things, but after migrating my big lab to my mini-lab, I'm confident that the warm migration is the way to go. It was fast and seamless as long as you have the right network adapters set up. I had to fool with one of my networks to make a VM function, but that was certainly something I overlooked while setting up the mini-lab. A little testing before moving the VMs should make this go easily if using the old servers is the option for this project.
OK, marking this solved... It was the phantom reboots causing issues because the tools were not fully installed, only thing I can't understand is why only 2 out of 3 Windows servers would quit. Any way, got the Xen Tools updated and a couple reboots later, everything is working again.
Also I can now report that a bug that was fixed maybe a year ago is confirmed to work in my system. The bug was backing up a Windows VM that had a lot of empty space in the drive (over 250GB free) and turning on compression. It would time out counting zeros because it thought the process wasn't doing work. Both of these VMs are now compressing with zstd, one of them is substantially smaller now, the other only shrunk about 20GB.
Because of the way Windows uses the drive, these are all full backups. The differential never seemed to be small because Windows does a lot of changes on the drive (there is a post about this somewhere in here from Olivier).
My three hosts are all EDT, drift on mine is pretty small because I have a local NTP server with GNSS.
So it could be a timezone issue and might be worth looking into.
Ok, so what's the scoop here? Safe to roll this out or wait a few days? I just saw the notice in my XO that there are updates and I don't want to wreck my system right now after fixing a few other "issues".
[edit] I see this is just the certificate update so pushing it out and listening to servers reboot.
OK, marking this solved... It was the phantom reboots causing issues because the tools were not fully installed, only thing I can't understand is why only 2 out of 3 Windows servers would quit. Any way, got the Xen Tools updated and a couple reboots later, everything is working again.
Also I can now report that a bug that was fixed maybe a year ago is confirmed to work in my system. The bug was backing up a Windows VM that had a lot of empty space in the drive (over 250GB free) and turning on compression. It would time out counting zeros because it thought the process wasn't doing work. Both of these VMs are now compressing with zstd, one of them is substantially smaller now, the other only shrunk about 20GB.
Because of the way Windows uses the drive, these are all full backups. The differential never seemed to be small because Windows does a lot of changes on the drive (there is a post about this somewhere in here from Olivier).
The UEFI and possibly SecureBoot requirements on a lot of stuff and going to turn a lot of old systems into garbage.
Also going to be an issue when more current stuff gets upgrade from 8.3 in BIOS mode to requiring UEFI for version 9. That's the boat I'm in for production.
Alma and openSuse may be a few of the last holdouts allowing v2 and BIOS boot.
Pretty sure I figured this out. I went to update the Xenserver tools, and the updater said the installer was busy. That meant the machine was in a "dirty condition" and couldn't work right. The dirty condition was the newest Xenserver tools needing a couple more reboots to get fully insatalled.
testing now and will have to wait on the DC until the network use get lighter this afternoon.
Running the second VM right now, but pretty sure it will now pass the health check. I'll update after I check the DC in a few hours.
@DustinB Everything I care about gets backed up and health checked each night, some things really should be several times a day, but 1 day is tollerable.
I'll have to get things updated after I deal with some php2shell stuff, that will need to come first. And now node.js in my XO machines.
I'm guessing this might be having updates applied, and now I need new Xen Tools. I have a feeling tomorrow will be filled with updating the tools and checking things because I bet all my backups fail tonight. And I bet all fail after they try to start.
Ok, generally I'm doing backups after everyone has gone home for the night, so changes to the DB should be minimal.
I'll have to look into this more deeply and maybe use the Server Backup tool to schedule both DC going forward. Used to use that when bare metal, and had to restore a couple times too.
@DustinB Hmmm... They have always completed in the past, but I'll admit that I haven't tried to restore one and see if it really works. too busy to perform the test the last couple years.
but now I know the time out is after the machine boots, and I've seen the console come up and wait longer than 4 minutes, so probably it is OK if it could connect to the other DC and compare the database.
Probably too long ago, mentioned it on my follow up post while you were posting.
Replying with an update. As usual, a manually trigger backup completes. 12 minutes to send to the remote, 10 minutes to bring it back. But with the domain controller, it look like I may need to update the management agent, it's taking a long time to get that running.
When a DC comes up, it reaches out to the network to check with it's peer and sync data while it was gone. During the health check the network is not there so it takes 5+ minutes to fully boot. it's now sitting waiting. took about 10 minutes total for the MA to report and then shut the health check down. Not sure if there is a better way to do this for a domain controller or not, mostly just a data point for other people that think their health check may be stuck.
Time to go fix the other VM that failed last night.
[edit] it said it failed again, though I watched the console for about 4 minutes and clearly it booted. I'll have to update the MA and drivers and see what happens. But it looks like 10 minutes waiting for the MA is too long now.