After installing updates: 0 bytes free, Control domain memory = 0B
-
@Dataslak Have you moved all your servers to 8.2.1 yet, or do you still have stuck 8.3 servers that we could have a look at to understand if this may be an issue in the 8.3 beta itself?
-
The grey circle next to the host name in XO means the host is disabled (you can hover with the mouse cursor and this will tell it). Why it is disabled is the true question here. It could be XO not having re-enabled it after the update, but you did try to leave maintenance mode, so it looks like the problem may be elsewhere.
/var/log/xensource.logcontains the answer (or/var/log/xensource.log.*) , but what to look for exactly I can't say off memory. -
@Danp said in After installing updates: 0 bytes free, Control domain memory = 0B:
You can export the VMs running on 8.3 and then import them to a host running 8.2
IIRC you can't (moving towards a newer release is always supported, but towards a lower release isn't), but you can use XO's warm migration to do so. However, to export the VMs you need an enabled host.
Also, be aware that UEFI VMs started on 8.3 will not start on 8.2 unless we wipe their EFI variables completely from the host's command line. Which in most cases is okay but we have seen at least one version of debian fail to start after that and require a bootloader reinstallation.
-
@stormi said in After installing updates: 0 bytes free, Control domain memory = 0B:
IIRC you can't (moving towards a newer release is always supported, but towards a lower release isn't), but you can use XO's warm migration to do so. However, to export the VMs you need an enabled host.
I thought that applied to live migration, but that export/import would still function when moving VMs to an earlier release.
-
@stormi Thank you for your input. It is much appreciated
I have not dared to move all hosts from 8.3 to 8.2.1 out of fear of losing VMs. If you wish to look at them to discover things that may be of value in your beta development then I wish you more than welcome. Please instruct me on how to assist you in this.
I have re-created all VMs on the one host that I restarted and could not get the VMs to boot on. Thus you can go berserk on this server in your bughunt :-D. If you can somehow get the VMs to start up then I only ask that I be allowed to run a quick powershell command on each of them to release their Windows licenses. Thereafter the host is yours for destructive testing.
-
@stormi and @Danp Thank you for all your helpful input.
I look forward to purchasing a contract that opens up tools to more easily handle VMs and pools.
I have tried to enable the host. But can still not get VMs to start, getting "no hosts available".
When I restart the host it is again disabled / put in mainenance (I am actually not sure what it is).I can easily SSH into the command line. I am familiar with DOS and somewhat PowerShell. But not with Linux/Unix. How risky and difficult do you think it will be for me to wipe EFI variables from VMs?
If Debian failed to start; Do you have any experience with Windows VMs? -
@stormi
I do not know how to read the xensource.log :
https://www.dropbox.com/scl/fi/gjm920w2fuskqz8ctyh85/xensource.log?rlkey=3v1tbcrxgiuhs6jh0c1ikl7i1&dl=0
Do you see anything interesting with your trained eye? -
@nikade Hello nikade. Thank you for pitching in.
-
@Dataslak said in After installing updates: 0 bytes free, Control domain memory = 0B:
@nikade Hello nikade. Thank you for pitching in.
Thats really weird, whats the output of "ps aux|grep qemu" ?
-
@Dataslak The easiest for us is if you install XOA (you can without a license), open a support ticket (you don't need a support subscription to open one, to my best knowledge. We check the status once the ticket is open). Then open a support tunnel from XOA, which will allow us to connect.
Or provide me with another way to SSH to your host, in private chat. No teamviewer :D. Real, actual, SSH.
-
@stormi
I get this message on all afflicted hosts when trying to
I have an alternative; Will PM you about this.
-
@Dataslak Yes, I didn't mention it, but it was obvious to me that you couldn't deploy a new VM on a host which can't start VMs :D. I thought you'd use the 8.2.1 one.
-
@nikade asked : "Thats really weird, whats the output of "ps aux|grep qemu" ?"
-
@stormi Forgive me for not keeping up
I will strive to do better.
I meant to say that the hosts with the failed updates all struggle to install XOA. And as you correctly say: There's no point since they can not start VMs. Install on the working 8.2.1. This I managed to do, and Stormi connected to it but could not get in touch with the 8.3 server from there.
Stormi has now found another route and can SSH directly to both servers outside XOA.
I am super exited to hear what he discovers, and to know if he can save one VM on the 8.3 server (005) that my backup has failed to keep safe, and whether he can get the remaining 8.3 servers up and running normally again. -
@Dataslak I haven't looked yet, but as long as you don't overwrite the disks, this VM is actually safe and can be recovered.
-
@stormi
I am very relieved!
The 8.3 host (005) does not seem to want to become enabled. How can I then migrate the VM? Is it as easy as packing up the VM files in an archive through SSH and upload to a different working host?
Will I then meet problems trying to run a 8.3 VM on 8.2.1 ? Will this require XOA with more features (warm migration was mentioned above) unlocked through a subscription? -
@Dataslak said in After installing updates: 0 bytes free, Control domain memory = 0B:
@nikade asked : "Thats really weird, whats the output of "ps aux|grep qemu" ?"
Im baffled, never seen this kind of issues before.
Are you sure the virtualization is enabled in BIOS? -
@nikade Thank you for your help! I deeply appreciate your insights as well as @stormi .
I have six servers which have run 8.3 for 1/2 year, with 3 and 40 VMs on each working very nicely. Can I therefore assume that BIOS settings have been correct? Or will a recent update to XCP-ng 8.3 have triggered the need for additional BIOS settings to be made? Or a newer BIOS?
This is an AMD Ryzen 9 CPU running on B650 chipset motherboard. I have verified in BIOS that :
SVM Mode = Enabled (Enable/disable CPU Virtualization)
SR-IOV Support = Disabled
IOMMU = Disabled
I have not found the HAP setting yet. Don't know if it is called something else. When I google it I do not get any hits indicating that there's even a setting called "HAP" or "Hardware assisted paging"?I have found these four BIOS-settings to be of matter to XCP-ng in forums/troubleshooting docs I've found. Are there any other BIOS settings I have overlooked and should be mindful of?
The BIOS is more than 6 months old; I will try to update BIOS on one of the hosts to see if this makes any change to the problem.
Edit:
I have now updated the BIOS.
XOA still reports:
The "Hardware-assisted virtualization is not enabled on this host" message appeared after installing these updates:
https://www.dropbox.com/scl/fi/fqh6gjc24j6fuvrpe9tdr/yum_history_info.txt?rlkey=cqbho9lm9pdbpds0dn409zub0&dl=1 -
Enable IOMMU too
-
@olivierlambert Thank you for your input.
IOMMU changed from Auto to Enabled
Edit:
After the host comes up again XO still complains " Hardware-assisted virtualization is not enabled on this host".Is there anything else I can do to debug this issue? On second thought: @stormi is on the case, so I should not do anything unless he tells me to.
