After installing updates: 0 bytes free, Control domain memory = 0B
-
@stormi Forgive me for not keeping up
I will strive to do better.
I meant to say that the hosts with the failed updates all struggle to install XOA. And as you correctly say: There's no point since they can not start VMs. Install on the working 8.2.1. This I managed to do, and Stormi connected to it but could not get in touch with the 8.3 server from there.
Stormi has now found another route and can SSH directly to both servers outside XOA.
I am super exited to hear what he discovers, and to know if he can save one VM on the 8.3 server (005) that my backup has failed to keep safe, and whether he can get the remaining 8.3 servers up and running normally again. -
@Dataslak I haven't looked yet, but as long as you don't overwrite the disks, this VM is actually safe and can be recovered.
-
@stormi
I am very relieved!
The 8.3 host (005) does not seem to want to become enabled. How can I then migrate the VM? Is it as easy as packing up the VM files in an archive through SSH and upload to a different working host?
Will I then meet problems trying to run a 8.3 VM on 8.2.1 ? Will this require XOA with more features (warm migration was mentioned above) unlocked through a subscription? -
@Dataslak said in After installing updates: 0 bytes free, Control domain memory = 0B:
@nikade asked : "Thats really weird, whats the output of "ps aux|grep qemu" ?"
Im baffled, never seen this kind of issues before.
Are you sure the virtualization is enabled in BIOS? -
@nikade Thank you for your help! I deeply appreciate your insights as well as @stormi .
I have six servers which have run 8.3 for 1/2 year, with 3 and 40 VMs on each working very nicely. Can I therefore assume that BIOS settings have been correct? Or will a recent update to XCP-ng 8.3 have triggered the need for additional BIOS settings to be made? Or a newer BIOS?
This is an AMD Ryzen 9 CPU running on B650 chipset motherboard. I have verified in BIOS that :
SVM Mode = Enabled (Enable/disable CPU Virtualization)
SR-IOV Support = Disabled
IOMMU = Disabled
I have not found the HAP setting yet. Don't know if it is called something else. When I google it I do not get any hits indicating that there's even a setting called "HAP" or "Hardware assisted paging"?I have found these four BIOS-settings to be of matter to XCP-ng in forums/troubleshooting docs I've found. Are there any other BIOS settings I have overlooked and should be mindful of?
The BIOS is more than 6 months old; I will try to update BIOS on one of the hosts to see if this makes any change to the problem.
Edit:
I have now updated the BIOS.
XOA still reports:
The "Hardware-assisted virtualization is not enabled on this host" message appeared after installing these updates:
https://www.dropbox.com/scl/fi/fqh6gjc24j6fuvrpe9tdr/yum_history_info.txt?rlkey=cqbho9lm9pdbpds0dn409zub0&dl=1 -
Enable IOMMU too
-
@olivierlambert Thank you for your input.
IOMMU changed from Auto to Enabled
Edit:
After the host comes up again XO still complains " Hardware-assisted virtualization is not enabled on this host".Is there anything else I can do to debug this issue? On second thought: @stormi is on the case, so I should not do anything unless he tells me to.
-
Is it better now?
-
@olivierlambert
I can not see any change to the problem.
The host still starts up in maintenance mode. Memory reports no space left despite no VMs running. Not even the host barebone OS itself is shown on the empty memory bar under the "General" tab.
When trying to start a VM:
The full message text:
-1(list index out of range, , Traceback (most recent call last):
File "/etc/xapi.d/plugins/xcpngutils/init.py", line 119, in wrapper
return func(*args, **kwds)
File "/etc/xapi.d/plugins/hyperthreading.py", line 17, in get_hyperthreading
return json.dumps(int(lines[0]) > 1)
IndexError: list index out of rangeWhen I google "AMD HyperThreading" (since this is not an Intel-system but AMD): Google says "Hyper-threading is the Intel technology name, and commonly known name, for the Computing concept known as simultaneous multithreading. AMD have an equivalent technology known as Cluster Multi-threading."
This is enabled by default in BIOS (and I reset BIOS to defaults before upgrading BIOS just an hour ago). -
What's the output of
xl dmesg -
-
You have an issue: I'm pretty sure you are using Xen 4.13 while using the 4.17 tooling around, causing this problem. So the question is "why are you booting on 4.13?". Double check your Grub menu when you boot, to see that you actually boots on Xen 4.17
-
@olivierlambert
https://xcp-ng.org/forum/assets/uploads/files/1714683479351-b73237cc-8f27-4789-a9d3-5e322c2cd244-image.pngAnswer to your question : I don't know
Six servers have been running fine for months. Then i accepted XO's request to install these updates and the problem manifested on all six.
I did nothing to XCP-ng for months before this happened. Did nothing other than reboot one host after, which then made me realize that something was seriously wrong since no VMs came up. I then also saw the other problems shown in screen clippings earlier in this chat.
Luckily the VMs continued running on the other hosts. I am very happy I only rebooted one first. Or all VMs would have been down for days.Lacking in mental capacity to understand what has happened I am therefore hoping you and your eminent staff can "Sherlock Holmes" this :-).
How can the abovementioned updates have triggered it?
Can it happen to others?I may have waited too long between checking if updates were waiting to be installed. Could this have caused the problem? Were they installed in incorrect order?
I can not find Xen 4.17 in the boot menu. How do I fix this? Should I create a new 8.3 installation media and try to upgrade the system? Can I do this safely without damaging VMs?
-
Well, first, you are on XCP-ng 8.3, right? It's still a beta, so by definition this could happen (especially a jump like this from Xen 4.13 to 4.17). If you want guarantee stability, you have to stick to our LTS version.
Anyway, to know which Xen version you are running, you can do a:
cat /sys/hypervisor/version/minor. If it's17, then you are on Xen 4.17 and that's your issue. I don't know how this could happen in the first place, like if the Grub file was never updated or tinkered before.To check a bit more, when on Grub, press "e" key to edit the first entry and make a picture to see what's displayed there. I'm thinking on a way to regen Grub that will take the right Xen version into account (ping @stormi )
-
@olivierlambert
I had to use 8.3 on these newest 6 hosts since 8.2.1 gave me a black screen mid-way in the installation. I guess because of missing drivers. The forum adviced 8.3 and that worked wonderfully. It has been working fine for more than 1/2 year (forgot when I started using it).I did not register well enough that i was running a risk by doing this. Mea culpa.
-
@Dataslak
I am late to a meeting with customers. I would much rather stay in the office to work with this. But when I come back in a couple of hours I will eagerly check for a recipe/howto on what to do next, and then carefully follow those.
I am very optimistic now. Thank you all for assisting and to @olivierlambert for the big development today. -
So now we know the problem, the next step will be to solve it
Let's check what's going on in
/bootwith als -la /boot -
@olivierlambert said in After installing updates: 0 bytes free, Control domain memory = 0B:
You have an issue: I'm pretty sure you are using Xen 4.13 while using the 4.17 tooling around, causing this problem. So the question is "why are you booting on 4.13?". Double check your Grub menu when you boot, to see that you actually boots on Xen 4.17
Nice catch, this is what community is all about
-
@olivierlambert
I love and respect your attitude
Agree wholeheartedly with @nikade
-
So
/boot/xen.gzis pointing toxen-4.17.3-4.gz, which sounds correct. So why you are still running on Xen 4.13?
It's like you did not reboot, but since you showed me the Grub menu, I'm assuming you already did It would be interesting to compare the existing xen file and see if it's the right one from our repo. Something is fishy here
