After installing updates: 0 bytes free, Control domain memory = 0B

Dataslak

@olivierlambert
https://xcp-ng.org/forum/assets/uploads/files/1714683479351-b73237cc-8f27-4789-a9d3-5e322c2cd244-image.png

Answer to your question : I don't know
Six servers have been running fine for months. Then i accepted XO's request to install these updates and the problem manifested on all six.
I did nothing to XCP-ng for months before this happened. Did nothing other than reboot one host after, which then made me realize that something was seriously wrong since no VMs came up. I then also saw the other problems shown in screen clippings earlier in this chat.
Luckily the VMs continued running on the other hosts. I am very happy I only rebooted one first. Or all VMs would have been down for days.

Lacking in mental capacity to understand what has happened I am therefore hoping you and your eminent staff can "Sherlock Holmes" this :-).
How can the abovementioned updates have triggered it?
Can it happen to others?

I may have waited too long between checking if updates were waiting to be installed. Could this have caused the problem? Were they installed in incorrect order?

I can not find Xen 4.17 in the boot menu. How do I fix this? Should I create a new 8.3 installation media and try to upgrade the system? Can I do this safely without damaging VMs?

olivierlambert

Well, first, you are on XCP-ng 8.3, right? It's still a beta, so by definition this could happen (especially a jump like this from Xen 4.13 to 4.17). If you want guarantee stability, you have to stick to our LTS version.

Anyway, to know which Xen version you are running, you can do a: cat /sys/hypervisor/version/minor. If it's 17, then you are on Xen 4.17 and that's your issue. I don't know how this could happen in the first place, like if the Grub file was never updated or tinkered before.

To check a bit more, when on Grub, press "e" key to edit the first entry and make a picture to see what's displayed there. I'm thinking on a way to regen Grub that will take the right Xen version into account (ping @stormi )

Dataslak

@olivierlambert
I had to use 8.3 on these newest 6 hosts since 8.2.1 gave me a black screen mid-way in the installation. I guess because of missing drivers. The forum adviced 8.3 and that worked wonderfully. It has been working fine for more than 1/2 year (forgot when I started using it).

I did not register well enough that i was running a risk by doing this. Mea culpa.

Dataslak

@Dataslak
I am late to a meeting with customers. I would much rather stay in the office to work with this. But when I come back in a couple of hours I will eagerly check for a recipe/howto on what to do next, and then carefully follow those.
I am very optimistic now. Thank you all for assisting and to @olivierlambert for the big development today.

olivierlambert

So now we know the problem, the next step will be to solve it

Let's check what's going on in /boot with a ls -la /boot

nikade

@olivierlambert said in After installing updates: 0 bytes free, Control domain memory = 0B:

You have an issue: I'm pretty sure you are using Xen 4.13 while using the 4.17 tooling around, causing this problem. So the question is "why are you booting on 4.13?". Double check your Grub menu when you boot, to see that you actually boots on Xen 4.17

Nice catch, this is what community is all about

Dataslak

@olivierlambert
I love and respect your attitude

Agree wholeheartedly with @nikade

olivierlambert

So /boot/xen.gz is pointing to xen-4.17.3-4.gz, which sounds correct. So why you are still running on Xen 4.13? It's like you did not reboot, but since you showed me the Grub menu, I'm assuming you already did

It would be interesting to compare the existing xen file and see if it's the right one from our repo. Something is fishy here

olivierlambert

Can you md5sum xen-4.17.3-4.gz

From the mirror & RPM, I have f011721be0c7b57563e29ed282558da3

olivierlambert

Adding @yann in the loop in case I'm missing something obvious

yann

@Dataslak what does lsblk -o name,mountpoint,label,size,uuid show?

Dataslak

@olivierlambert

Dataslak

@yann
Hello Yann, thank you for pitching in.

yann

@Dataslak can you please request a commandline from GRUB (hit c on the boot menu), and issue the following commands:

echo $root
search --label --set root root-eqjpzg
echo $root

olivierlambert

Also a cat /proc/mdstat in the Dom0 would help.

Dataslak

@yann

Info: I am mirroring two M.2 SSDs ! Software RAID established by the installation routine of v8.3.
Could the mirror be broken and cause this somehow?

Dataslak

@olivierlambert said in After installing updates: 0 bytes free, Control domain memory = 0B:

Also a cat /proc/mdstat in the Dom0 would help.

Please forgive my ignorance: How do I execute this command in Dom0 ?

I've read https://wiki.xenproject.org/wiki/Dom0 and it helped a little. Do I run the command in the console within XOA?

yann

@Dataslak so it is choosing to "boot from the 1st disk of the raid1", we could try to tell him to boot from the 2nd one:

on the grub menu hit e to edit the boot commands
replace that search ... line with set root=hd1,gpt1
then hit Ctrl-x to boot

olivierlambert

@Dataslak Dom0 is "the host" (if you think it's the host it's not really but anyway), ie the machine you are connected to and showing results since the start

Dataslak

@yann
Wohoo!!
All VMs came up!
Host is not in maintenance mode.
Control domain memory = 12GiB
Stats are back
Etc....

As far as I can see (which is limited) everything looks good?

How can I see the status of the RAID1 and see if the mirror is intact ?