After installing updates: 0 bytes free, Control domain memory = 0B
-
XCP-ng 8.3
Server is the only member of its pool, being pool master.
CPU: AMD Ryzen 9 7950X 16core
RAM: 192GiB
HDD: ~250GiB of 2TiB free, using only local storage.
#VMs: 43
Has worked fine for >6 months.I installed updates today. After this was completed and XO asked for a restart of the host, I shut down all VMs and restarted the host.
When the host came up again, no VMs started automatically like they normally do.
When checking XO -> Home -> Hosts -> The host shows:
-
General tab: RAM: 192GiB used of 192GiB (0 B free). And the bar that is normally populated with green blocks showing running VMs RAM usage is completely empty, not even the host RAM usage is shown.
-
Advanced tab: The "Disable maintenance mode" and "Enable" buttons are green. Control domain memory = 0 B.
-
Logs shows only "No logs so far" and is otherwise empty
-
Patches shows "Host up-to-date!" and nothing else.
-
Storage tab looks normal.
-
Network tab: Only the management interface replies to ping. Normally I should be able to ping the "Utside" interface. Why is this interface not configured? Because Safe Mode is active?
-
Stats: All show "No stats."
I read in a forum post about someone who had a somewhat similar problem, who disabled maintenance mode and rebooted, solving his problem successfully. This did not work for me. Clicking "Disable maintenance mode" causes both it an "Enable" to turn orange. When host comes back up in XO after restart, the "Disable maintenance mode" and "Enable" buttons are again green, Control domain memory is still 0B.
Trying to set the Control domain memory to 12GB (recommended size for 192GiB?) makes the host restart immediately as the manual says it should do. When the server comes back up in XO it still shows Control domain memory = 0B.
Is there a way for me to determine if the host is running in safe mode?
I have googled for hours without finding relevant info.
Reading about rollback makes me afraid that I will do more harm than good. I have supplied the yum history output below; Are these safe to rollback?
yum history : https://www.dropbox.com/scl/fi/fqh6gjc24j6fuvrpe9tdr/yum_history_info.txt?rlkey=cqbho9lm9pdbpds0dn409zub0&dl=0
When I choose Safe Mode in the boot menu I get the same values on the tab pages. Is there a command I can run to determine if the host has booted up in safe mode even when I choose regular XCP-ng in the boot menu?
Other logs:
daemon.log : https://www.dropbox.com/scl/fi/gvs2q5x74qagjdvmkxqp0/daemon.log?rlkey=bv8phbus8g56muiytwy5riqj2&dl=0
kern.log : https://www.dropbox.com/scl/fi/7yzgn5hwq53tfg1mvzpj7/kern.log?rlkey=wj49pa1b54yrgugnro06aa4qg&dl=0
I found no crash.log.
I am pretty new to XCP-ng; please tell me if there are other logs/files/diag I should include.
All help is deeply appreciated.
Best wishes
Aslak -
-
@Dataslak I have tried the "Enable"-button too, yielding same result.
-
How about xensource.log? Anything interesting there?
-
@Danp Thank you for your reply
Xensource.log : https://www.dropbox.com/scl/fi/gjm920w2fuskqz8ctyh85/xensource.log?rlkey=3v1tbcrxgiuhs6jh0c1ikl7i1&dl=0 -
Have you tried performing a
force reboot(found on the host's Advanced tab in XO) on the host? -
@Danp I have now done a
.The host still reports :
All my servers running 8.3 have this problem. But the one server still running 8.2 does not (the one on top of the list):
When I click the orange triangles, all 8.3 servers say:
The XCP-ng-005 server - which i have rebooted - says:
XCP-ng-002 (the 8.2 server) says:
I have not dared rebooting all servers out of fear that they will not be able to load up VMs afterwars; The VMs on hosts that I have not rebooted work fine.
Trying to start up VMs on XCP-ng-005 - the host that I have rebooted - results in the following error:
-
@Danp Regarding "No XCP-ng Pro Support enabled on this host" displayed on the seven 8.3-servers:
Can you see any of the latest updates to 8.3 introducing this error?yum history report:
https://www.dropbox.com/scl/fi/fqh6gjc24j6fuvrpe9tdr/yum_history_info.txt?rlkey=cqbho9lm9pdbpds0dn409zub0&dl=0)All 8.3 servers have the same BIOS settings, hardware, etc as they had pre-updates.
In vain hopes I have tried to reset virtualization support in BIOS on the botherboard of XCP-ng-005. But no change.
-
A few comments (in no particular order) --
-
8.3 is still a beta product and it isn't recommended for production environments
-
There have been a few reports that the jump in Xen from 4.13 to Xen 4.17 has caused similar issues. My understanding is that they were resolved by rebooting the hosts, which is why I suggested the forced reboot.
-
With as many servers as you are running, you should consider adding our Pro Support so that you have a dedicated team to assist you with these types of issues
-
The
Reboot to apply updatesalert indicates that these servers haven't been rebooted (or maybe something interfered with the reboot occurring). -
You may want to fully shutdown these hosts and then check to see if these alerts are gone once they are back up.
-
The
No XCP-ng Pro Support enabled on this hostalert is expected when you are running XO from sources or XOA Free.
-
-
@Danp
Thank you for your replies. I deeply appreciate your support!I have clearly not read the documentation good enough, not having caught that 8.3 is not released for production yet. Mea culpa!
Pro support:
I am not earning enough money on these servers to carry the full 8.000 USD extra cost per year. I have sent an e-mail to Vates previously and again today asking if they will give me a discount to help me get this business up and running. If I can get this afloat then I hope to bring much more income to both vates and my employer.Being an old IT-consultant I believe in paying for products and support. I do want to support Vates in its development of XCP-ng.
Being the only one among my colleagues working with this, having spent many evenings and nights on this project, I have longed for Pro Support. But my employer is not a wealthy company. Wanting to spread the risk on several servers instead of gathering all VMs (eggs) on one server (one basket) we will have to pay a lot more with Vates' pricing structure of 1000USD per server. We can not - so far - carry the cost of large proper servers with vast amounts of RAM to host many VMs on which would reduce the Vates cost.
If i lose money then my employer will terminate the project.If I can pay to have someone connect to the host to fix the problem then I will gladly do so.
Reboot to apply updates: I understood as much. I restarted only one server first, and when that failed to load VMs I did not dare to restart the remaining ones. The VMs on those work fine, for now.
I have shut down XCP-ng-005 completely. All the way until only the on/off switch could start it up again.
To get the VMs running again:
Is it "trivial" to downgrade a 8.3 host to 8.2 without ruining the VMs?
Or will I have to export VMs from a 8.3 server to a 8.2?
(Is a VM image backwards compatible?)I fear doing this since I've had lots of problems moving VMs between hosts using XO. It has worked "50% of the time". Can you recommend a "howto" with commands I can type to copy the VMs more reliably to USB media, and import them in to a 8.2 host?
I am now cobbling together a new server from old parts, that I hope will be able to host the VMs temporarily until I can install 8.2 on the old one and move the VMs back.
-
-
You might want to check out the Essential or Essential+ options at https://vates.tech/pricing-and-support/. This would allow you get obtain support for up to 3 hosts.
-
You can export the VMs running on 8.3 and then import them to a host running 8.2
-
Maybe @stormi will have some ideas on how to recover your 8.3 hosts
-
-
@Danp Thank you for your advice.
If I only purchase support for 3 servers; How will XO behave with all 8? You do not have to reply; I will try contacting sales on monday.
I will look closer at @stormi's articles
Now I remember why I landed on 8,3 instead of 8.2 : Black screen during initial installation routine after "xcp-ng xen is relinquishing vga console" :
This post describes the problem precisely:
https://forum.qubes-os.org/t/blank-screen-after-xen-xen-is-relinquishing-vga-console/10243Seems the super simple and cheap fanless gforce 710 GPU is not supported in 8.2 (but it works fine with 8.3).
I've failed finding 8.2 ISO with drivers for PCs included. Promising links have turned up dead. Efforts to make own build failed and cost me a lot of time.
Is there a guide - that actually works - on how one can include a standard gforce driver for linux on the USB installation drive for XCP-ng?
Or: Is it possible to use serial interface, SSH or something to complete installation?
Or is it easiest to just install 8.3 and not install any updates until the problem has been fixed?
I see this issue has been discussed - and fixed - previously: https://github.com/xcp-ng/xcp/issues/436
-
@Dataslak Are you booting the newest 8.2.1 IOS?
-
@Dataslak XO won't treat them any differently. You will simply have a support contract that covers three of your hosts. They would need to be segmented into their own pool (simply for support reasons), but your other servers will still be manageable from XOA.
Have you tried the new 8.2 ISO that was released back in December? I thought that it now included the i915 driver.
https://updates.xcp-ng.org/isos/8.2/xcp-ng-8.2.1-20231130.iso -
@Andrew I started looking at https://xcp-ng.org/docs/releases.html, which led me to standard release 8.2.1.
@Danp I have now downloaded and tried installing with your link. And this installation image worked
Thank you so much!I guess there are good reasons for not listing this on the download page (https://xcp-ng.org/#easy-to-install) ?
Had it been available then perhaps I and others would have tried it earlier, avoided 8.3, and perhaps also avoided the big mess I'm in right now?
I will now try to migrate VMs.
-
@Dataslak said in After installing updates: 0 bytes free, Control domain memory = 0B:
I guess there are good reasons for not listing this on the download page (https://xcp-ng.org/#easy-to-install) ?
I had the same question. Perhaps it is because 8.2 is a LTS release and they didn't want to switch to the newer ISO by default. It's also possible that someone simply forgot to update the link.
@olivierlambert will know for sure.
Had it been available then perhaps I and others would have tried it earlier, avoided 8.3, and perhaps also avoided the big mess I'm in right now?
Not sure that is true in your case since it sounds like you've been on 8.3 for > 6 months and the new ISO has been out for less time than that.
I will now try to migrate VMs.
I don't think that you can live migrate from 8.3 to 8.2, so you will need to export to XVA format and then import to the 8.2 host.
-
I'm not sure to understand. The link on XCP-ng.org is pointing to https://mirrors.xcp-ng.org/isos/8.2/xcp-ng-8.2.1-20231130.iso?https=1 (which is the latest ISO). What am I missing?
-
@olivierlambert The link at the top of https://xcp-ng.org/docs/release-8-2.html leads to https://mirrors.xcp-ng.org/isos/8.2/xcp-ng-8.2.1.iso?https=1
-
Ah so in the doc itself, understood, fixing it now!
edit: that's the old doc, cf https://docs.xcp-ng.org/releases/release-8-2/
I will be sure to redirect the old one to the new one.
-
@Danp where did you find the https://xcp-ng.org/docs/release-8-2.html link originally?
-
