@Tristis-Oris Do you have HA enabled? Also, I believe that maintenance mode is supposed to be automatically disabled after a reboot
Posts made by nick.lloyd
-
RE: Maintenance mode auto disabled after some time
-
RE: "Xenserver 8" patching
@infecticide I won't speak for the folks at Vates, but in my (and by extension my employer's) experience, they do! And they treat you just like a regular paying customer. They've been super prompt and resolve issues faster than VMware!
-
RE: Getting MESSAGE_METHOD_UNKNOWN(VDI.get_cbt_enabled) with XenServer 7.1
@marcom The only thing that I would add (and I'm guessing you knew, adding this just in case) is to restore the backup of XO(A) and the pool metadata.
-
RE: Adding new host to pool failed
@tony And again, double check the BIOS updates. There may be a bug fix in one of them for Xen
-
RE: Adding new host to pool failed
@tony That should be fine, are you adding the older host to the newer host's pool, or vice-versa? My guess is that the newer one needs to be added to the older server's pool so that it can "properly" mask the CPU features
-
RE: Adding new host to pool failed
@tony Huh, interesting... What hardware is in each of the hosts? And can you double check that you're running similar-ish firmware/bios on both servers?
-
RE: Adding new host to pool failed
@tony It shouldn't matter since the hosts are in the same pool, it should list both of them. You can run it against both hosts to be sure
-
RE: Adding new host to pool failed
@tony Just to make sure, these are 8.2.1 hosts, or just 8.2?
Also, can you runxe host-all-editions
on one of the hosts? I'm sure this command may help too, but someone with more experience than me can chime in on that.Never mind, this is actually what you should run (for real, probably)
host-license-remove [host-uuid=host_uuid]
-
RE: Ubuntu 24.04 VMs not reporting IP addresses to XCP-NG 8.2.1
@salvadorb What happens if you uninstall those, and install the guest tools that come with XO? I believe the latest tools should 8.x (I don't remember what number off the top of my head).
-
RE: Recovering hosts from a "fire": Multiple hosts in a pool becoming the pool master
@Danp Just to make sure I have this correct, if I edited that file, entered the master IP address, and reboot the hosts, that would have resolved the issue?
-
RE: Recovering hosts from a "fire": Multiple hosts in a pool becoming the pool master
@Danp That makes sense, I did have HA enabled
-
Recovering hosts from a "fire": Multiple hosts in a pool becoming the pool master
This is technically a response to this post, but since the issue was resolved (and my fix wasn't the same), I thought that I'd make a write up here in case a poor soul finds themselves in this same situation. I'd also like suggestions on how I could've done this better.
Background
My homelab has some key oddities that should be mentioned first.
- My homelab is entirely colocated. I have no physical access to my servers.
- I have a VPN connection from the colocation to AWS. Since I didn't have direct router access (from the colocation) at the time, this was acting as my reverse proxy out to the public internet with Cloudflare ZeroTrust in front for some security (it's a homelab, I'm not storing HIPPAA info, I'm not too concerned about it).
- On AWS, I have one Docker VM with some services/a reverse proxy. Another VM is my XO From Source that manages the colocated servers. My last VM was a Windows VM just for managing my homelab easier than having to open more ports and configure more routing.
The Issue
One day, I logged into XO to add another host to my pool, and found that the pool didn't show up. So I restarted and updated XO, which didn't resolve the issue. I went to the BMC on one of my servers, and found this:
I restarted the server, but after 300 seconds (and every 300 seconds subsequently) the error would appear again. I logged into the BMC on my other servers, and they were in a similar state.
Now for the reason I mentioned the post at the beginning of my post. I could not SSH into Dom0 because the management network and NICs were no longer seen on any of my XCP-ng hosts despite showing up properly in
ifconfig
.After many hours of troubleshooting, I eventually found out that each of the hosts in the pool thought they were the "master" server, each with a different list of hosts they thought were slaves. Only one server would be accessible from XO at a time, and I couldn't force remove them from the pool since all of them thought they were the pool master.
Resolution
I figured I'd have to migrate these hosts out of the pool somehow. I ended up installing XCP-ng on another server to get another pool created (although I'm sure a VM would have worked if necessary). From XO, I could select "Add Hosts" from the pool option and move one server at a time (
xe pool-join
probably would have worked too, but after 5 hours in the CLI that day, I was ready to do things from the UI). Not only did it move the host to the new pool, but the management network came back and the host was accessible, problem resolved!RCA/Questions
- This was not entirely an XCP-ng fault. The day that the issues started happening, there were record temperatures in the environment where these servers were located (105F or 41C), causing them to power off in the first place.
- Should I have done anything differently? I couldn't find a way to forcefully demote a server from Master to Slave.
-
RE: XO & XOA New Look?
@wtdrisco They will both be getting a facelift, but at different times! You can check out this article for more.
-
RE: Threadripper TRX50 7960x....yaay/nay?
@sluflyer06 IIRC some people were having issues with XCP-ng 8.2.1 and AMD platforms, but their issues were resolved in 8.3. So if I recalled correctly, it's a good idea, but wait a few weeks for 8.3 to be released
-
RE: problem with export or moving VM between pools
@wtdrisco XOA is an appliance that is directly installed on your XCP-ng host. Think of it like any modern vCenter appliance. However because Xen Orchestra from sources is built on Linux, it's sort-of equivalent to Windows Admin Center for Hyper-V.
-
RE: Help: Clean shutdown of Host, now no network or VMs are detected
@CodeMercenary Ahh okay. One of my hosts dropped its network interfaces too, similar to what you stated. Unfortunately I had to shut my hosts down for a power upgrade at my colocation, but I should have them back up tonight and I'll report back what I find. What's the hardware on the affected host (just in case I find something similar between our setups)?
-
RE: Help: Clean shutdown of Host, now no network or VMs are detected
@CodeMercenary I'd certainly give it a reboot, it can't hurt if there's no VMs on it. There were patches released a few days ago as well!
-
RE: Some guidelines for sizing a XO server ?
@HolgiB Perfect! The other thought I had would be adding more XO VMs to distribute the backup load, or add a Backup Proxy (assuming you used this script, you can deploy a proxy in XO by running the script again and selecting the "deploy proxy" option).
-
RE: Mirror backup confusion Full vs Delta vs Incremental
@mjr99 You should just be able to add another target in the same backup job. Unfortunately I had to shut off my home lab for a bit, so I can't grab screenshots
-
RE: Some guidelines for sizing a XO server ?
@HolgiB XO doesn't require a huge amount of resources. There are limits of having 64 hosts in one pool, but that's the only one I'm aware of.
I'd start with 2c 4gb RAM for anything relatively small (<100 VMs). All of the processing is done on the pool master (unlike VMware), so it's unlikely you'll run into performance hits unless you plan on using the backup feature heavily. It's also pretty easy to change afterwards too. Just select the XO VM, click on the field you want to change, enter desired values, and reboot!