Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2

Aimdev

@hannes_5253
This is a failure re getting usb pass through on boot up of VM correctly.
We use GPS usb devices for time keeping and despite hacking the various files (overridden on upgrades/patches etc) this issue has never been resolved despite comments on this board.

UPS & GPS support is important for obvious reasons, and needs an educated review into the problem.

We moved our UPS to a NetBsd server (not a VM, a separate system) and used NUT (added outside the tight xcp-nt regime) to
provide the essential power loss situation of the hypervisors.

Same with GPS, hosted on same BSD system.

Hannes_5253

@peek Thanks for the answer! I did this. I have to take another look at the configuration.

Hannes_5253

@aimdev thanks for the information!

Hannes_5253

@peek said in Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2:

yum --enablerepo=* install nut.x86_64

This video solved it for me (altough its in german)
YouTube Tut

Thread can be closed.

olivierlambert

Would you mind posting exactly the steps you did to make it work? That might be really helpful for the community (and maybe pushed to our official doc at some point!)

okynnor

I was able to make NUT work with XCP-NG. I used and followed the instructions in this post.

On issuing the command:

[21:56 xenserver1 ups]# ./xen-shutdown.sh

All VMs shutdown EXCEPT for the Xen Orchestra. I tried remove and reinstall Xen Tools

apt install xe-guest-utilities

I'm on Ubuntu 18.04

Is there something that should be done with Xen Orchestra? Like I said, all other VMs shutdown quickly and properly. I only have Ubuntu 18.04 and one instance of FreePBX. FreePBX shutdown properly too.

Update: it seems that after 5-10 minutes (not sure yet as I haven't timed it), the entire XCP-NG does shutdown but not sure if it was a forced of graceful shutdown on the VM running XO.

Hannes_5253

@olivierlambert Oh man, only saw this answer because I went back to my old post. I will do a full guide when I have some time and send it to you.

Hannes_5253

@olivierlambert
I wrote down the to-dos. You can use it for the official docs if you like! When you have questions, just answer on this Post.

Network UPS Tools
Installation on XCP-ng Host

At first: Connect your UPS.
Then run this commands to install NUT:

yum update

yum –enablerepo=* install epel-release

yum install nut

yum install nut-client

cd /etc/ups

wget https://github.com/serrc-techops/NUT-Configuration/blob/master/slave/xen/xen-shutdown.sh

chmod +x xen-shutdown.sh

Important Commands

upsc apcups (check UPS connection)
systemctl restart nut-server
(NUT-server reboot) | (Use this to apply configuration changes and to connect the UPS after System startup)
systemctl restart nut-monitor (NUT-monitor restart)
nut-scacnner -U (shows connected devices)

Configure the files stored in /etc/ups like this:
upsmon.conf

RUN_AS_USER root
MONITOR apcups@localhost 1 admin [PASSWORD] master
SHUTDOWNCMD „/etc/ups/xen-shutdown.sh“

nut.conf

MODE=netserver

ups.conf

pollinterval = 1
maxretry = 3
[apcups]
driver = usbhid-ups 
#->when you use USB devices
port = auto
desc = „[NAME]“
vendorid = [VENDORID]
productid = [PRODUCTID]
serial = „[SERIALNO]“

upsd.conf

LISTEN 0.0.0.0 3493

upsd.users

[admin]
password = [PASSWORD]
actions = set
actions= fsd
instcmds = ALL

Finish config Changes

systemctl restart nut-server
systemctl restart nut-monitor

Finished
Good to know: You can now also connect to your NUT server via the pfSense plugin by using the credentials in uspd.users to have a gui for your UPS. To do this you have to open up Port 3493 on your host.
Run this command to open the Port:

iptables -I INPUT -p tcp -m tcp --dport 3493 -j ACCEPT

Then save your changes to a config file and adjust Year, Month and Day in the config name. In my case it's the 22th of december in 2022:

iptables-save > /etc/sysconfig/iptables
mkdir /etc/iptables/
iptables-save > /etc/iptables/rules_2022-12-22_INPUT_ACCEPT_3493_NUT.v4 --> Optional, extra saving for Backups.

I recommend using this convention of naming your rule files. Its a lot easier to track your changes and to restore files.

lknite

@Hannes_5253 awesome! Thank you for this posting, it's helping me to get a nut client installed. Note for folks coming across this later I needed to use the following to install the nut-client:

yum -y install nut-client --enablerepo=epel

It would be cool if xcp-ng came pre-installed with nut-client and a default configuration that shuts down the xen server & turns off the ups after it finishes shutting down... similar to how truenas has a service which can be enabled to do so. You'd have to configure it of course, with the nut server to monitor, credentials, and some other options such as whether to actually powerdown the ups after shutting down or not, but maybe through a nice gui after turning it on. Seems like everyone should be using this setup ... just a matter of time till they are ready to.

Another reason this would be nice to be integrated, is that docs recommend setting up a user account for nut, one for the monitor so clients can access and another for the shutdown so one of the watchers doesn't change the shutdown script & nut provides a file that indicates when the ups should shutdown, so having that be the same on all xcp-ng systems rather than custom solution would be good.

Hannes_5253

@lknite Glad I could help and I agree.

olivierlambert

Hi,

Sadly, this use case is more meant for "non server-grade", where your machine isn't hosted in a datacenter (which is the default XCP-ng target/market).

However, I'm not against helping people doing the configuration or having an official guide to document it (I mean, in our doc)

Keep in mind we are always trying to find a balance between maintaining something (meaning time+money over other features) vs where the money is coming from (companies paying for support). If we pre-install all of this, it means we'll need to maintain it, while we still have a huge backlog of critical feature for our main (paying) target I hope you understand. Also, if we succeed and continue to grow that way with a far larger team than today, we could probably use that extra money from companies to improve integrations for the rest. But before doing it, we need to focus.

2b2bff

@olivierlambert I wonder how datacenters are doing this then in case of power loss? Or isn't that something of their concern?

olivierlambert

In a DC, you have usually 2x power feed per rack (one on the left, one on the right). So even if one power line/feed is down, it's not a problem, since all your machines are usually dual PSU.

Also, everything is backed by UPS (both feeds) inside the building, and then diesel generators are taking the load after 10/15 minutes (diesel takes few minutes to start).

So in a colo, you will never have to get your own UPS.

2b2bff

@olivierlambert in simple words: they just don't shut down. And if they would have to there are bigger problems to solve.
Thanks for answering...

gskger

@2b2bff I think that there are still some business scenarios, where this functionality would come in handy.

For example, we are running server rooms at our main production sites worldwide (no data centers, no colo, all on-site). Each features a three VMware hosts cluster with shared storage, backup, switches, and firewalls in an air-conditioned server room with adequate fire prevention/protection. All have two power feeds (A and B or left and right as Olivier called them) and all systems have dual PSUs, but the sites usually have only one power supplier and no generators.

Feed A goes to an UPS and feed B directly to the power line. If the power goes down, the UPS automatically triggers the vCenter, storage and backup to shut down when the UPS remaining runtime is below a certain threshold.

But I can perfectly understand Oliviers reasoning and there are workarounds. It all depends on your risk appetite (if that is a word in English).

2b2bff

@gskger yeah same on my place. I play around with xcp-ng at home, where this is not important, and there is no UPS. So it is not important here.
But I have built two hosts at work in a server room as well. Together with TrueNAS and a couple of other things. The APC UPS has a management card, so it can be reached via network, but somehow the hosts have to shut down gracefully if needed. That's how I stumpled upon this thread in the first place.

olivierlambert

I'm not saying it's not important in general (it depends on the use cases) But in our priorities (the "regular" DC world), it is not. For the "Edge computing" case, it might be more important, but the word "focus" doesn't have any meaning if we do everything at the same time.

And as usual, contributions are always welcome!

odeawan

Would just having the nut client be sufficient enough on the base XCP-NG to tigger a shutdown via NUT? Maintaining the NUT server seems to be a hold up and add extra complexity for XCP-NG. We all may have different DC power situations, but we could all use tools in our belt at home or the data center.

Hannes_5253

@odeawan Sure but you need a nut server. When you use the XCP-NG Server as a client you have to install the nut server on a different system. You can't install the nut server on a vm on the same host because the nut server must be installed on a system to shut down last.
So running this scenario is the most redundant and efficient way.

odeawan

Well I still don't think it is out of scope then for just the NUT client for enterprises or HomeLabs on XCP-NG.

SBCs are everywhere these days. Using a Raspberry Pi as the NUT Server in a homelab or a flavor of a more robust server for NUT seems like a solid choice. No USB passthrough to worry about, uptime on a 1500VA+ UPS, forget about it! There are a handful of guides how to get it running for a Pi or anything else (including this post).

As I think this though, I would still feel ok with another external device watching, waiting, and notifying me on a successful power off of any number of my hosts and VMs. Not all enterprises or homelabs can afford (even if their appetites want it) to get enterprise level power automations. We have our power. Break the chain. @olivierlambert