Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2

Hannes_5253

Hey there,
I have a very specific question:
Is there a way to get PowerChute running on a XCP-ng Host? I already found this article from Citrix:
https://support.citrix.com/article/CTX119910
and this from APC:
https://www.apc.com/us/en/faqs/FA159764/

The problem is I don't have a Network Management Card installed or the possibility of adding one. I'm dependent on a USB Connection. So I was wondering how I can configure a shutdown of all VMs and the Host with a USB Connection? It's important for me to understand the process. Do I need an seperate PC to manage this? Is there a solution with USB passthrough and one VM executes a shutdown script on the Host? I would be thankful on some help/related articles or anything you can give me to figure it out myself.

I'm running on XCP-ng Version 8.2 an I have a APC Back-UPS 700.

Hannes_5253

I found this post:
https://voice1.me/automated-shutdown-xenserver-7-x-with-network-ups-tool/

I just used the Driver type for USB: usbhid-ups

You can find the correct driver type for your system here:
https://networkupstools.org/stable-hcl.html

Here is the guide for USB (partly copied from the article):

IMPORTANT: The way I made is is not recommend! Your pfSense should normally not run on a VM using this method but it still works. It's recommended to use a physical pfSense to perform this setup.

Requirement:

a VM running pfSense
all VMs on your XCP-ng host have to run guest gtilities.
a USP with USB Connection
XCP-ng Center

Connecting the USP to a VM via USB

Plug-in the USP via USB into the Server
Enable USB-passthrough in the USB Tab of your Host (XCP-ng Center)
Shutdown the pfSense, attach the USB device to the pfSense VM in the Properties Tab of the VM
Start the VM

Installing NUT
If you are using pfSense, you can install nut from the Package Manager. Once installed you will be able to access the setup from Services > UPS.

Select “Local USB” from the UPS Type and provide a UPS Name.
add this to your uspd.conf:

ACL all 0.0.0.0/0
ACCEPT all

add this to your uspd.users:

[(YOUR USERNAME)]
password = (YOUR PASSWORD)
allowfrom = all
instcmds = ALL

Attention! The username has to be written with the brackets! ie: [admin]

click on save and pfSense should now detect your UPS and give you its information in the UPS Dashboard.

Configure the XCP-ng host as NUT slave (pfSense is master)

By default the built in repository does not have the required library, so we need to enable the epel repository before we can install the nut-client

yum --enablerepo=* install epel-release

yum --enablerepo=* install nut-client.x86_64

Now we need to tell NUT that we want it to listen to an existing NUT Server, in other words, we want to be a “client”.

nano /etc/ups/nut.conf
MODE=netclient

Now that NUT knows it will be retrieving information from another network server, we need to tell it what network server to “MONITOR”. You will need to modify only 2 variables in the file /etc/ups/upsmon.conf but take a look around because you can tweak how the NUT Client responds by modifying the settings in this file. For now, lets just tell NUT where to retrieve its information from.

nano /etc/ups/upsmon.conf
MONITOR (NAME OF YOUR UPS)@(PFSENSE IP) 1 (USERNAME) (PASSWORD) slave
SHUTDOWNCMD "/etc/ups/xen-shutdown.sh"

You probably already have a SHUTDOWNCMD in your configuration Just add a comment “#” to the start of the line and add the one provided above. Speaking of the SHUTDOWNCMD, you will notice that we are calling a specific script. This script can be downloaded here.

cd /etc/ups
wget https://raw.githubusercontent.com/serrc-techops/NUT-Configuration/master/slave/xen/xen-shutdown.sh
chmod +x xen-shutdown.sh

!!! I wasn't able to perform the next step because the service started to fail afterwards!!!
The shutdown still works. I try to figure out a solution and add it here.

So on a power failure, the NUT Master sends a message to all subscribed NUT Clients, and the NUT Client shuts down based on its configuration. Wonderful Now our Xen Guests can shutdown, and so can our server. But what happens when we power them back up? We need to make sure that NUT starts on our XenServer.

To do so create a new file, /etc/systemd/system/nut-monitor.service and add the following to it.

[Unit]
Description=Network UPS Tools - XenServer Shutdown
After=local-fs.target network.target
 
[Service]
ExecStart=/usr/sbin/upsmon
PIDFile=/run/nut/upsmon.pid
Type=forking
 
[Install]
WantedBy=multi-user.target

Now that we have a system startup script in place, lets make sure it runs on start up.

systemctl enable nut-monitor.service
systemctl daemon-reload
systemctl start nut-monitor

Testing shutdown

Execute this command in the Shell of your pfSense:
This will simulate a power cut!!!

upsmon -c fsd

If you have any questions ask me below.

There is one problem: you have to reattach the USB to the VM after every host reboot!

Peek

@hannes_5253 Any reason for not prefering to run the NUT server directly on the XCP-NG host ?

Hannes_5253

@peek said in Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2:

@hannes_5253 Any reason for not prefering to run the NUT server directly on the XCP-NG host ?

I didn't got it with the Host, maybe I'm too stupid.

Peek

I'm using XCP-ng and except for requiring the testing repo, (specifically only for NUT), it's working perfectly. The package name is a bit "non-standard".

yum --enablerepo=* install nut.x86_64

The UPS is connected over USB and on a power failure, the hypervisor shuts all the VMs gracefully down with a simple "shutdown -h +0" specified in upsmon.conf.

Aimdev

@hannes_5253
This is a failure re getting usb pass through on boot up of VM correctly.
We use GPS usb devices for time keeping and despite hacking the various files (overridden on upgrades/patches etc) this issue has never been resolved despite comments on this board.

UPS & GPS support is important for obvious reasons, and needs an educated review into the problem.

We moved our UPS to a NetBsd server (not a VM, a separate system) and used NUT (added outside the tight xcp-nt regime) to
provide the essential power loss situation of the hypervisors.

Same with GPS, hosted on same BSD system.

Hannes_5253

@peek Thanks for the answer! I did this. I have to take another look at the configuration.

Hannes_5253

@aimdev thanks for the information!

Hannes_5253

@peek said in Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2:

yum --enablerepo=* install nut.x86_64

This video solved it for me (altough its in german)
YouTube Tut

Thread can be closed.

olivierlambert

Would you mind posting exactly the steps you did to make it work? That might be really helpful for the community (and maybe pushed to our official doc at some point!)

okynnor

I was able to make NUT work with XCP-NG. I used and followed the instructions in this post.

On issuing the command:

[21:56 xenserver1 ups]# ./xen-shutdown.sh

All VMs shutdown EXCEPT for the Xen Orchestra. I tried remove and reinstall Xen Tools

apt install xe-guest-utilities

I'm on Ubuntu 18.04

Is there something that should be done with Xen Orchestra? Like I said, all other VMs shutdown quickly and properly. I only have Ubuntu 18.04 and one instance of FreePBX. FreePBX shutdown properly too.

Update: it seems that after 5-10 minutes (not sure yet as I haven't timed it), the entire XCP-NG does shutdown but not sure if it was a forced of graceful shutdown on the VM running XO.

Hannes_5253

@olivierlambert Oh man, only saw this answer because I went back to my old post. I will do a full guide when I have some time and send it to you.

Hannes_5253

@olivierlambert
I wrote down the to-dos. You can use it for the official docs if you like! When you have questions, just answer on this Post.

Network UPS Tools
Installation on XCP-ng Host

At first: Connect your UPS.
Then run this commands to install NUT:

yum update

yum –enablerepo=* install epel-release

yum install nut

yum install nut-client

cd /etc/ups

wget https://github.com/serrc-techops/NUT-Configuration/blob/master/slave/xen/xen-shutdown.sh

chmod +x xen-shutdown.sh

Important Commands

upsc apcups (check UPS connection)
systemctl restart nut-server
(NUT-server reboot) | (Use this to apply configuration changes and to connect the UPS after System startup)
systemctl restart nut-monitor (NUT-monitor restart)
nut-scacnner -U (shows connected devices)

Configure the files stored in /etc/ups like this:
upsmon.conf

RUN_AS_USER root
MONITOR apcups@localhost 1 admin [PASSWORD] master
SHUTDOWNCMD „/etc/ups/xen-shutdown.sh“

nut.conf

MODE=netserver

ups.conf

pollinterval = 1
maxretry = 3
[apcups]
driver = usbhid-ups 
#->when you use USB devices
port = auto
desc = „[NAME]“
vendorid = [VENDORID]
productid = [PRODUCTID]
serial = „[SERIALNO]“

upsd.conf

LISTEN 0.0.0.0 3493

upsd.users

[admin]
password = [PASSWORD]
actions = set
actions= fsd
instcmds = ALL

Finish config Changes

systemctl restart nut-server
systemctl restart nut-monitor

Finished
Good to know: You can now also connect to your NUT server via the pfSense plugin by using the credentials in uspd.users to have a gui for your UPS. To do this you have to open up Port 3493 on your host.
Run this command to open the Port:

iptables -I INPUT -p tcp -m tcp --dport 3493 -j ACCEPT

Then save your changes to a config file and adjust Year, Month and Day in the config name. In my case it's the 22th of december in 2022:

iptables-save > /etc/sysconfig/iptables
mkdir /etc/iptables/
iptables-save > /etc/iptables/rules_2022-12-22_INPUT_ACCEPT_3493_NUT.v4 --> Optional, extra saving for Backups.

I recommend using this convention of naming your rule files. Its a lot easier to track your changes and to restore files.

lknite

@Hannes_5253 awesome! Thank you for this posting, it's helping me to get a nut client installed. Note for folks coming across this later I needed to use the following to install the nut-client:

yum -y install nut-client --enablerepo=epel

It would be cool if xcp-ng came pre-installed with nut-client and a default configuration that shuts down the xen server & turns off the ups after it finishes shutting down... similar to how truenas has a service which can be enabled to do so. You'd have to configure it of course, with the nut server to monitor, credentials, and some other options such as whether to actually powerdown the ups after shutting down or not, but maybe through a nice gui after turning it on. Seems like everyone should be using this setup ... just a matter of time till they are ready to.

Another reason this would be nice to be integrated, is that docs recommend setting up a user account for nut, one for the monitor so clients can access and another for the shutdown so one of the watchers doesn't change the shutdown script & nut provides a file that indicates when the ups should shutdown, so having that be the same on all xcp-ng systems rather than custom solution would be good.

Hannes_5253

@lknite Glad I could help and I agree.

olivierlambert

Hi,

Sadly, this use case is more meant for "non server-grade", where your machine isn't hosted in a datacenter (which is the default XCP-ng target/market).

However, I'm not against helping people doing the configuration or having an official guide to document it (I mean, in our doc)

Keep in mind we are always trying to find a balance between maintaining something (meaning time+money over other features) vs where the money is coming from (companies paying for support). If we pre-install all of this, it means we'll need to maintain it, while we still have a huge backlog of critical feature for our main (paying) target I hope you understand. Also, if we succeed and continue to grow that way with a far larger team than today, we could probably use that extra money from companies to improve integrations for the rest. But before doing it, we need to focus.

2b2bff

@olivierlambert I wonder how datacenters are doing this then in case of power loss? Or isn't that something of their concern?

olivierlambert

In a DC, you have usually 2x power feed per rack (one on the left, one on the right). So even if one power line/feed is down, it's not a problem, since all your machines are usually dual PSU.

Also, everything is backed by UPS (both feeds) inside the building, and then diesel generators are taking the load after 10/15 minutes (diesel takes few minutes to start).

So in a colo, you will never have to get your own UPS.

2b2bff

@olivierlambert in simple words: they just don't shut down. And if they would have to there are bigger problems to solve.
Thanks for answering...

gskger

@2b2bff I think that there are still some business scenarios, where this functionality would come in handy.

For example, we are running server rooms at our main production sites worldwide (no data centers, no colo, all on-site). Each features a three VMware hosts cluster with shared storage, backup, switches, and firewalls in an air-conditioned server room with adequate fire prevention/protection. All have two power feeds (A and B or left and right as Olivier called them) and all systems have dual PSUs, but the sites usually have only one power supplier and no generators.

Feed A goes to an UPS and feed B directly to the power line. If the power goes down, the UPS automatically triggers the vCenter, storage and backup to shut down when the UPS remaining runtime is below a certain threshold.

But I can perfectly understand Oliviers reasoning and there are workarounds. It all depends on your risk appetite (if that is a word in English).