XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2

    Scheduled Pinned Locked Moved Compute
    29 Posts 12 Posters 16.8k Views 13 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H Offline
      Hannes_5253
      last edited by Hannes_5253

      Hey there,
      I have a very specific question:
      Is there a way to get PowerChute running on a XCP-ng Host? I already found this article from Citrix:
      https://support.citrix.com/article/CTX119910
      and this from APC:
      https://www.apc.com/us/en/faqs/FA159764/

      The problem is I don't have a Network Management Card installed or the possibility of adding one. I'm dependent on a USB Connection. So I was wondering how I can configure a shutdown of all VMs and the Host with a USB Connection? It's important for me to understand the process. Do I need an seperate PC to manage this? Is there a solution with USB passthrough and one VM executes a shutdown script on the Host? I would be thankful on some help/related articles or anything you can give me to figure it out myself.

      I'm running on XCP-ng Version 8.2 an I have a APC Back-UPS 700.

      H 1 Reply Last reply Reply Quote 0
      • H Offline
        Hannes_5253 @Hannes_5253
        last edited by Hannes_5253

        I found this post:
        https://voice1.me/automated-shutdown-xenserver-7-x-with-network-ups-tool/

        I just used the Driver type for USB: usbhid-ups

        You can find the correct driver type for your system here:
        https://networkupstools.org/stable-hcl.html

        Here is the guide for USB (partly copied from the article):

        IMPORTANT: The way I made is is not recommend! Your pfSense should normally not run on a VM using this method but it still works. It's recommended to use a physical pfSense to perform this setup.

        Requirement:

        • a VM running pfSense
        • all VMs on your XCP-ng host have to run guest gtilities.
        • a USP with USB Connection
        • XCP-ng Center
        1. Connecting the USP to a VM via USB
        • Plug-in the USP via USB into the Server
        • Enable USB-passthrough in the USB Tab of your Host (XCP-ng Center)
        • Shutdown the pfSense, attach the USB device to the pfSense VM in the Properties Tab of the VM
        • Start the VM
        1. Installing NUT
          If you are using pfSense, you can install nut from the Package Manager. Once installed you will be able to access the setup from Services > UPS.
        • Select “Local USB” from the UPS Type and provide a UPS Name.
        • add this to your uspd.conf:
        ACL all 0.0.0.0/0
        ACCEPT all
        
        • add this to your uspd.users:
        [(YOUR USERNAME)]
        password = (YOUR PASSWORD)
        allowfrom = all
        instcmds = ALL
        

        Attention! The username has to be written with the brackets! ie: [admin]

        • click on save and pfSense should now detect your UPS and give you its information in the UPS Dashboard.
        1. Configure the XCP-ng host as NUT slave (pfSense is master)
        • By default the built in repository does not have the required library, so we need to enable the epel repository before we can install the nut-client
        yum --enablerepo=* install epel-release
        
        yum --enablerepo=* install nut-client.x86_64
        
        • Now we need to tell NUT that we want it to listen to an existing NUT Server, in other words, we want to be a “client”.
        nano /etc/ups/nut.conf
        MODE=netclient
        
        • Now that NUT knows it will be retrieving information from another network server, we need to tell it what network server to “MONITOR”. You will need to modify only 2 variables in the file /etc/ups/upsmon.conf but take a look around because you can tweak how the NUT Client responds by modifying the settings in this file. For now, lets just tell NUT where to retrieve its information from.
        nano /etc/ups/upsmon.conf
        MONITOR (NAME OF YOUR UPS)@(PFSENSE IP) 1 (USERNAME) (PASSWORD) slave
        SHUTDOWNCMD "/etc/ups/xen-shutdown.sh"
        
        • You probably already have a SHUTDOWNCMD in your configuration Just add a comment “#” to the start of the line and add the one provided above. Speaking of the SHUTDOWNCMD, you will notice that we are calling a specific script. This script can be downloaded here.
        cd /etc/ups
        wget https://raw.githubusercontent.com/serrc-techops/NUT-Configuration/master/slave/xen/xen-shutdown.sh
        chmod +x xen-shutdown.sh
        

        !!! I wasn't able to perform the next step because the service started to fail afterwards!!!
        The shutdown still works. I try to figure out a solution and add it here.

        So on a power failure, the NUT Master sends a message to all subscribed NUT Clients, and the NUT Client shuts down based on its configuration. Wonderful Now our Xen Guests can shutdown, and so can our server. But what happens when we power them back up? We need to make sure that NUT starts on our XenServer.

        • To do so create a new file, /etc/systemd/system/nut-monitor.service and add the following to it.
        [Unit]
        Description=Network UPS Tools - XenServer Shutdown
        After=local-fs.target network.target
         
        [Service]
        ExecStart=/usr/sbin/upsmon
        PIDFile=/run/nut/upsmon.pid
        Type=forking
         
        [Install]
        WantedBy=multi-user.target
        

        Now that we have a system startup script in place, lets make sure it runs on start up.

        systemctl enable nut-monitor.service
        systemctl daemon-reload
        systemctl start nut-monitor
        
        1. Testing shutdown
        • Execute this command in the Shell of your pfSense:
          This will simulate a power cut!!!
        upsmon -c fsd
        

        If you have any questions ask me below.

        There is one problem: you have to reattach the USB to the VM after every host reboot!

        PeekP A 2 Replies Last reply Reply Quote 1
        • PeekP Offline
          Peek @Hannes_5253
          last edited by

          Hannes_5253 Any reason for not prefering to run the NUT server directly on the XCP-NG host ?

          H 1 Reply Last reply Reply Quote 0
          • H Offline
            Hannes_5253 @Peek
            last edited by

            Peek said in Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2:

            Hannes_5253 Any reason for not prefering to run the NUT server directly on the XCP-NG host ?

            I didn't got it with the Host, maybe I'm too stupid.

            PeekP 1 Reply Last reply Reply Quote 0
            • PeekP Offline
              Peek @Hannes_5253
              last edited by

              I'm using XCP-ng and except for requiring the testing repo, (specifically only for NUT), it's working perfectly. The package name is a bit "non-standard".

              yum --enablerepo=* install nut.x86_64
              

              The UPS is connected over USB and on a power failure, the hypervisor shuts all the VMs gracefully down with a simple "shutdown -h +0" specified in upsmon.conf.

              H 1 Reply Last reply Reply Quote 0
              • A Offline
                Aimdev @Hannes_5253
                last edited by

                Hannes_5253
                This is a failure re getting usb pass through on boot up of VM correctly.
                We use GPS usb devices for time keeping and despite hacking the various files (overridden on upgrades/patches etc) this issue has never been resolved despite comments on this board.

                UPS & GPS support is important for obvious reasons, and needs an educated review into the problem.

                We moved our UPS to a NetBsd server (not a VM, a separate system) and used NUT (added outside the tight xcp-nt regime) to
                provide the essential power loss situation of the hypervisors.

                Same with GPS, hosted on same BSD system.

                H 1 Reply Last reply Reply Quote 1
                • H Offline
                  Hannes_5253 @Peek
                  last edited by

                  Peek Thanks for the answer! I did this. I have to take another look at the configuration.

                  1 Reply Last reply Reply Quote 0
                  • H Offline
                    Hannes_5253 @Aimdev
                    last edited by

                    Aimdev thanks for the information!

                    1 Reply Last reply Reply Quote 0
                    • H Offline
                      Hannes_5253
                      last edited by Hannes_5253

                      Peek said in Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2:

                      yum --enablerepo=* install nut.x86_64

                      This video solved it for me (altough its in german)
                      YouTube Tut

                      Thread can be closed.

                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by olivierlambert

                        Would you mind posting exactly the steps you did to make it work? That might be really helpful for the community 🙂 (and maybe pushed to our official doc at some point!)

                        H 2 Replies Last reply Reply Quote 1
                        • okynnorO Offline
                          okynnor
                          last edited by okynnor

                          I was able to make NUT work with XCP-NG. I used and followed the instructions in this post.

                          On issuing the command:

                          [21:56 xenserver1 ups]# ./xen-shutdown.sh 
                          

                          All VMs shutdown EXCEPT for the Xen Orchestra. I tried remove and reinstall Xen Tools

                          apt install xe-guest-utilities
                          

                          I'm on Ubuntu 18.04

                          Is there something that should be done with Xen Orchestra? Like I said, all other VMs shutdown quickly and properly. I only have Ubuntu 18.04 and one instance of FreePBX. FreePBX shutdown properly too.

                          Update: it seems that after 5-10 minutes (not sure yet as I haven't timed it), the entire XCP-NG does shutdown but not sure if it was a forced of graceful shutdown on the VM running XO.

                          1 Reply Last reply Reply Quote 0
                          • H Offline
                            Hannes_5253 @olivierlambert
                            last edited by Hannes_5253

                            olivierlambert Oh man, only saw this answer because I went back to my old post. I will do a full guide when I have some time and send it to you.

                            1 Reply Last reply Reply Quote 2
                            • H Offline
                              Hannes_5253 @olivierlambert
                              last edited by Hannes_5253

                              olivierlambert
                              I wrote down the to-dos. You can use it for the official docs if you like! When you have questions, just answer on this Post.

                              Network UPS Tools
                              Installation on XCP-ng Host

                              At first: Connect your UPS.
                              Then run this commands to install NUT:

                              yum update
                              
                              yum –enablerepo=* install epel-release
                              
                              yum install nut
                              
                              yum install nut-client
                              
                              cd /etc/ups
                              
                              wget https://github.com/serrc-techops/NUT-Configuration/blob/master/slave/xen/xen-shutdown.sh
                              
                              chmod +x xen-shutdown.sh
                              

                              Important Commands

                              • upsc apcups (check UPS connection)

                              • systemctl restart nut-server
                                (NUT-server reboot) | (Use this to apply configuration changes and to connect the UPS after System startup)

                              • systemctl restart nut-monitor (NUT-monitor restart)

                              • nut-scacnner -U (shows connected devices)

                              Configure the files stored in /etc/ups like this:
                              upsmon.conf

                              RUN_AS_USER root
                              MONITOR apcups@localhost 1 admin [PASSWORD] master
                              SHUTDOWNCMD „/etc/ups/xen-shutdown.sh“
                              

                              nut.conf

                              MODE=netserver
                              

                              ups.conf

                              pollinterval = 1
                              maxretry = 3
                              [apcups]
                              driver = usbhid-ups 
                              #->when you use USB devices
                              port = auto
                              desc = „[NAME]“
                              vendorid = [VENDORID]
                              productid = [PRODUCTID]
                              serial = „[SERIALNO]“
                              

                              upsd.conf

                              LISTEN 0.0.0.0 3493
                              

                              upsd.users

                              [admin]
                              password = [PASSWORD]
                              actions = set
                              actions= fsd
                              instcmds = ALL
                              

                              Finish config Changes

                              systemctl restart nut-server
                              systemctl restart nut-monitor
                              

                              Finished
                              Good to know: You can now also connect to your NUT server via the pfSense plugin by using the credentials in uspd.users to have a gui for your UPS. To do this you have to open up Port 3493 on your host.
                              Run this command to open the Port:

                              iptables -I INPUT -p tcp -m tcp --dport 3493 -j ACCEPT
                              

                              Then save your changes to a config file and adjust Year, Month and Day in the config name. In my case it's the 22th of december in 2022:

                              iptables-save > /etc/sysconfig/iptables
                              mkdir /etc/iptables/
                              iptables-save > /etc/iptables/rules_2022-12-22_INPUT_ACCEPT_3493_NUT.v4 --> Optional, extra saving for Backups.
                              

                              I recommend using this convention of naming your rule files. Its a lot easier to track your changes and to restore files.

                              lkniteL 1 Reply Last reply Reply Quote 3
                              • lkniteL Offline
                                lknite @Hannes_5253
                                last edited by lknite

                                Hannes_5253 awesome! Thank you for this posting, it's helping me to get a nut client installed. Note for folks coming across this later I needed to use the following to install the nut-client:

                                yum -y install nut-client --enablerepo=epel
                                

                                It would be cool if xcp-ng came pre-installed with nut-client and a default configuration that shuts down the xen server & turns off the ups after it finishes shutting down... similar to how truenas has a service which can be enabled to do so. You'd have to configure it of course, with the nut server to monitor, credentials, and some other options such as whether to actually powerdown the ups after shutting down or not, but maybe through a nice gui after turning it on. Seems like everyone should be using this setup ... just a matter of time till they are ready to.

                                Another reason this would be nice to be integrated, is that docs recommend setting up a user account for nut, one for the monitor so clients can access and another for the shutdown so one of the watchers doesn't change the shutdown script & nut provides a file that indicates when the ups should shutdown, so having that be the same on all xcp-ng systems rather than custom solution would be good.

                                H 1 Reply Last reply Reply Quote 3
                                • H Offline
                                  Hannes_5253 @lknite
                                  last edited by Hannes_5253

                                  lknite Glad I could help and I agree. 👍🏻

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Offline
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by olivierlambert

                                    Hi,

                                    Sadly, this use case is more meant for "non server-grade", where your machine isn't hosted in a datacenter (which is the default XCP-ng target/market).

                                    However, I'm not against helping people doing the configuration or having an official guide to document it (I mean, in our doc)

                                    Keep in mind we are always trying to find a balance between maintaining something (meaning time+money over other features) vs where the money is coming from (companies paying for support). If we pre-install all of this, it means we'll need to maintain it, while we still have a huge backlog of critical feature for our main (paying) target 🙂 I hope you understand. Also, if we succeed and continue to grow that way with a far larger team than today, we could probably use that extra money from companies to improve integrations for the rest. But before doing it, we need to focus.

                                    2b2bff2 1 Reply Last reply Reply Quote 2
                                    • 2b2bff2 Offline
                                      2b2bff @olivierlambert
                                      last edited by

                                      olivierlambert I wonder how datacenters are doing this then in case of power loss? Or isn't that something of their concern?

                                      1 Reply Last reply Reply Quote 1
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by olivierlambert

                                        In a DC, you have usually 2x power feed per rack (one on the left, one on the right). So even if one power line/feed is down, it's not a problem, since all your machines are usually dual PSU.

                                        Also, everything is backed by UPS (both feeds) inside the building, and then diesel generators are taking the load after 10/15 minutes (diesel takes few minutes to start).

                                        So in a colo, you will never have to get your own UPS.

                                        2b2bff2 1 Reply Last reply Reply Quote 1
                                        • 2b2bff2 Offline
                                          2b2bff @olivierlambert
                                          last edited by

                                          olivierlambert in simple words: they just don't shut down. And if they would have to there are bigger problems to solve.
                                          Thanks for answering...

                                          gskgerG 1 Reply Last reply Reply Quote 1
                                          • gskgerG Offline
                                            gskger Top contributor @2b2bff
                                            last edited by gskger

                                            2b2bff I think that there are still some business scenarios, where this functionality would come in handy.

                                            For example, we are running server rooms at our main production sites worldwide (no data centers, no colo, all on-site). Each features a three VMware hosts cluster with shared storage, backup, switches, and firewalls in an air-conditioned server room with adequate fire prevention/protection. All have two power feeds (A and B or left and right as Olivier called them) and all systems have dual PSUs, but the sites usually have only one power supplier and no generators.

                                            Feed A goes to an UPS and feed B directly to the power line. If the power goes down, the UPS automatically triggers the vCenter, storage and backup to shut down when the UPS remaining runtime is below a certain threshold.

                                            But I can perfectly understand Oliviers reasoning and there are workarounds. It all depends on your risk appetite (if that is a word in English).

                                            2b2bff2 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post