XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2

    Scheduled Pinned Locked Moved Compute
    29 Posts 12 Posters 17.0k Views 13 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • PeekP Offline
      Peek @Hannes_5253
      last edited by

      @hannes_5253 Any reason for not prefering to run the NUT server directly on the XCP-NG host ?

      H 1 Reply Last reply Reply Quote 0
      • H Offline
        Hannes_5253 @Peek
        last edited by

        @peek said in Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2:

        @hannes_5253 Any reason for not prefering to run the NUT server directly on the XCP-NG host ?

        I didn't got it with the Host, maybe I'm too stupid.

        PeekP 1 Reply Last reply Reply Quote 0
        • PeekP Offline
          Peek @Hannes_5253
          last edited by

          I'm using XCP-ng and except for requiring the testing repo, (specifically only for NUT), it's working perfectly. The package name is a bit "non-standard".

          yum --enablerepo=* install nut.x86_64
          

          The UPS is connected over USB and on a power failure, the hypervisor shuts all the VMs gracefully down with a simple "shutdown -h +0" specified in upsmon.conf.

          H 1 Reply Last reply Reply Quote 0
          • A Offline
            Aimdev @Hannes_5253
            last edited by

            @hannes_5253
            This is a failure re getting usb pass through on boot up of VM correctly.
            We use GPS usb devices for time keeping and despite hacking the various files (overridden on upgrades/patches etc) this issue has never been resolved despite comments on this board.

            UPS & GPS support is important for obvious reasons, and needs an educated review into the problem.

            We moved our UPS to a NetBsd server (not a VM, a separate system) and used NUT (added outside the tight xcp-nt regime) to
            provide the essential power loss situation of the hypervisors.

            Same with GPS, hosted on same BSD system.

            H 1 Reply Last reply Reply Quote 1
            • H Offline
              Hannes_5253 @Peek
              last edited by

              @peek Thanks for the answer! I did this. I have to take another look at the configuration.

              1 Reply Last reply Reply Quote 0
              • H Offline
                Hannes_5253 @Aimdev
                last edited by

                @aimdev thanks for the information!

                1 Reply Last reply Reply Quote 0
                • H Offline
                  Hannes_5253
                  last edited by Hannes_5253

                  @peek said in Performing automated shutdown during a power failure using a USB-UPS with NUT - XCP-ng 8.2:

                  yum --enablerepo=* install nut.x86_64

                  This video solved it for me (altough its in german)
                  YouTube Tut

                  Thread can be closed.

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates πŸͺ Co-Founder CEO
                    last edited by olivierlambert

                    Would you mind posting exactly the steps you did to make it work? That might be really helpful for the community πŸ™‚ (and maybe pushed to our official doc at some point!)

                    H 2 Replies Last reply Reply Quote 1
                    • okynnorO Offline
                      okynnor
                      last edited by okynnor

                      I was able to make NUT work with XCP-NG. I used and followed the instructions in this post.

                      On issuing the command:

                      [21:56 xenserver1 ups]# ./xen-shutdown.sh 
                      

                      All VMs shutdown EXCEPT for the Xen Orchestra. I tried remove and reinstall Xen Tools

                      apt install xe-guest-utilities
                      

                      I'm on Ubuntu 18.04

                      Is there something that should be done with Xen Orchestra? Like I said, all other VMs shutdown quickly and properly. I only have Ubuntu 18.04 and one instance of FreePBX. FreePBX shutdown properly too.

                      Update: it seems that after 5-10 minutes (not sure yet as I haven't timed it), the entire XCP-NG does shutdown but not sure if it was a forced of graceful shutdown on the VM running XO.

                      1 Reply Last reply Reply Quote 0
                      • H Offline
                        Hannes_5253 @olivierlambert
                        last edited by Hannes_5253

                        @olivierlambert Oh man, only saw this answer because I went back to my old post. I will do a full guide when I have some time and send it to you.

                        1 Reply Last reply Reply Quote 2
                        • H Offline
                          Hannes_5253 @olivierlambert
                          last edited by Hannes_5253

                          @olivierlambert
                          I wrote down the to-dos. You can use it for the official docs if you like! When you have questions, just answer on this Post.

                          Network UPS Tools
                          Installation on XCP-ng Host

                          At first: Connect your UPS.
                          Then run this commands to install NUT:

                          yum update
                          
                          yum –enablerepo=* install epel-release
                          
                          yum install nut
                          
                          yum install nut-client
                          
                          cd /etc/ups
                          
                          wget https://github.com/serrc-techops/NUT-Configuration/blob/master/slave/xen/xen-shutdown.sh
                          
                          chmod +x xen-shutdown.sh
                          

                          Important Commands

                          • upsc apcups (check UPS connection)

                          • systemctl restart nut-server
                            (NUT-server reboot) | (Use this to apply configuration changes and to connect the UPS after System startup)

                          • systemctl restart nut-monitor (NUT-monitor restart)

                          • nut-scacnner -U (shows connected devices)

                          Configure the files stored in /etc/ups like this:
                          upsmon.conf

                          RUN_AS_USER root
                          MONITOR apcups@localhost 1 admin [PASSWORD] master
                          SHUTDOWNCMD β€ž/etc/ups/xen-shutdown.shβ€œ
                          

                          nut.conf

                          MODE=netserver
                          

                          ups.conf

                          pollinterval = 1
                          maxretry = 3
                          [apcups]
                          driver = usbhid-ups 
                          #->when you use USB devices
                          port = auto
                          desc = β€ž[NAME]β€œ
                          vendorid = [VENDORID]
                          productid = [PRODUCTID]
                          serial = β€ž[SERIALNO]β€œ
                          

                          upsd.conf

                          LISTEN 0.0.0.0 3493
                          

                          upsd.users

                          [admin]
                          password = [PASSWORD]
                          actions = set
                          actions= fsd
                          instcmds = ALL
                          

                          Finish config Changes

                          systemctl restart nut-server
                          systemctl restart nut-monitor
                          

                          Finished
                          Good to know: You can now also connect to your NUT server via the pfSense plugin by using the credentials in uspd.users to have a gui for your UPS. To do this you have to open up Port 3493 on your host.
                          Run this command to open the Port:

                          iptables -I INPUT -p tcp -m tcp --dport 3493 -j ACCEPT
                          

                          Then save your changes to a config file and adjust Year, Month and Day in the config name. In my case it's the 22th of december in 2022:

                          iptables-save > /etc/sysconfig/iptables
                          mkdir /etc/iptables/
                          iptables-save > /etc/iptables/rules_2022-12-22_INPUT_ACCEPT_3493_NUT.v4 --> Optional, extra saving for Backups.
                          

                          I recommend using this convention of naming your rule files. Its a lot easier to track your changes and to restore files.

                          lkniteL 1 Reply Last reply Reply Quote 3
                          • lkniteL Offline
                            lknite @Hannes_5253
                            last edited by lknite

                            @Hannes_5253 awesome! Thank you for this posting, it's helping me to get a nut client installed. Note for folks coming across this later I needed to use the following to install the nut-client:

                            yum -y install nut-client --enablerepo=epel
                            

                            It would be cool if xcp-ng came pre-installed with nut-client and a default configuration that shuts down the xen server & turns off the ups after it finishes shutting down... similar to how truenas has a service which can be enabled to do so. You'd have to configure it of course, with the nut server to monitor, credentials, and some other options such as whether to actually powerdown the ups after shutting down or not, but maybe through a nice gui after turning it on. Seems like everyone should be using this setup ... just a matter of time till they are ready to.

                            Another reason this would be nice to be integrated, is that docs recommend setting up a user account for nut, one for the monitor so clients can access and another for the shutdown so one of the watchers doesn't change the shutdown script & nut provides a file that indicates when the ups should shutdown, so having that be the same on all xcp-ng systems rather than custom solution would be good.

                            H 1 Reply Last reply Reply Quote 3
                            • H Offline
                              Hannes_5253 @lknite
                              last edited by Hannes_5253

                              @lknite Glad I could help and I agree. πŸ‘πŸ»

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates πŸͺ Co-Founder CEO
                                last edited by olivierlambert

                                Hi,

                                Sadly, this use case is more meant for "non server-grade", where your machine isn't hosted in a datacenter (which is the default XCP-ng target/market).

                                However, I'm not against helping people doing the configuration or having an official guide to document it (I mean, in our doc)

                                Keep in mind we are always trying to find a balance between maintaining something (meaning time+money over other features) vs where the money is coming from (companies paying for support). If we pre-install all of this, it means we'll need to maintain it, while we still have a huge backlog of critical feature for our main (paying) target πŸ™‚ I hope you understand. Also, if we succeed and continue to grow that way with a far larger team than today, we could probably use that extra money from companies to improve integrations for the rest. But before doing it, we need to focus.

                                2b2bff2 1 Reply Last reply Reply Quote 2
                                • 2b2bff2 Offline
                                  2b2bff @olivierlambert
                                  last edited by

                                  @olivierlambert I wonder how datacenters are doing this then in case of power loss? Or isn't that something of their concern?

                                  1 Reply Last reply Reply Quote 1
                                  • olivierlambertO Offline
                                    olivierlambert Vates πŸͺ Co-Founder CEO
                                    last edited by olivierlambert

                                    In a DC, you have usually 2x power feed per rack (one on the left, one on the right). So even if one power line/feed is down, it's not a problem, since all your machines are usually dual PSU.

                                    Also, everything is backed by UPS (both feeds) inside the building, and then diesel generators are taking the load after 10/15 minutes (diesel takes few minutes to start).

                                    So in a colo, you will never have to get your own UPS.

                                    2b2bff2 1 Reply Last reply Reply Quote 1
                                    • 2b2bff2 Offline
                                      2b2bff @olivierlambert
                                      last edited by

                                      @olivierlambert in simple words: they just don't shut down. And if they would have to there are bigger problems to solve.
                                      Thanks for answering...

                                      gskgerG 1 Reply Last reply Reply Quote 1
                                      • gskgerG Offline
                                        gskger Top contributor @2b2bff
                                        last edited by gskger

                                        @2b2bff I think that there are still some business scenarios, where this functionality would come in handy.

                                        For example, we are running server rooms at our main production sites worldwide (no data centers, no colo, all on-site). Each features a three VMware hosts cluster with shared storage, backup, switches, and firewalls in an air-conditioned server room with adequate fire prevention/protection. All have two power feeds (A and B or left and right as Olivier called them) and all systems have dual PSUs, but the sites usually have only one power supplier and no generators.

                                        Feed A goes to an UPS and feed B directly to the power line. If the power goes down, the UPS automatically triggers the vCenter, storage and backup to shut down when the UPS remaining runtime is below a certain threshold.

                                        But I can perfectly understand Oliviers reasoning and there are workarounds. It all depends on your risk appetite (if that is a word in English).

                                        2b2bff2 1 Reply Last reply Reply Quote 0
                                        • 2b2bff2 Offline
                                          2b2bff @gskger
                                          last edited by

                                          @gskger yeah same on my place. I play around with xcp-ng at home, where this is not important, and there is no UPS. So it is not important here.
                                          But I have built two hosts at work in a server room as well. Together with TrueNAS and a couple of other things. The APC UPS has a management card, so it can be reached via network, but somehow the hosts have to shut down gracefully if needed. That's how I stumpled upon this thread in the first place.

                                          1 Reply Last reply Reply Quote 1
                                          • olivierlambertO Offline
                                            olivierlambert Vates πŸͺ Co-Founder CEO
                                            last edited by

                                            I'm not saying it's not important in general (it depends on the use cases) πŸ™‚ But in our priorities (the "regular" DC world), it is not. For the "Edge computing" case, it might be more important, but the word "focus" doesn't have any meaning if we do everything at the same time.

                                            And as usual, contributions are always welcome!

                                            1 Reply Last reply Reply Quote 2
                                            • First post
                                              Last post