XCP-ng Shutdown Hangs When an SR Is Unavailable

Divo

Hello everyone,

I’m using NUT on a Raspberry Pi to automatically shut down my infrastructure in the event of a power outage. The Raspberry Pi connects to my XCP-ng hosts via SSH and shuts it down via a SSH Command Restriction.

While testing this setup, I noticed that XCP-ng can hang during shutdown if an SR is no longer available.

For example, let’s say I have a VM running a Samba server that hosts ISO files. The XCP-ng host has an SR/repository configured to access those ISOs via the SMB share.

When a shutdown is triggered, the VM hosting the Samba share is shut down first. As a result, the SMB share is no longer available. However, when the XCP-ng host itself tries to shut down, it appears to wait for the SMB repository instead of continuing with the shutdown.

At the moment, I work around this by manually unplugging the SMB repositories when not in use.

Is there a way to prevent XCP-ng from hanging when an SR becomes unavailable during shutdown or is there an alternative shutdown command or option that forces the host to shut down without waiting for the unavailable SR?

Any help or suggestions would be greatly appreciated!

Thank you in advance!

Divo

Hello everyone,

I solved the issue by doing the following, and it seems to work perfectly:

I created a custom shutdown script located at /usr/local/bin/nut-emergency-cleanup.sh:

#!/bin/bash
# 1. Force eject all mounted ISOs/CDs across VMs
xe vm-cd-eject --multiple 2>/dev/null

# 2. Find all PBDs connected to NFS, ISO, SMB, or iSCSI storage types and unplug them
NETWORK_PBDS=$(xe pbd-list currently-attached=true | grep -B 1 -E "storage-type: (nfs|iso|smb|iscsi)" | grep "uuid ( RO)" | awk '{print $5}')

for pbd_uuid in $NETWORK_PBDS; do
        echo "Force unplugging PBD: $pbd_uuid"
        xe pbd-unplug uuid="$pbd_uuid" --force 2>/dev/null
done

# 3. Finally, trigger the immediate hardware shutdown
/sbin/shutdown -h now

Instead of trigging a direct shutdown, this script now gets executed via a command forced in the authorized_keys file:

command="sudo /usr/local/bin/nut-emergency-cleanup.sh",no-port-forwarding,no-x11-forwarding,no-agent-forwarding [PUBLIC_KEY]

Thank you for your help!

acebmxer

@Divo

You would want to adjust your script to disable the smb share or put it in maintenance mode prior to shutting down.

I ran into an issues for when I was shutting down a Remote just to save power - https://xcp-ng.org/forum/topic/12339/xo-unresponsive-when-remote-storage-is-offline.?_=1791151494204

It still didnt help with the unresponsivenes. This was with a Backup Remote not a SR but enabled maintenance mode before shutdown and disable when back online.

poddingue

This came up in https://xcp-ng.org/forum/topic/11728 last winter, with the same kind of setup, an ISO SR served by a VM on the host.

Olivier's answer there was to eject the CDs from the VMs, unplug the ISO SR's PBD, and only then shut the host down, using xe vm-cd-eject --multiple and xe pbd-unplug uuid=<PBD UUID> for the first two steps (the PBD one is in the CLI reference: https://docs.xcp-ng.org/appendix/cli_reference#pbd-unplug).

@mickwilli then added the unplug to his UPS shutdown script, and his host shut down cleanly after that. I haven't tried it with an SMB share myself, so I can't promise it behaves like his NFS one, but your NUT script on the Pi looks like the natural place for those two steps, just before the shutdown command.

Divo

Thank you both for your input! I'll implement your suggestions during the next maintenance window and keep you both posted.

Forza

If a mount uses the hard mount option, it will lock any process touching it, including unmount during shutdown, if the mount source is unavailable. It is a kernel thing and I believe there is no solution for it at present.

Divo

Hello everyone,

I solved the issue by doing the following, and it seems to work perfectly:

I created a custom shutdown script located at /usr/local/bin/nut-emergency-cleanup.sh:

#!/bin/bash
# 1. Force eject all mounted ISOs/CDs across VMs
xe vm-cd-eject --multiple 2>/dev/null

# 2. Find all PBDs connected to NFS, ISO, SMB, or iSCSI storage types and unplug them
NETWORK_PBDS=$(xe pbd-list currently-attached=true | grep -B 1 -E "storage-type: (nfs|iso|smb|iscsi)" | grep "uuid ( RO)" | awk '{print $5}')

for pbd_uuid in $NETWORK_PBDS; do
        echo "Force unplugging PBD: $pbd_uuid"
        xe pbd-unplug uuid="$pbd_uuid" --force 2>/dev/null
done

# 3. Finally, trigger the immediate hardware shutdown
/sbin/shutdown -h now

Instead of trigging a direct shutdown, this script now gets executed via a command forced in the authorized_keys file:

command="sudo /usr/local/bin/nut-emergency-cleanup.sh",no-port-forwarding,no-x11-forwarding,no-agent-forwarding [PUBLIC_KEY]

Thank you for your help!