seanmcg182

For anyone who finds this in the future, I ran into a similar issue today when passing through a LSI Card,
Coincidentally on Address 05:00.0, also coincidentally on a Supermicro Board (mine is X10DRH-CT, OPs was a X8DA8)
I was originally writing a comment here to see if a solution had ever been found, but while taking/marking up screenshots of my Motherboards System Block Diagram, I discovered my problem...

Human Error of course.

Long story short, I added a PCI to 2xM.2 card, and in my BIOS, I bifurcated the wrong CPU/PCI Port.

Instead of Bifurcating the M.2 Card, I accidentally Bifurcated the LSI Card, which caused the Unknown Header Error AND since there was an unknown header, the VM Would throw the "Device or Resource is busy" error when trying to start.

Correcting the Bifurcation fixed the issue in my case.

Not sure if that was the issue in OPs case, but this might help someone else who stumbles across this

After a good sleep, I resolved half of my issue. I got my LSI Cards messed up, I accidentally hidan internal LSI3108 (Address 01:00.0) instead of the PCI Card LSI3008 (Address 05:00.0)...
My Boot Drives run off of the internal card.

From the fallback kernel, I was able to Modify /etc/grub-efi.cfg to remove the internal card form the hidden list.

I now have a separate issue, but will make another post.

Before today, I had XCP-NG 8.3 installed on the Metal of my server, and a LSI 3008 SAS Card going to a backplane, which was hidden in dom0 (Location 03:00.0, this is important), and passed through to a TrueNAS VM. No Issues, operated for years.

Today, I installed a PCI to M.2 adapter, and to do so, I moved the LSI card to another slot, and put the M.2 adapter in the slot the LSI Card was in. Long story short, the new arrangement just made more sense to handle the bifurcation stuff needed.

I booted XCP-NG, and TrueNAS reported that my Dataset was missing, but it saw the NVMe drive.
Made sense once i checked a 'xe pci-list' , as the M.2 Adapter was now 03:00.0... Why it changed? don't know, I can't imagine that's an important detail.

I saw the LSI 3008 card was on 01:00.0
I pulled the usual xe pci-disable-dom0-access uuid=<pci uuid>
Reboot the host.
And it throws me into the Emergency Shell...

After Google searching I find the google page about "Initrd is missing after an update"
The weird part here is that the Emergency Shell is NOT reporting "/boot/initrd-4.19.0+1.img is missing"
The Error I get is "Warning: /dev/disk/by-label/root-lfgrma does not exist"

For thoroughness, I did load into the fallback kernel, and it loaded fine...
Not sure where to go from here though.
For testing purposes, I did attempt to Rebuild the initrd with 'dracut -f /boot/initrd-4.19.0+1.img 4.19.0+1' even though the error wasnt saying the img was the issue... but wanted to make sure
A restart throws it back into the emergency shell.

Any advice would be appreciated
Any questions, I will gladly answer... I'm stumped

@nuentes For reference, the “PCI Passthrough” that was referenced involves buying a USB PCI card, and passing the card through.

If you need any help getting the script up, just respond here and I’ll do what I can to help. My comments in the script arent immaculate

@ph7 unsure about @nuentes ’s hardware setup, but at least in my case I can’t pass through USB PCIe cards for some reason.

I can’t remember the exact reason for this, as I discovered this years ago. Mightve been something about how older motherboard’s handle Serial HBAs? Something dealing with IOMMU and/or RMRR?

Also in my case, itd be impractical to get 6 USB cards and pass then thru to 6 different VMs

@nuentes I made a script about a year ago that I modified yesterday and today in response to this issue we're having.
The updated script will reattach and restart VMs after a disconnection is detected.
I have mine set on a cronjob every minute

You will need to modify it a bit to match your VM name, and your single VM vs my 6 VMs, but hopefully this helps to some degree.

Obviously, its not necessarily a "fix" to prevent the disconnections from happening in the first place, but it does at least minimize the downtime.

https://xcp-ng.org/forum/topic/9050/script-to-auto-mount-usbs-on-boot-reboot-monitoring-multiple-ups

good luck

Just an update if anyone ever comes back to this topic:

I held off on updating this script, as my disconnects had been very minimal over the last year...

However, due to my recent issues in 8.3 (along with someone else's https://xcp-ng.org/forum/topic/10590/vusb-keeps-disappearing ) I decided to put some heavier investment into this script.
Long story short, it will attach USB Devices to VMs and start them automatically, Shutdown VMs and reattach VUSBs if they become disconnected and restart the VMs afterwards.

To Recap my setup, I have 6 CyberPower UPS's, and 6 VMs named "UPS1", "UPS2", etc to "UPS6"

I have updated this script to:
(Before this new logic, it does the previously written gathering of PUSB UUIDs, USB Group UUIDs, VM UUIDs, VM Power States, in addition to enabling passthrough for any PUSBs that need it)
First, logic checks existing VUSBs to see if they are in a "Disconnected" state from a running VM they are attached to... If so, it will shut off the VM
Second, if a VUSB is not detected as attached for a running Running UPS VM, it will shut off the VM
If a VM shutdown command was executed, the program will wait 5 seconds (MY VMs are minimal, you may need to adjust this timer if your VMs take longer to power off)
It will then rescan the VM Power States
After this is done, it runs my previous logic of Attaching the appropriate USB Group to a Virtual Machine if it's needed, and starts any non-running VM.

I have this setup on a cron timer every minute.

if run manually, you may see the following errors:
1)

The server failed to handle your request, due to an internal error. The given message may give details useful for debugging the problem.
message: xenopsd internal error: Device_common.QMP_Error(51, "{\"error\":{\"class\":\"DeviceNotFound\",\"desc\":\"Device 'vusb2-1.3' not found\",\"data\":{}},\"id\":\"qmp-000368-51\"}")

This error appears the first time a shutdown command to a VM is executed after the VUSB enters a "Disconnected". Happens when attempting to shutdown VM in XOA as well... Due to this, the code just executes the command a second time.
2)

USB_groups are currently restricted to contain no more than one VUSB.

This appears in the scenario where the VUSB is still present in the VM, just in a "disconnected" state, as it can't be reattached.
I'm sure I could modify the code to remove this dialogue, but it doesn't affect functionality.

Hopefully this helps someone.

#####set working directory
cd /home/UPSAutoConnect/TempFiles
#####remove old text files
rm -f *.txt

#####create list of all pUSBs
xe pusb-list > pusb.txt
#####split list per device and remove old list
csplit -sz pusb.txt /uuid/ '{*}'
rm -f pusb.txt
#####for all split files, if Vendor ID is CyberPower, create a txt file named by Physical USB Path, containing the USB uuid, and enable for passthrough
for i in xx*
do
        var1=$(awk '/vendor-id/ {print $4}' "$i")
        if [ "$var1" -eq 0764 ]
        then path=$(awk '/path/ {print $4}' "$i")
        awk '/uuid/ {print $5}' "$i" > "$path".txt
        xe pusb-param-set uuid=$(cat "$path".txt) passthrough-enabled=true
        xe usb-group-list PUSB-uuids=$(cat "$path".txt) > "$i"-2
        awk '/uuid/ {print $5}' "$i"-2 > "$path"group.txt
        fi
done
#####remove all csplit files
rm -f xx*



#####create list of all VMs
xe vm-list > vm.txt
#####split list per vm and remove old list
csplit -sz vm.txt /uuid/ '{*}'
rm -f vm.txt
#####for all split files, get UUID of needed VMs
for f in xx*
do
        vmname=$(awk '/name/ {print $4}' "$f")
        if [ "$vmname" == 'UPS1' ] || [ "$vmname" == 'UPS2' ] || [ "$vmname" == 'UPS3' ] || [ "$vmname" == 'UPS4' ] || [ "$vmname" == 'UPS5' ] || [ "$vmname" == 'UPS6' ]
        then  awk '/uuid/ {print $5}' "$f" > "$vmname".txt
        awk '/power/ {print $4}' "$f" > "$vmname"state.txt
        fi
done
#####remove all csplit files
rm -f xx*



#####create list of all vUSBs
xe vusb-list > vusb.txt
#####split list per device and remove old list
awk '/uuid/ {print $5}' "vusb.txt" > vusb2.txt
awk 'NF > 0' "vusb2.txt" > vusb3.txt
split -dl 1  vusb3.txt
rm -f vusb.txt
rm -f vusb2.txt
rm -f vusb3.txt
#####for all split files, if VM Running and device disconnected, halt VM
for g in x*
do
        vmname2=$(xe vusb-param-get uuid=$(cat "$g") param-name=vm-name-label)
        xe vusb-param-get uuid=$(cat "$g") param-name=vm-name-label > "$vmname2"vusb.txt
        vmstate=$(cat "$vmname2"state.txt)
        vusbattached=$(xe vusb-param-get uuid=$(cat "$g") param-name=currently-attached)
        if ([ "$vmname2" == 'UPS1' ] || [ "$vmname2" == 'UPS2' ] || [ "$vmname2" == 'UPS3' ] || [ "$vmname2" == 'UPS4' ] || [ "$vmname2" == 'UPS5' ] || [ "$vmname2" == 'UPS6' ]) && [ "$vusbattached" == 'false' ] && [ "$vmstate" == 'running' ]
        then xe vm-shutdown uuid=$(cat "$vmname2".txt) force=true
        then xe vm-shutdown uuid=$(cat "$vmname2".txt) force=true
        waitforshutdown=1
        fi
done
#####remove all split files
rm -f x*



#####If no VUSB detected, halt VM
if [ ! -f UPS1vusb.txt ] && [ $(cat UPS1state.txt) == 'running' ]
then xe vm-shutdown uuid=$(cat UPS1.txt) force=true
waitforshutdown=1
fi
if [ ! -f UPS2vusb.txt ] && [ $(cat UPS2state.txt) == 'running' ]
then xe vm-shutdown uuid=$(cat UPS2.txt) force=true
waitforshutdown=1
fi
if [ ! -f UPS3vusb.txt ] && [ $(cat UPS3state.txt) == 'running' ]
then xe vm-shutdown uuid=$(cat UPS3.txt) force=true
waitforshutdown=1
fi
if [ ! -f UPS4vusb.txt ] && [ $(cat UPS4state.txt) == 'running' ]
then xe vm-shutdown uuid=$(cat UPS4.txt) force=true
waitforshutdown=1
fi
if [ ! -f UPS5vusb.txt ] && [ $(cat UPS5state.txt) == 'running' ]
then xe vm-shutdown uuid=$(cat UPS5.txt) force=true
waitforshutdown=1
fi
if [ ! -f UPS6vusb.txt ] && [ $(cat UPS6state.txt) == 'running' ]
then xe vm-shutdown uuid=$(cat UPS6.txt) force=true
waitforshutdown=1
fi



#####if any VM was shutdown by code, wait for process to finish
if [ "$waitforshutdown" == 1 ]
then sleep 5s
fi



#####rerun previous acquisition to update VM Power States
#####create list of all VMs
xe vm-list > vm.txt
#####split list per vm and remove old list
csplit -sz vm.txt /uuid/ '{*}'
rm -f vm.txt
#####for all split files, get UUID of needed VMs
for h in xx*
do
        vmname=$(awk '/name/ {print $4}' "$h")
        if [ "$vmname" == 'UPS1' ] || [ "$vmname" == 'UPS2' ] || [ "$vmname" == 'UPS3' ] || [ "$vmname" == 'UPS4' ] || [ "$vmname" == 'UPS5' ] || [ "$vmname" == 'UPS6' ]
        then  awk '/uuid/ {print $5}' "$h" > "$vmname".txt
        awk '/power/ {print $4}' "$h" > "$vmname"state.txt
        fi
done
#####remove all csplit files
rm -f xx*


#####if expected UPS is present, attach to expected VM
if test -f *-1.1group.txt
then
        if [ $(cat UPS1state.txt) == 'halted' ]
        then
        xe vusb-create usb-group-uuid=$(cat *-1.1group.txt) vm-uuid=$(cat UPS1.txt)
        xe vm-start uuid=$(cat UPS1.txt)
        fi
fi

if test -f *-1.2group.txt
then
        if [ $(cat UPS2state.txt) == 'halted' ]
        then
        xe vusb-create usb-group-uuid=$(cat *-1.2group.txt) vm-uuid=$(cat UPS2.txt)
        xe vm-start uuid=$(cat UPS2.txt)
        fi
fi

if test -f *-1.3group.txt
then
        if [ $(cat UPS3state.txt) == 'halted' ]
        then
        xe vusb-create usb-group-uuid=$(cat *-1.3group.txt) vm-uuid=$(cat UPS3.txt)
        xe vm-start uuid=$(cat UPS3.txt)
        fi
fi

if test -f *-1.4group.txt
then
        if [ $(cat UPS4state.txt) == 'halted' ]
        then
        xe vusb-create usb-group-uuid=$(cat *-1.4group.txt) vm-uuid=$(cat UPS4.txt)
        xe vm-start uuid=$(cat UPS4.txt)
        fi
fi

if test -f *-2.1group.txt
then
        if [ $(cat UPS5state.txt) == 'halted' ]
        then
        xe vusb-create usb-group-uuid=$(cat *-2.1group.txt) vm-uuid=$(cat UPS5.txt)
        xe vm-start uuid=$(cat UPS5.txt)
        fi
fi

if test -f *-2.2group.txt
then
        if [ $(cat UPS6state.txt) == 'halted' ]
        then
        xe vusb-create usb-group-uuid=$(cat *-2.2group.txt) vm-uuid=$(cat UPS6.txt)
        xe vm-start uuid=$(cat UPS6.txt)
        fi
fi

#remove leftover files
rm -f *.txt
waitforshutdown=0

@nuentes Same issue here. I have 6 Cyberpower UPSs all attached to different VMs for monitoring. Theyre all disconnecting randomly. This did not happen before I updated XCP in the last couple months.

And I know all 6 of them are not being physucally unplugged haha.

While I have no solutions for you, I hope this gains some traction as its severely messing with my Energy monitoring.

If you still want a way to know when it “disconnects”, I would recommend installing Powerpanel on the VM you’re using with the UPS, as it will log when communications are lost… However knowing when it happened won’t help you solve it unfortunately…

BUT if you have a mail server, you could set it up to send you an email when comms are lost

@olivierlambert The never ending struggle of development. always more questions.

while I did notice that the overall path changed, I did notice that some parts were consistent still...
IE, My Paths changed from 2-2.1, 2-2.2, and 2-2.4 to 1-2.1, 1-2.2, and 1-2.4 respectively.

Ideally I would be using the Serial Number readout to sort these more efficiently, but for some reason my rack mounted CyberPower UPSs don't have Serial Numbers reporting even in the software when direct connected to my desktop... hence why I'm trying to find another way to "statically" identify my devices.

I'm going to keep working on this program, and will update at a later point when I make some more significant progress, maybe with some "Options" for people to swap in.

Next objective is for it to detect whether a running VM has its proper USB attached still, and if it became detached, to stop the VM, mount the USB, and then start it... main reason for this is in case a UPS gets unplugged, or is removed for maintenance.

@olivierlambert Y'know, I thought I smelled a bit of sarcasm there, so I went and double checked... and i think now I'm more confused lmao

So I rebooted, UUIDs and Paths didn't change.
Then I unplugged, and moved one of the USBs I'm currently using.... and moving that one USB changed the Path and UUID of all 3 USBs that were attached.

Guess the Paths weren't as static as I thought