VM failed and won't start. Displaying "An emulator required to run this VM failed to start"
-
[12:52 iahost-xcpng-server3 ~]# xe vm-reset-powerstate vm=dfcdcce6-b6e1-d865-6f2b-472a35b1fae7 --force
This operation cannot be completed because the server is still live.
host: 6985c3a2-091d-488b-8de6-4bf40985987b (iahost-xcpng-server3) -
On the pool master:
- Restart the toolstack
- Immediately after, run the vm reset powerstate command.
-
@olivierlambert Ok. I was able to get that VM started on another Host. Now, I need to figure out what is going on with server1. I appreciate all your help! If you have any suggestions please let me know.
-
My suggestion stands from previous messages, check the
dmesgoutput from the problematic host
-
@olivierlambert It is full of these:
43342477.643853] sd 7:0:0:1: rejecting I/O to offline device
[43342477.643861] print_req_error: I/O error, dev sdb, sector 0 -
That's exactly what I suspected. You have a connectivity issue with your shared storage.
The device is considered offline because it cannot do any I/O on sector 0
-
said in VM failed and won't start. Displaying "An emulator required to run this VM failed to start":
rejecting I/O to offline device
I was able to reboot the host. Now things seam fine. Does that make sense? I mean have you seen this happen then a reboot of the host takes care of it?
-
@olivierlambert I had the same issue this morning on one of the hosts I rebooted and was working fine. An emulator required to run this VM Failed to start. sd 7:0:0:1: rejecting I/O to offline device. I am not sure what to check next.
-
xcp-ng iscsiadm -m session -P 3 Attached scsi disk sdb State: offline
-
There's something fishy about the connectivity to your storage.
-
@olivierlambert I am looking into that. I just do not have much experience with the networking.
-
I found that the SR has two NICs bonded to two different hardware switches on the iSCSI LUN. Then each host has a management network(Bonded NICS on hosts) and VM network (Bonded NICS). Should I break the SR bond and just go with one NIC? If so, I assume I would be best to shutdown the VMs until I have one NIC on SR reconfigured?
-
Additional finding. The two switches (LAG) are Cisco with STP enabled.
-
I am still getting " sd 7:0:0:1: rejecting I/O to offline device" taking all VMs on host down. I then shutdown the VM and power them on, on another Host. I have 4 host in a pool connected iSCSI to a Synology LUN. What is the best way to troubleshoot network issues on xcp-ng.
Thank you -
Just adding another note. When I reboot the host everything is fine then the issues pops back up. There doesn't seem to be any consistency as far as timeframe in between. It just happens randomly
-
Any help is greatly appreciated.
-
That's not a trivial problem that you have, it seems to be intermittent and could be anything between host and the storage. I assume you can see the issue as soon you have the
rejecting I/O to offline deviceright? -
Yes
-
At least we know there's a root cause: the device becomes offline. Could it be a power setting or something on the device itself?
-
Well, it doesn't happen to every host at the same time, it's just random.
