XenServer VM Fails to Start - HeadersTimeoutError & Hotplug.Device_timeout(_) After Power Outage

spcmediaco

Please be aware (I know it's obvious) but I was using chatgtp to help me sort this, I figured it would be the best way to summarize my issues:

Version 8.2.1 ()
Build number release/yangtze/master/58

Issue Overview

I am experiencing issues with starting any VMs on my XenServer host (EXAMPLESERVER) after a power outage. No VMs on this host are working.

I am using Xen Orchestra (XO Community Edition), which is running on a separate machine. The EXAMPLESERVER host is visible in Xen Orchestra, but all VM start operations fail with the following errors:

HeadersTimeoutError in Xen Orchestra
Hotplug.Device_timeout(_) when starting via xe vm-start
VMs have blocked operations, preventing shutdown or reboot
Possible storage issues related to the VM’s virtual disks (VDIs)

Error Messages & Logs

1. VM Start Error in Xen Orchestra (XO)

vm.start  
{  
  "id": "EXAMPLE-VM-ID",  
  "bypassMacAddressesCheck": false,  
  "force": false  
}  
{  
  "name": "HeadersTimeoutError",  
  "code": "UND_ERR_HEADERS_TIMEOUT",  
  "message": "Headers Timeout Error",  
  "call": {  
    "duration": 300642,  
    "method": "VM.start",  
    "params": [  
      "* session id *",  
      "OpaqueRef:EXAMPLE-OPAQUE-REF",  
      false,  
      false  
    ]  
  },  
  "stack": "HeadersTimeoutError: Headers Timeout Error  
    at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/.../dispatcher/client-h1.js:642:28)  
    at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/.../util/timers.js:162:13)  
    at listOnTimeout (node:internal/timers:569:17)  
    at processTimers (node:internal/timers:512:7)"  
}

2. CLI Error - Hotplug Device Timeout

[22:22 EXAMPLESERVER ~]# xe vm-start uuid=EXAMPLE-VM-UUID  
The server failed to handle your request, due to an internal error.  
message: xenopsd internal error: Hotplug.Device_timeout(_)

3. VMs are Blocked from Shutdown or Restart

[22:59 EXAMPLESERVER ~]# xe vm-shutdown uuid=EXAMPLE-VM-UUID force=true  
You attempted an operation that was explicitly blocked (see the blocked_operations field of the given object).  
ref: EXAMPLE-VM-UUID  
code: true

4. VM Parameters Show It is in a Blocked State

xe vm-param-list uuid=EXAMPLE-VM-UUID params=blocked-operations  
blocked-operations (MRW): clean_reboot: true; hard_reboot: true; pause: true;  
shutdown: true; clean_shutdown: true; hard_shutdown: true; destroy: true

5. Storage Issues Detected - LVM Shows Active VHDs but Possible Issues

pvscan  
  PV /dev/sdc    VG VG_XenStorage-EXAMPLE   lvm2 [<1.82 TiB / <161.53 GiB free]  
  PV /dev/sdb3   VG VG_XenStorage-EXAMPLE   lvm2 [182.06 GiB / 141.96 GiB free]  
  PV /dev/sda    VG VG_XenStorage-EXAMPLE   lvm2 [1.09 TiB / <575.00 GiB free]

Troubleshooting Steps Tried

1. Restarted XenAPI and XenOpsD

systemctl restart xapi  
systemctl restart xenopsd-xc

Result: No improvement, VMs still do not start.

2. Attempted to Unpause the VM

xe vm-unpause uuid=EXAMPLE-VM-UUID

Result: xe command hangs indefinitely.

3. Checked for Blocked Operations

xe vm-param-list uuid=EXAMPLE-VM-UUID params=blocked-operations

Result: Shows that shutdown, reboot, and destroy operations are all blocked.

4. Reset VM Power State

xe vm-reset-powerstate uuid=EXAMPLE-VM-UUID force=true

Result: No change, VM remains in a paused state.

5. Checked VDI Attachments

xe vbd-list vm-uuid=EXAMPLE-VM-UUID

Result: Disks appear attached but VMs still do not start.

6. Restarted Xen Orchestra

systemctl restart xo-server  
systemctl restart xo-web

Result: No change.

7. Scanned Storage Repositories

xe sr-list  
xe sr-scan uuid=EXAMPLE-SR-UUID

Result: Storage seems intact, but VMs still do not boot.

Additional Notes

The host lost power, and since rebooting, no VMs on this host will start.
Xen Orchestra is running on a separate machine and can see the host.
Other hosts in the pool (if any) are functioning properly.
The error Hotplug.Device_timeout(_) suggests a VDI issue, but VDIs appear attached.
The HeadersTimeoutError in XO indicates XAPI is slow or unresponsive.
The VM is blocked from shutdown/reboot and cannot be started via CLI or XO.

Request for Help

What could be preventing VMs from starting?
How can I clear the blocked operations?
Could this be a corrupted VM state, and if so, how can I recover it?
Is there a way to manually attach the VDIs and force a boot?
Any ideas on why Xen Orchestra times out when starting the VM?

Would really appreciate any guidance on how to force the VMs to start or recover from this state.

Thanks in advance!

I'm no expert but I haven't had any issues in a year of using xcp-ng.

SPC

olivierlambert

You need to remove the blocked operations.

DustinB

@spcmediaco Great first post.

@olivierlambert Great support!

spcmediaco

@olivierlambert I appreciate the response, and I'm trying to do this (with the help of LLMs) however, could you point me in the direction on how to do this as I'm still having difficulties with this issue.

Danp

@spcmediaco Check the VM's Advanced tab in XO and make sure these options are not enabled --

spcmediaco

@Danp said in XenServer VM Fails to Start - HeadersTimeoutError & Hotplug.Device_timeout(_) After Power Outage:

Check the VM's Advanced tab in XO and make sure these options are not enabled --

Thank you, I did check and they are unchecked. Still stuck, it seems like XO running on my other host is still trying to start it over and over again with bad power state or something:

spcmediaco

@spcmediaco

I restored a backup VM to the host that is currently working and that VM worked, now I'm trying to migrate a VM from the 'broken' host to the working host and see if that works. Maybe I need to format the 'broken' host and reinstall it.

spcmediaco

@spcmediaco

FYI, I never figured out how to fix. I am doing backup recovery now.