Long delays at 46% when creating or starting a new VM

[CodeMercenary]

I've been trying to create a new Ubuntu 24.04 VM today and it's behaving very weird. All my hosts have local storage. I've tried creating VMs on two of my hosts, one with a 500GB drive (thin) on SSD and one with a 50GB drive (thin) on HDD. I attached the ubuntu boot disk to the VM.

When I tell XO (from source) to create the VM, the creation process hangs at 46% for 15 or 20 minutes. It normally completes within a matter of seconds.

They were both set to autostart but neither started. Once the creation task finally finished, I tried to start them. Again, they hung at 46% for 15 or 20 minutes and then didn't start.

I realized my XO was last updated on Monday and there were more updates so I updated it and it did not change the behavior.

The only log entry that seems like it might be related is:

vm.start
{
  "id": "4921a56a-c099-92d9-4ee4-939d8bc1adcf",
  "bypassMacAddressesCheck": false,
  "force": false
}
{
  "name": "HeadersTimeoutError",
  "code": "UND_ERR_HEADERS_TIMEOUT",
  "message": "Headers Timeout Error",
  "call": {
    "duration": 300806,
    "method": "VM.start",
    "params": [
      "* session id *",
      "OpaqueRef:b66b9e3e-ac91-47d9-81ce-383de2ed326f",
      false,
      false
    ]
  },
  "stack": "HeadersTimeoutError: Headers Timeout Error
    at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502270017/node_modules/undici/lib/dispatcher/client-h1.js:642:28)
    at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502270017/node_modules/undici/lib/util/timers.js:162:13)
    at listOnTimeout (node:internal/timers:581:17)
    at processTimers (node:internal/timers:519:7)"
}

There was also this but I didn't cancel anything around when this happened:

vm.start
{
  "id": "4921a56a-c099-92d9-4ee4-939d8bc1adcf",
  "bypassMacAddressesCheck": false,
  "force": false,
  "host": "f23f627d-007e-499e-97cf-678574566429"
}
{
  "message": "task canceled"
}

How do I go about investigating this?

Update: I figured it out. A few weeks ago I changed the IP addresses in XO for backups and SMB ISO storage so they would use the 10Gbe network adapter in that server. I've had no trouble with backups since then, however, it appears the SMB ISO storage is no longer accessible from XO/XCP. I added a new NFS ISO storage to the same location and it works. Now the VMs will boot.

I think I'll leave this here in case someone else runs into the same issue. If you have an ISO image attached to the VM and the server can't get to it, it will get stuck during boot and will kick out that timeout error without indicating that it timed out on connecting to the ISO storage.