Andrew

Andrew

@stormi Microcode updated on affected Gen11 i7. Running normally.

Andrew

@gduperrey I jumped in all the way by mistake... I updated a wrong host, so I just did them all. Older AMD, Intel E3/E5, NUC11, etc. So far, so good. Add/migrate/backup/etc VMs are working as usual. Good for guest tools too, but mine are mostly Debian 7-11. Stuff is as usual so far.

Andrew

I agree and I'll say it again, Thanks! It's not just Linux/Xen stuff. It is XCP-ng and XO that make everything work as a cohesive vertical open-source solution (some nice buzz words). Thanks to the Vates team and community that have built and support it. I look forward to ongoing continuous improvement and innovation!

Andrew

@bleader Installed and running.

Andrew

@bleader Updates running on several old and new intel machines (including microcode update). Working fine so far. Rolling Pool Reboot is a helpful feature.

Andrew

@bleader I installed it on a bunch of busy hosts. All are fine, but none used PCI passthrough. The Rolling Pool Reboot in XO was very helpful.

Andrew

@dinhngtu On a quick test, the ISO boots and installs. When the VM boots from the HD it crashes (and on reboots).

Updating XCP 8.3 with qemu-4.2.1-5.2.12.2~fixvwc1.1.xcpng8.3.x86_64 solves the issue and the installed windows image boots correctly without any additional changes.

Andrew

@abudef @olivierlambert @thomas-dkmt I agree. I read block the same way... how about disable or prevent. (French? empêcher)

Andrew

@julien-f @olivierlambert

Thanks for the ability to delete XO task logs feature! (XO commit f6e6e)

Andrew

@gduperrey Installed and running on Intel systems, and Zen3 system that sees the microcode update.

Andrew

@gleh After the update and a manual primary host reboot I was able to start new VMs and migrate VMs, but still could not run rolling pool update. I tried restarting tool stack, but it did not help.

pool.rollingReboot
{
  "pool": "40a08640-88dc-7ff9-9190-e13b89025b4d",
  "bypassBackupCheck": false,
  "shutdownPinnedVms": false
}
{
  "code": "CANNOT_EVACUATE_HOST",
  "params": [
    "NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8"
  ],
  "call": {
    "duration": 4,
    "method": "host.assert_can_evacuate",
    "params": [
      "* session id *",
      "OpaqueRef:a8fb168d-bd82-5c1a-60fe-e965eefc43ef"
    ]
  },
  "message": "CANNOT_EVACUATE_HOST(NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8)",
  "name": "XapiError",
  "stack": "XapiError: CANNOT_EVACUATE_HOST(NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8)
    at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:65:5)
    at processImmediate (node:internal/timers:502:9)"
}

Andrew

@gleh @flakpyro I have the same problem after this update. Can't migrate a VM. Can't run rolling pool reboot. Did a tool stack restart, did not help. Big issue as the pool master will have to be rebooted with live VMs running. I was able to suspend/unsuspend a VM. I could stop but not start a VM on the pool.

vm.migrate
{
  "targetHost": "388f9aad-2e39-41fa-acac-18a498eeac34",
  "vm": "450015b0-e878-5067-e8b6-8d0513a750f6"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\")"
  ],
  "task": {
    "uuid": "8e1d9888-9f9f-ebab-8a9f-87d2cd651113",
    "name_label": "Async.VM.pool_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260815T18:38:30Z",
    "finished": "20260815T18:38:32Z",
    "status": "failure",
    "resident_on": "OpaqueRef:a8fb168d-bd82-5c1a-60fe-e965eefc43ef",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\")"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2917))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2966))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 414))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2865))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3412))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3422))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3443))((process xenopsd-xc)(filename ocaml/xapi-idl/lib/task_server.ml)(line 192))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3481))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3652))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 262))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 268))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 293))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 434))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3490))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 458))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2519))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "INTERNAL_ERROR(xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\"))
    at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/index.mjs:1329:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/index.mjs:1363:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/index.mjs:1353:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/index.mjs:1560:14)"
}

Andrew

@florent

Running delta backup with CBT to S3.
NBD connections=1
Yes, NBD. No VHD only.
Yes, NBD is enabled and working on all pool hosts.
Yes, XO can access the network used for NBD.
No, a default backup network as not been assigned.
No more logs from journalctl.

My normal large nightly delta backup to S3 ran and did not produce any new errors. So far, just the one error once.

Andrew

@florent @olivierlambert I also have the same issue. I see there have been a lot of XO updates to the backups. Maybe this is fixing a hidden problem?

CR works, but normal delta backups to S3 have the issue on some of the VMs (on a test run). All the VMs on this backup are Linux.

A second run of the same delta backups all worked. I'll have to see what happens for the normal nightly backup with a lot more VMs...

(edit fix logs):

 2026-07-28T23:17:34.544Z xo:xapi:vdi WARN invalid HTTP header in response body {
   body: 'HTTP/1.1 500 Internal Error\r\n' +
     'content-length: 318\r\n' +
     'content-type: text/html\r\n' +
     'connection: close\r\n' +
     'cache-control: no-cache, no-store\r\n' +
     '\r\n' +
     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:4a172cde-c772-2b6c-100d-f63ea5e4d842; CBT metadata ]</body></html>'
 }
 2026-07-28T23:17:34.552Z xo:xapi:xapi-disks WARN can't compute delta OpaqueRef:6ec8ab35-d377-cd33-ce94-72a7a4e9e3b0 from OpaqueRef:4a172cde-c772-2b6c-100d-f63ea5e4d842, fall back to a full {
   error: Error: invalid HTTP header in response body
       at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/vdi.mjs:39:19)
       at process.processTicksAndRejections (node:internal/process/task_queues:104:5)
       at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/vdi.mjs:263:5)
       at async #getExportStream (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:123:20)
       at async XapiVhdStreamSource.init (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:135:23)
       at async #openExportStream (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/Xapi.mjs:186:7)
       at async #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/Xapi.mjs:97:22)
       at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/xapi/disks/Xapi.mjs:268:18)
       at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
       at async file:///opt/xo/xo-builds/xen-orchestra-202607281243/@xen-orchestra/backups/_incrementalVm.mjs:104:5
 }
 2026-07-28T23:17:34.553Z xo:xapi:xapi-disks INFO export through vhd

Andrew

@olivierlambert Working again. Thanks!

Andrew

@olivierlambert I have a full Realtek 8125/8126/8127 update to replace the existing r8125 driver, as required to support newer chips.

The new 8125 driver drops the 8126 support but you just install the 8126 if needed (requires newer 8125 driver to avoid module conflicts).

Andrew

@andriy.sultanov No error... returns nothing.

Andrew

@rzr Updates installed and running. I also run the snapshot-fixer.py and xapi_leaked_vbds.py scripts... The fixer found/fixed lots on many pools...

@andriy.sultanov I have the same error... On one pool, I ran snapshot-fixer.py and it fixed about 50 of them, but xapi_leaked_vbds.py still returns an error (before and after fix):

Traceback (most recent call last):
  File "./xapi_leaked_vbds.py", line 23, in <module>
    snap = xapi.VM.get_parent(snap)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 317, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 198, in xenapi_request
    result = _parse_result(getattr(self, methodname)(*full_params))
  File "/usr/lib/python3.6/site-packages/XenAPI.py", line 292, in _parse_result
    raise Failure(result['ErrorDescription'])
XenAPI.Failure: ['HANDLE_INVALID', 'VM', 'Ref:119']

Andrew

@acebmxer This has been reported before. XO needs to report an actual error for this problem.

Andrew

@rzr Again, installed on several active pools. Running correctly for my use.

Andrew

@Andrew