XCP-ng 8.3 updates announcements and testing

MajorP93

@gleh Wow! This sounds like a big release especially in the QCOW2 / storage department.
A huge thanks to the whole XCP-ng team!
The platform keeps improving and improving which is awesome to see.

I will test this batch of packages next week when my test environment is available again.

Best regards

acebmxer

@gleh

I notice the following error...

(1/4): xcp-ng-candidates/primary_db                                                            | 4.0 kB  00:00:00     
(2/4): xcp-ng-testing/primary_db                                                               | 193 kB  00:00:00     
xcp-ng-updates/primary_db      FAILED                                               ] 1.2 MB/s | 2.1 MB  00:00:03 ETA 
http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/repodata/e3780b9c3ab9712c33f7966c80311e07fa34e345f4242fc43b745e07a2f06826-primary.sqlite.bz2: [Errno -1] Metadata file does not match checksum
Trying other mirror.
(3/4): xcp-ng-base/primary_db                                                                  | 3.9 MB  00:00:02     
xcp-ng-updates/primary_db      FAILED                                               ]  0.0 B/s |    0 B  --:--:-- ETA 
http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/repodata/e3780b9c3ab9712c33f7966c80311e07fa34e345f4242fc43b745e07a2f06826-primary.sqlite.bz2: [Errno -1] Metadata file does not match checksum
Trying other mirror.
xcp-ng-updates/primary_db                                                                      | 1.6 MB  00:00:01     
Resolving Dependencies
--> Running transaction check

Update completed...

Dependency Installed:
  xcp-efivar-utils.noarch 0:1.0.0-1.xcpng8.3                                                                          

Updated:
  amd-microcode.noarch 0:20260519-1.1.xcpng8.3                  bash.x86_64 0:4.2.46-30.1.xcpng8.3                    
  blktap.x86_64 0:3.55.5-9.3.xcpng8.3                           ca-certificates.noarch 0:2021.2.50-73.1.xcpng8.3      
  edk2.x86_64 0:20220801-1.7.11.2.xcpng8.3                      forkexecd.x86_64 0:26.1.16-1.1.xcpng8.3               
  gmp.x86_64 1:6.2.1-8.1.xcpng8.3                               gpumon.x86_64 0:24.1.0-96.1.xcpng8.3                  
  kernel.x86_64 0:4.19.19-8.0.46.10.xcpng8.3                    krb5-libs.x86_64 0:1.21.3-4.1.xcpng8.3                
  message-switch.x86_64 0:26.1.16-1.1.xcpng8.3                  openssh.x86_64 0:9.8p1-1.2.6.xcpng8.3                 
  openssh-clients.x86_64 0:9.8p1-1.2.6.xcpng8.3                 openssh-server.x86_64 0:9.8p1-1.2.6.xcpng8.3          
  p11-kit.x86_64 0:0.24.1-4.xcpng8.3                            p11-kit-trust.x86_64 0:0.24.1-4.xcpng8.3              
  qcow-stream-tool.x86_64 0:26.1.16-1.1.xcpng8.3                redhat-lsb-core.x86_64 0:4.1-28.2.1.xcpng8.3          
  redhat-lsb-submod-security.x86_64 0:4.1-28.2.1.xcpng8.3       rrdd-plugins.x86_64 0:26.1.16-1.1.xcpng8.3            
  sm.x86_64 0:3.2.12-23.4.xcpng8.3                              sm-cli.x86_64 0:26.1.16-1.1.xcpng8.3                  
  sm-fairlock.x86_64 0:3.2.12-23.4.xcpng8.3                     squeezed.x86_64 0:26.1.16-1.1.xcpng8.3                
  varstored.x86_64 0:1.3.4-2.1.xcpng8.3                         varstored-guard.x86_64 0:26.1.16-1.1.xcpng8.3         
  varstored-tools.x86_64 0:1.3.4-2.1.xcpng8.3                   vhd-tool.x86_64 0:26.1.16-1.1.xcpng8.3                
  wsproxy.x86_64 0:26.1.16-1.1.xcpng8.3                         xapi-core.x86_64 0:26.1.16-1.1.xcpng8.3               
  xapi-nbd.x86_64 0:26.1.16-1.1.xcpng8.3                        xapi-rrd2csv.x86_64 0:26.1.16-1.1.xcpng8.3            
  xapi-storage-script.x86_64 0:26.1.16-1.1.xcpng8.3             xapi-tests.x86_64 0:26.1.16-1.1.xcpng8.3              
  xapi-xe.x86_64 0:26.1.16-1.1.xcpng8.3                         xcp-emu-manager.x86_64 0:1.2.1-2.xcpng8.3             
  xcp-featured.x86_64 0:1.2.1-4.xcpng8.3                        xcp-networkd.x86_64 0:26.1.16-1.1.xcpng8.3            
  xcp-ng-generic-lib.x86_64 0:1.1.1-5.xcpng8.3                  xcp-ng-xapi-plugins.noarch 0:1.17.0-1.xcpng8.3        
  xcp-rrdd.x86_64 0:26.1.16-1.1.xcpng8.3                        xen-dom0-libs.x86_64 0:4.17.6-12.2.xcpng8.3           
  xen-dom0-tools.x86_64 0:4.17.6-12.2.xcpng8.3                  xen-hypervisor.x86_64 0:4.17.6-12.2.xcpng8.3          
  xen-libs.x86_64 0:4.17.6-12.2.xcpng8.3                        xen-tools.x86_64 0:4.17.6-12.2.xcpng8.3               
  xenopsd.x86_64 0:26.1.16-1.1.xcpng8.3                         xenopsd-cli.x86_64 0:26.1.16-1.1.xcpng8.3             
  xenopsd-xc.x86_64 0:26.1.16-1.1.xcpng8.3                      xo-lite.noarch 0:0.24.0-1.xcpng8.3                    
  xsconsole.x86_64 0:11.0.9.1-1.3.xcpng8.3                      zlib.x86_64 0:1.2.7-17.1.xcpng8.3                     

Complete!

flakpyro

Installed on my usual test hosts. Stand alone hosts all rebooted fine.

My test pool of 2 servers failed to rolling pool reboot until i restarted the tool stack on both hosts. Error log is attached.
2026-08-14T17_30_10.463Z - XO.txt

JeffBerntsen

@gleh Installed and seems to be running well so far on my test server.

bufanda

@gleh installed on my lab pool. So far no issues.

Andrew

@gleh @flakpyro I have the same problem after this update. Can't migrate a VM. Can't run rolling pool reboot. Did a tool stack restart, did not help. Big issue as the pool master will have to be rebooted with live VMs running. I was able to suspend/unsuspend a VM. I could stop but not start a VM on the pool.

vm.migrate
{
  "targetHost": "388f9aad-2e39-41fa-acac-18a498eeac34",
  "vm": "450015b0-e878-5067-e8b6-8d0513a750f6"
}
{
  "code": "INTERNAL_ERROR",
  "params": [
    "xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\")"
  ],
  "task": {
    "uuid": "8e1d9888-9f9f-ebab-8a9f-87d2cd651113",
    "name_label": "Async.VM.pool_migrate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260815T18:38:30Z",
    "finished": "20260815T18:38:32Z",
    "status": "failure",
    "resident_on": "OpaqueRef:a8fb168d-bd82-5c1a-60fe-e965eefc43ef",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "INTERNAL_ERROR",
      "xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\")"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2917))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2966))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xenopsd-xc)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xenopsd-xc)(filename ocaml/libs/stunnel/stunnel.ml)(line 414))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 2865))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 103))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_task.ml)(line 111))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3412))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3422))((process xenopsd-xc)(filename ocaml/xenopsd/lib/xenops_server.ml)(line 3443))((process xenopsd-xc)(filename ocaml/xapi-idl/lib/task_server.ml)(line 192))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3481))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3652))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 262))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 268))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 293))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 434))((process xapi)(filename ocaml/xapi/xapi_xenops.ml)(line 3490))((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 458))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2519))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "INTERNAL_ERROR(xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\"))",
  "name": "XapiError",
  "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\"))
    at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/index.mjs:1329:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/index.mjs:1363:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/index.mjs:1353:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/index.mjs:1560:14)"
}

Andrew

@gleh After the update and a manual primary host reboot I was able to start new VMs and migrate VMs, but still could not run rolling pool update. I tried restarting tool stack, but it did not help.

pool.rollingReboot
{
  "pool": "40a08640-88dc-7ff9-9190-e13b89025b4d",
  "bypassBackupCheck": false,
  "shutdownPinnedVms": false
}
{
  "code": "CANNOT_EVACUATE_HOST",
  "params": [
    "NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8"
  ],
  "call": {
    "duration": 4,
    "method": "host.assert_can_evacuate",
    "params": [
      "* session id *",
      "OpaqueRef:a8fb168d-bd82-5c1a-60fe-e965eefc43ef"
    ]
  },
  "message": "CANNOT_EVACUATE_HOST(NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8)",
  "name": "XapiError",
  "stack": "XapiError: CANNOT_EVACUATE_HOST(NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8)
    at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:65:5)
    at processImmediate (node:internal/timers:502:9)"
}

XCP-ng-JustGreat

Applied the latest candidate updates to my four-node pool home lab. The once or twice that I previously tried the RPU feature, it didn't work for me so have since performed the updates from the dom0 CLI followed by properly sequenced manual host reboots. (I'm happy to try RPU in any future update where correcting that functionality is a focus.) Post update, everything appears to be working well including live migration during which the VM is deliberately made busy e.g. Windows and Linux updates; taking and deleting snapshots and tailing the SMlog to see if any GC errors occur. (Didn't see any.) I also like the new XCP-ng VM firmware boot logo. That's a nice touch as it further emphasizes Vates' commitment to the platform from the firmware on up. That's good marketing! All in all, this looks like a solid set of updates.

MajorP93

Hmmm I have been reading quite often in the last months that people have issues with RPU.

I tried RPU during every single update within the last year and for me it only failed when one (or multiple) VMs have not been live migrateable.

That is not really the "fault" of the RPU process but rather related to the VMs.

Finding out the cause has been a bit finicky though. (reading through log file)

Maybe the biggest improvement to RPU would be some form of better error reporting?

That would make it more clear to people what actually caused the issues rather than thinking "RPU is broken".

olivierlambert

Yes, here that's an error that could be translated like this: some (7?) VMs cannot boot on the other available hosts, because they are using a resource local to the host they are running on right now. So they can't be evacuated.