XCP-ng 8.3 updates announcements and testing

Andrew

@gleh After the update and a manual primary host reboot I was able to start new VMs and migrate VMs, but still could not run rolling pool update. I tried restarting tool stack, but it did not help.

pool.rollingReboot
{
  "pool": "40a08640-88dc-7ff9-9190-e13b89025b4d",
  "bypassBackupCheck": false,
  "shutdownPinnedVms": false
}
{
  "code": "CANNOT_EVACUATE_HOST",
  "params": [
    "NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8"
  ],
  "call": {
    "duration": 4,
    "method": "host.assert_can_evacuate",
    "params": [
      "* session id *",
      "OpaqueRef:a8fb168d-bd82-5c1a-60fe-e965eefc43ef"
    ]
  },
  "message": "CANNOT_EVACUATE_HOST(NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8)",
  "name": "XapiError",
  "stack": "XapiError: CANNOT_EVACUATE_HOST(NO_HOSTS_AVAILABLE,OpaqueRef:26968b78-860d-1842-a001-c93c8f07c895|NO_HOSTS_AVAILABLE,OpaqueRef:535b4eba-bc66-85a2-db6a-a857495bfafc|NO_HOSTS_AVAILABLE,OpaqueRef:49b9453d-5725-26ad-c1db-be85f5bf0cc4|NO_HOSTS_AVAILABLE,OpaqueRef:9c17b691-1b3d-3aee-138b-8e3a7ea7e58a|NO_HOSTS_AVAILABLE,OpaqueRef:c29f5630-a246-8413-8477-73f75cfb4c61|NO_HOSTS_AVAILABLE,OpaqueRef:5440c209-4e36-97e8-6d8c-74f20f82d710|NO_HOSTS_AVAILABLE,OpaqueRef:00581b21-7700-0803-9708-d0ffac2bf2f8)
    at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/_XapiError.mjs:16:12)
    at file:///opt/xo/xo-builds/xen-orchestra-202608130941/packages/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:65:5)
    at processImmediate (node:internal/timers:502:9)"
}

XCP-ng-JustGreat

Applied the latest candidate updates to my four-node pool home lab. The once or twice that I previously tried the RPU feature, it didn't work for me so have since performed the updates from the dom0 CLI followed by properly sequenced manual host reboots. (I'm happy to try RPU in any future update where correcting that functionality is a focus.) Post update, everything appears to be working well including live migration during which the VM is deliberately made busy e.g. Windows and Linux updates; taking and deleting snapshots and tailing the SMlog to see if any GC errors occur. (Didn't see any.) I also like the new XCP-ng VM firmware boot logo. That's a nice touch as it further emphasizes Vates' commitment to the platform from the firmware on up. That's good marketing! All in all, this looks like a solid set of updates.

MajorP93

Hmmm I have been reading quite often in the last months that people have issues with RPU.

I tried RPU during every single update within the last year and for me it only failed when one (or multiple) VMs have not been live migrateable.

That is not really the "fault" of the RPU process but rather related to the VMs.

Finding out the cause has been a bit finicky though. (reading through log file)

Maybe the biggest improvement to RPU would be some form of better error reporting?

That would make it more clear to people what actually caused the issues rather than thinking "RPU is broken".

olivierlambert

Yes, here that's an error that could be translated like this: some (7?) VMs cannot boot on the other available hosts, because they are using a resource local to the host they are running on right now. So they can't be evacuated.

semarie

@Andrew said:

"xenopsd internal error: Hotplug.Hotplug_error(\"Failed to read /xapi/450015b0-e878-5067-e8b6-000000000001/private/vif/1/trunks\")"

This is related to new code in the upgrade. the trunks attribute is used for VLAN filtering (new feature).

it seems that the running code is the new one (it is knowning about trunks) but the VM doesn't have the trunks attribute.

I would be interested to have a clean picture of the situation:

on which host this VM is resident on
version of the master
version of the host where the VM is resident

flakpyro

@semarie In my case the pool i tested these updates on was a 2 host pool, i patched each host using yum and then kicked off a rolling pool reboot. So the VMs would have been resident on the pool master since it is the first rebooted when running a rolling pool reboot. I was unable to manually migrate the VMs resident on this host as well, attempting to do so would show the same error.

In my case though restarting the tool stack on both hosts allowed migrations to complete and a rolling pool reboot to run successfully.

gduperrey

@acebmxer This error does not appear to be linked to the update, but rather to a repo that has not yet had time to synchronize or is in the process of doing so at the time of your update.

semarie

@flakpyro it seems to me the proper upgrade path is:

put the master in maintenance mode (xe host-disable host=$MASTER)
evacuate the master (xe host-evacuate host=$MASTER)
yum update the master
reboot the master (xe host-reboot host=$MASTER)
once done, do the same of the others hosts

the VM would have been updated with the new trunks attribute when migrating to some updated host (in your case, when migrating to the master).

flakpyro

@semarie If that's the case then the update will likely go smoothly once its released to the stable repos and users use Xen Orchestra to run a rolling pool update which follows that workflow. Since this was a test pool and using the test repos i ran the update via yum without evacuating the host first.

semarie

@flakpyro yes. thanks for your test and to reporting the problem anyway. it is helping us to see what kind of problems users could have.