XCP-ng 8.3 updates announcements and testing

Greg_E

The move is more about spending yet more personal money for something that is primarily used for learning what I need to know for work (a rack, UPS, and some odds and ends at $800). Or half of it is, half of it is a VMware system and that's to let me learn what I need to leave this job and find one that pays more. And then move that work to something easier to use that costs less. Still amazed at how many places just took the price increase and are still not making plans to move to something else, even if they are cutting core counts in half to save half of that new money. But I'm also seeing that the general trend in IT around where I live is to get into a Silo, and never ever take on another task to fill a need. This way you never want to move laterally to other products, just keep doing the same things the same ways until management gets tired of hearing "we can't do that" and fires everyone to replace them with an MSP (or other contractor).

So for the off topic rant, feeling salty again today.

stormi

New security update candidates for you to test!

We're still working on the updates that you started testing (and a few more), but right now there's an emergency: a security update.

Yet more vulnerabilities in Intel hardware, addressed in two complementary ways: patching Xen and updating Intel microcode.

Test on XCP-ng 8.3

From an up-to-date host, or a host on which previous update candidates had been installed. Both fit.

yum clean metadata --enablerepo=xcp-ng-candidates
yum update --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

intel-microcode: 20250501-1.xcpng8.3
xen: 4.17.5-13.1.xcpng8.3

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

~24h. That's an urgent one.

ph7

@stormi
Stats still not good
Only rebooted

After Restart tool stack

andriy.sultanov

@ph7 This update only covers the security issue described above. Fix for the stats issue will roll out later.

flakpyro

@stormi installed on the same test machines i have the other batch of updates installed on. No issues after a reboot.

ph7

@andriy.sultanov
Sorry, if I only could read...
Anyhow, My updated host running on intel seems to work just fine.

Greg_E

@stormi

My lab is down for a few days, so no testing for me. And it is AMD so maybe not useful. I probably won't be back running until Friday.

stormi

@Greg_E Thanks for letting me know. It is useful to make sure that it's still working on any kind of hardware, but your lab won't participate this time

bufanda

Installed the patches on my lab pool and both hosts are up and running and no issues so far.

XCP-ng-JustGreat

Latest urgent updates installed on 3-node Intel pool. Everything is working as before including the pre-production code "no stats" issue, but that still resolves following xe-toolstack-restart command. Since it is currently Microsoft patch Tuesday here, the latest Windows 11-24H2 2025-05 cumulative update was installed to VM along with various Linux VM updates and live VM host migrations. All working well including latest af03c Xen Orchestra from source (XOS).

Andrew

@stormi Upgraded my test 8.3 hosts, several Intel and AMD Zen 3. So far, so good.

bleader

Update published: https://xcp-ng.org/blog/2025/05/14/may-2025-security-update-for-xcp-ng-8-2-8-3/

Thank your for the tests.

Greg_E

@bleader

I see this for my 8.2 pool this morning, I'll kick it off when we all go into a meeting so it will be done when we finish.

manilx

@Greg_E Updated 3 pools @business (3 Intel hosts, 2 EPYC hosts) all OK with stats. Also 1 pool @home (2 Intel Protectlis) all OK with stats.

archw

I updated the master pool...all went well. I updated one of the other hosts. After it rebooted, I got lots of messages about "Async.VM.clean_shutdown: 8%" on the various VMs.

My bad....XO was a two builds behind and well as XO was not up to date.

I updated to commit 95e72 and updated XOA version to 5.106.2 and all is well.

User error on my part but I hope this helps someone.

Greg_E

@Greg_E

No issues so far with my production system (8.2.x) and this batch of important updates, these are on Intel Silver (v2?) processors.

But my system is so vanilla that I doubt there would be any issues anyway. The only "out of normal" thing I've done today is storage migrate a VM off of one Truenas to another so I can apply some updates to Truenas. Then I'll migrate everything to the updated server so I can update the "main" storage.

ph7

Can't migrate VHD between pools

I updated my Intel test host/pool at the New update candidates for you to test! 8 days ago.

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

I then ran the New security update candidates for you to test! 2 days ago

yum clean metadata --enablerepo=xcp-ng-candidates
yum update --enablerepo=xcp-ng-candidates
reboot

I was no longer able to migrate a VHD from my Intel "test" host/pool to my AMD "production" host/pool
My thought was I had to wait for the "release" update.

When the update was published, on my "production" host I ran New update candidates for you to test! and New security update candidates for you to test!
(why didn't I just run yum update ??)
Still no migration

Tried to update my hosts

[10:23 x1 ~]# yum update
Inlästa insticksmoduler: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
No packages marked for update

[11:03 x2 ~]# yum update
Inlästa insticksmoduler: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
No packages marked for update

I then checked the software versions and there is a diff. in
git_id , date , xapi_build and db_schema

[10:23 x1 ~]# xe host-list params=software-version
software-version (MRO)    : product_version: 8.3.0; product_version_text: 8.3; product_version_text_short: 8.3; platform_name: XCP; platform_version: 3.4.0; product_brand: XCP-ng; build_number: 8.3.0; git_id: 0; hostname: localhost; date: 20250507T15:15:51Z; dbv: 0.0.1; xapi: 25.6; xapi_build: 25.6.0; xen: 4.17.5-13; linux: 4.19.0+1; xencenter_min: 2.21; xencenter_max: 2.21; network_backend: openvswitch; db_schema: 5.786

[10:20 x2 ~]# xe host-list params=software-version
software-version (MRO)    : product_version: 8.3.0; product_version_text: 8.3; product_version_text_short: 8.3; platform_name: XCP; platform_version: 3.4.0; product_brand: XCP-ng; build_number: 8.3.0; git_id: 2; hostname: localhost; date: 20250211T18:05:31Z; dbv: 0.0.1; xapi: 24.19; xapi_build: 24.19.2; xen: 4.17.5-13; linux: 4.19.0+1; xencenter_min: 2.21; xencenter_max: 2.21; network_backend: openvswitch; db_schema: 5.780

What is the next step to fix this?

ph7

Now I am not sure if I did run the New update candidates for you to test!
Is there a way to force the updates?

gduperrey

@ph7 As David mentioned, the security updates were released yesterday. They are no longer in the candidates repository, but in the updates repository.

Note that the updates in the testing repository have not yet been released. They include a more recent version of the XAPI. This could explain why you can no longer migrate this VHD between your test and production environments.

Are you trying to perform a live migration or with the VM powered off?

ph7

@gduperrey said in XCP-ng 8.3 updates announcements and testing:

Are you trying to perform a live migration or with the VM powered off?

No live migration, different pools, VDI migration only, powered off.
Warm migration is working.

It's OK, I can wait for the release