XCP-ng 8.3 updates announcements and testing

manilx

@Greg_E Updated 3 pools @business (3 Intel hosts, 2 EPYC hosts) all OK with stats. Also 1 pool @home (2 Intel Protectlis) all OK with stats.

archw

I updated the master pool...all went well. I updated one of the other hosts. After it rebooted, I got lots of messages about "Async.VM.clean_shutdown: 8%" on the various VMs.

My bad....XO was a two builds behind and well as XO was not up to date.

I updated to commit 95e72 and updated XOA version to 5.106.2 and all is well.

User error on my part but I hope this helps someone.

Greg_E

@Greg_E

No issues so far with my production system (8.2.x) and this batch of important updates, these are on Intel Silver (v2?) processors.

But my system is so vanilla that I doubt there would be any issues anyway. The only "out of normal" thing I've done today is storage migrate a VM off of one Truenas to another so I can apply some updates to Truenas. Then I'll migrate everything to the updated server so I can update the "main" storage.

ph7

Can't migrate VHD between pools

I updated my Intel test host/pool at the New update candidates for you to test! 8 days ago.

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

I then ran the New security update candidates for you to test! 2 days ago

yum clean metadata --enablerepo=xcp-ng-candidates
yum update --enablerepo=xcp-ng-candidates
reboot

I was no longer able to migrate a VHD from my Intel "test" host/pool to my AMD "production" host/pool
My thought was I had to wait for the "release" update.

When the update was published, on my "production" host I ran New update candidates for you to test! and New security update candidates for you to test!
(why didn't I just run yum update ??)
Still no migration

Tried to update my hosts

[10:23 x1 ~]# yum update
Inlästa insticksmoduler: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
No packages marked for update

[11:03 x2 ~]# yum update
Inlästa insticksmoduler: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
No packages marked for update

I then checked the software versions and there is a diff. in
git_id , date , xapi_build and db_schema

[10:23 x1 ~]# xe host-list params=software-version
software-version (MRO)    : product_version: 8.3.0; product_version_text: 8.3; product_version_text_short: 8.3; platform_name: XCP; platform_version: 3.4.0; product_brand: XCP-ng; build_number: 8.3.0; git_id: 0; hostname: localhost; date: 20250507T15:15:51Z; dbv: 0.0.1; xapi: 25.6; xapi_build: 25.6.0; xen: 4.17.5-13; linux: 4.19.0+1; xencenter_min: 2.21; xencenter_max: 2.21; network_backend: openvswitch; db_schema: 5.786

[10:20 x2 ~]# xe host-list params=software-version
software-version (MRO)    : product_version: 8.3.0; product_version_text: 8.3; product_version_text_short: 8.3; platform_name: XCP; platform_version: 3.4.0; product_brand: XCP-ng; build_number: 8.3.0; git_id: 2; hostname: localhost; date: 20250211T18:05:31Z; dbv: 0.0.1; xapi: 24.19; xapi_build: 24.19.2; xen: 4.17.5-13; linux: 4.19.0+1; xencenter_min: 2.21; xencenter_max: 2.21; network_backend: openvswitch; db_schema: 5.780

What is the next step to fix this?

ph7

Now I am not sure if I did run the New update candidates for you to test!
Is there a way to force the updates?

gduperrey

@ph7 As David mentioned, the security updates were released yesterday. They are no longer in the candidates repository, but in the updates repository.

Note that the updates in the testing repository have not yet been released. They include a more recent version of the XAPI. This could explain why you can no longer migrate this VHD between your test and production environments.

Are you trying to perform a live migration or with the VM powered off?

ph7

@gduperrey said in XCP-ng 8.3 updates announcements and testing:

Are you trying to perform a live migration or with the VM powered off?

No live migration, different pools, VDI migration only, powered off.
Warm migration is working.

It's OK, I can wait for the release

Andrew

@ph7 @gduperrey With current updates: Cold (VM off) migration works for me. Live migration, when forced due to incompatible CPU fails (badly, host toolstack restart required).

With the VM off, normal VM/VDI migration worked for me in the following process (VM power on after each migration, and then off again, as a test):

XCP 8.2 Intel (Pool 1) -> XCP 8.2 Intel (Pool 2) -> XCP 8.3 Intel (Pool 3) -> XCP 8.3 AMD (Pool 4) -> XCP 8.3 Intel (Pool 3)

Note: Each host/pool uses local storage. Software versions on hosts match for the same version of XCP. @ph7, looks like you need to yum clean all and yum update again.

ph7

@Andrew

looks like you need to yum clean all and yum update again.

Tried it but it didn't work
And I ran the rm -rf /var/cache/yum

[13:09 x2 ~]# yum clean all
Inlästa insticksmoduler: fastestmirror
Rensar förråd: xcp-ng-base xcp-ng-updates
Cleaning up everything
Maybe you want: rm -rf /var/cache/yum, to also free up space taken by orphaned data from disabled or removed repos
Cleaning up list of fastest mirrors

[13:14 x2 ~]# rm -rf /var/cache/yum

[13:14 x2 ~]# yum update
Inlästa insticksmoduler: fastestmirror
Determining fastest mirrors
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
xcp-ng-base/signature                     |  473 B  00:00:00     
xcp-ng-base/signature                     | 3.0 kB  00:00:00 !!! 
xcp-ng-updates/signature                  |  473 B  00:00:00     
xcp-ng-updates/signature                  | 3.0 kB  00:00:00 !!! 
(1/2): xcp-ng-updates/primary_db          | 238 kB  00:00:00     
(2/2): xcp-ng-base/primary_db             | 3.9 MB  00:00:00     
No packages marked for update

And rebooted

ph7

Cold migration doesn't work, but Warm migration does.
No panic, I can wait for the release

stormi

Just to be sure, is there an issue for us to investigate here, or expected failure due to version mismatch?

ph7

@stormi
If You are asking me, I can run the New update candidates for you to test! on my "production" home lab server if the testing repo still exist

stormi

Yes update candidates that were not urgent security fixes are still in the xcp-ng-testing repository (and more is coming soon, today or on
monday).

ph7

@stormi
I ran the xcp-ng-testing and now the migration seems to work

stormi

New update update candidates for you to test!

Unless major issues are found, this should be the last wave of update candidates before we publish everything as official updates for XCP-ng 8.3.

cifs-utils: update and rebuild based on the sources for the RHEL9 package. This fixes several low priority CVEs (in the context of XCP-ng) and will make future vulnerability patching easier.
curl: update to version 8.9.1, based on RHEL 10 package, and apply an additional fix for CVE-2024-8096 (low impact in XCP-ng context).
intel-e1000e: major driver update, backported from Linux kernel 5.10.179, to fix issues with recent hardware.
kernel: Fix support of dynamic tracepoints when debugging the dom0 Linux kernel with the perf tool
ncurses: Revert -devel package ABI to version 5 to avoid potential library conflicts in packages built against it
openssh: rebuild against updated ncurses package
python3-docutils: new dependency of cifs-utils
samba:
- Fix CVE-2016-2124, a flaw on SMB1 auth. An attacker could retrieve the password by using NT1.
- Fix CVE-2021-44142, an out-of-bounds heap read write vulnerability that allows remote attackers to execute arbitrary code by using VFS_fruit module.
systemtap: rebuild against updated ncurses package
xapi: Remove pvsproxy.service from the list of units restarted on xcp-rrdd update. The service in question attempts to start a proprietary component from XenServer that isn't present in XCP-ng, which led to displaying a not so pretty error in the logs.
xcp-ng-release: Enable missing xcp-rrdd plugins by default. Yes, failure to do so was what caused the empty stats issue you have been seeing in previous update candidates.
xen: rebuild against updated ncurses package + some fixes.
xo-lite: Update to 0.10.1.

Test on XCP-ng 8.3

From an up-to-date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update --enablerepo=xcp-ng-testing
reboot

The usual update rules apply: pool coordinator first, etc.

Versions

cifs-utils: 7.1-2.1
curl: 8.9.1-5.1.xcpng8.3
intel-e1000e: 5.10.179-1.xcpng8.3
kernel: 4.19.19-8.0.38.2.xcpng8.3
ncurses: 6.4-6.20240309.xcpng8.3
openssh: 7.4p1-23.3.3.xcpng8.3
python3-docutils: 0.14-1.el7
samba: 4.10.16-25.1.xcpng8.3
systemtap: 4.0-5.2.xcpng8.3
xapi: 25.6.0-1.5.xcpng8.3
xcp-ng-release: 8.3.0-32
xen: 4.17.5-10.1.xcpng8.3
xo-lite: 0.10.1-1.xcpng8.3

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Special focus:

We updated the e1000e driver. If you have Intel PCI-Express network chipsets, please test this update and verify that network connectivity and features that you depend on work as expected.
SMB shares and SRs.
yum still appearing to work correctly after the update.
SSH connection to hosts.
Stats. But I'm sure that's the first thing several among you will test already.

Test window before official release of the updates

Around one week, unless major issues are found.

flakpyro

@stormi Updated both of my test hosts.

Machine 1:
Intel Xeon E-2336
SuperMicro board.

Machine 2:
Minisforum MS-01
i9-13900H
32 GB Ram
Using Intel X710 onboard NIC

Everything rebooted and came up fine. The MS-01 i test with uses i40e and intel-igc not the e1000 driver. The other machine with the SuperMicro board uses igb so im afraid i'm not much help in testing that driver.

yum commands did seem to work from the small handful i ran.

And yes, stats do indeed work again

I never noticed the issue with Server 2025 and hanging on reboot since the updates from last week. Were you able to see anything in the dump files i sent?

Greg_E

@flakpyro

Off topic, how does XCP handle the E and P cores in that 13900 machine? That was one of the considerations that made me skip these Intel processors in favor of AMD.

I'll get mine updated as soon as possible, on going work to clean out my tech room so it can get cut in half for class space. Goodbye future expansion if we need it!

Andrew

@stormi Minor issue: e1000e does not show driver version with ethtool:

# ethtool -i eth0
driver: e1000e
version:
firmware-version: 0.1-4
expansion-rom-version:
bus-info: 0000:00:1f.6
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

flakpyro

@Greg_E It seems to handle it fine, i have 2 of these systems running without issue with 8.3!

stormi

@flakpyro No, I'm waiting for more feedback from the devs. All I have from them for now is it looks like either a firmware or a passthrough issue. I don't think we have changed anything to fix it.