XCP-ng 8.3 betas and RCs feedback 🚀

mbven

Somewhere between 8.3b2 and 8.3rc1 Windows 2016 was broken, where it immediately has a "kmode exception not handled" bluescreen on boot.

I had to downgrade back to 8.3b2 for an existing VM. I can reproduce the failure on a clean install from 2016 Server Essentials, using the Windows server 2016 template. It installs ok until it reboots and see the bluescreen immediately on each boot.

https://go.microsoft.com/fwlink/p/?LinkID=2195170&clcid=0x409&culture=en-us&country=US

It still fails even with latest patches, which I believe is 8.3rc2.

Any suggestions?

flakpyro

@mbven Strange we have a few Windows Server 2016 servers remaining that seem to be working since since the last batch of updates. I even tried to reboot one just to be sure. They are running "Microsoft Windows Server 2016 Datacenter". What CPU are your servers running? Any errors in xl dmesg when the crash occurs?

Andrew

@mbven I too have Windows Server 2016 (with 2024-09 updates) running on current 8.3 RC2...

mbven

@flakpyro Thanks for the feedback. Just tried the eval iso of Windows Server 2016 Datacenter and it spun up ok with latest xcp-ng patches.

So it looks like it's a problem specifically with the Windows Server 2016 Essentials edition and 8.2rc1 and later.

Unfortunately that's the edition that I'm licensed for and use. It's failed on both an Intel 12700k and a Xeon E3-1275v3 with the same blue screen message.

xisco

After doing a yum update on one of my 8.3 RC1 server and rebooting, there were no ethernet cards detected.
It happened from time to time in older versions of xcp-ng but after rebooting some times and doing emergency network reset worked fine before. This time it didn't.
Finally I reinstalled RC2 from ISO.
I have two HPE (ProLiant ML350 Gen10), and the master is the one showing this problem, the other one doesn't show this problem.
Doing a yum update in the secondary server promoted to master worked as a charm.
Any ideas?

stormi

@xisco I'd wager it's not directly related to the updates, but rather to the reboot process and something (a bug? a hardware issue?) triggered by your specific setup.

When you say no ethernet cards detected, do you mean even at the PCI level?

bleader

We have 2 packages updated for the first 8.3 security update, a bit late to be part of the final ISO but they will be available at release time. For people willing to test them and provide feedback, see the announcement below.

New security update candidates (xen, intel-microcode)

A new XSA was published on September 24th 2024.
Intel published a microcode update on the September 10th 2024.

XSA-462 a malicious HVM or PVH guest can trigger a DoS of the host.

SECURITY UPDATES

xen-*:
* Fix XSA-462 - x86: Deadlock in vlapic_error(). The handling of x86's APIC (Advanced Programmable Interrupt Controller) allows a guest to configure an illegal vector to handle error interrupts. This causes the vlapic_error() to recurse, this is protected, but the lock used for this protection will try to be taken recursiveley, leading to a deadlock.
intel-microcode:
    * Latest Intel microcode update, still named IPU 2024.3, including security updates for:
        * INTEL-SA-01103
        * INTEL-SA-01097

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-candidates
yum update "xen-*" intel-microcode --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: xen-4.17.5-3.xcpng8.3
intel_microcode: intel-microcode-20240815-1.xcpng8.3

What to test

Normal use and anything else you want to test.

Test window before official release of the update

Until 8.3 release.

ThierryC01

@bleader Update successful, no issue so far. There is just a message appearing when starting the update: "Delta RPMs disabled because /usr/bin/applydeltarpm not installed."

All VM working for now.

john.c

@ThierryC01 said in XCP-ng 8.3 betas and RCs feedback :

@bleader Update successful, no issue so far. There is just a message appearing when starting the update: "Delta RPMs disabled because /usr/bin/applydeltarpm not installed."

All VM working for now.

That's harmless. It's just notifying that the system doesn't have deltarpm package installed. This package reduces the size of updates based on what's already installed.

flakpyro

@bleader Updated a small test server without any issues.

ph7

I powered on my spare test "server" and ran this update.
All went fine.
After the reboot I started the fully updated XOA (local installed) and ran it for a few minutes.
then I shut it down but it sat for a few minutes so I did a force shut down from xsconsole.
Rebooted the host again and started the XOA, shut it down and it worked fine
Tested one more time and it went OK.

steff22

Not sure if this is Xcp-ng 8.3 or Nvidia based error.

But when I try to take gpu passthrough with a gtx 1070ti with windows 10. I get the drivers installed fine without errors but after reboot I get an error triangle in the windows device mangager-. error43 and no image on the screen

Wasn't Xcp-ng 8.3 related or windows drivers I think.

Tested with another PC with Xcp-ng 8.3 and everything worked immediately. So think it must be Asrock rack B650D4U3-2L2Q bios related or ipmi trying to use the same gpu.

XCP-ng-JustGreat

Applying the latest XCP-ng 8.3 RC2 "xen-*" and intel-microcode updates from the candidate repository worked great here as well. Everything appears to be running well.

gskger

@bleader There is a xcp-ng-8.3.0.iso on the ISO repository. Is that the release of XCP-ng 8.3 ? Looking forward to an official announcement .

manilx

@gskger Good catch

olivierlambert

@gskger Yes, but the official announcement is for tomorrow. Be careful because the release notes are crucial for this release, so wait for them before doing anything production related

(and yes we've already seen PAID users with XCP-ng 8.3 in PRODUCTION environment, people are crazy Even if it's really stable, I wouldn't have bet my entire production on it)

flakpyro

@olivierlambert I am one of those crazy people as you well know! But we needed the better hardware and driver support with our new servers.

That said if we already did the ISO upgrade to RC2 + latest updates are we essentially on the final version already or will there be additional new packages?

olivierlambert

Yes

flakpyro

@olivierlambert Thats great news and makes for an easy Monday then tomorrow! Happy to actually be on a stable release now!

Also congrats on the release! We have been running 8.3 on remote single host sites since March of this year and in our core production environment since August, other than a few NFS4 related woes (Solved by using NFS3 instead) its been a very solid experience even for something considered "pre-release".

gskger

@olivierlambert People might be a little crazy, but they also have trust in the test and RC lifecycle of XCP-ng, which has proven to be very reliable thanks to the dedication of the Vates team and also the community. But I agree, for production use it's better to wait for a GA announcement. Anyway, congratulation to this important milestone in the development of XCP-ng .