XCP-ng 8.3 betas and RCs feedback 🚀

olivierlambert

kern.log, dmesg, the usual suspects.
if you have a metadata backup yes. Otherwise, you'll need to re introduce the SR, and just have to recreate your VMs and attach the existing disks to it (and without the metadata info, if your disks are all the same size, it could take same some time to find who's who)

ThierryC01

@olivierlambert I could access the SSD, filesystem was corrupted. I don't think I have a backup at hand just in case I would not be able to boot at all. What files could I transfer to get all my VMs back to normal after a clean install on a new disk?

ThierryC01

@olivierlambert System restarted, not sure how to backup XCP-NG server side properly though. I ran a backup of XO metadata and pool parameters.

olivierlambert

XCP-ng pool metadata backup is exactly what you needed in that case

archw

@stormi
FWIW...one of the boxes where I installed the latest updates (from a few days ago) did not autostart any of the VMs. I had to manually start each of them.

XCP-ng-JustGreat

Latest updates over ISO-installed 8.3 RC2 worked fine for me. I did experience one host in my three-host pool to which no VMs could be migrated. After looking at the networking from bash in DOM0, it showed that both 10G ports for the storage and migration networks were DOWN. These ports are on a genuine IBM-branded Intel X540-T2 card I bought used on eBay so it might have gone bad. Since the card has worked well for some time, I figured it couldn't hurt to re-seat it in the PCIe slot. Sure enough, that fixed it. Moral of the story: check the mundane stuff first; it's not always the fault of new updates.

olivierlambert

haha nice catch, PCI reseat is like black magic sometimes

stormi

@archw and the other hosts did?

archw

@stormi
Yes

I've not had a chance to reboot the host since then to see if something else is going on. Will do so tonight.

TopoLogical

Hi,

I'm currently testing our the RC2 with ceph backed rbd devices, which works perfectly for us on 8.2.1. After installation I tried to add an existing shared storage, without success. Then I tried to create a new one and ran into following problem. As you see I can create a volume group manually without a problem.

xe sr-create fails:

xe sr-create name-label="RC2StorageTest" shared=true type=lvm device-config:device=/dev/rbd0
Error code: SR_BACKEND_FAILURE_77
Error parameters: , Logical Volume group creation failed,

vgcreate from commandline works:

vgcreate RC2StorageTest /dev/rbd0
Physical volume "/dev/rbd0" successfully created.
Volume group "RC2StorageTest" successfully created

If I repeat the xe sr-create after manually creating a VG the VG will be removed by xe sr-create, but is still failing with the same error.

Any idea where to look to solve this issue?

archw

@stormi
Just rebooted the problematic host. All the VMs autostarted just fine.

Odd!

nomad

@stormi I see plug-late-sr is available now. Just finished testing it. Looks like it works just fine.

thanks

mbven

Somewhere between 8.3b2 and 8.3rc1 Windows 2016 was broken, where it immediately has a "kmode exception not handled" bluescreen on boot.

I had to downgrade back to 8.3b2 for an existing VM. I can reproduce the failure on a clean install from 2016 Server Essentials, using the Windows server 2016 template. It installs ok until it reboots and see the bluescreen immediately on each boot.

https://go.microsoft.com/fwlink/p/?LinkID=2195170&clcid=0x409&culture=en-us&country=US

It still fails even with latest patches, which I believe is 8.3rc2.

Any suggestions?

flakpyro

@mbven Strange we have a few Windows Server 2016 servers remaining that seem to be working since since the last batch of updates. I even tried to reboot one just to be sure. They are running "Microsoft Windows Server 2016 Datacenter". What CPU are your servers running? Any errors in xl dmesg when the crash occurs?

Andrew

@mbven I too have Windows Server 2016 (with 2024-09 updates) running on current 8.3 RC2...

mbven

@flakpyro Thanks for the feedback. Just tried the eval iso of Windows Server 2016 Datacenter and it spun up ok with latest xcp-ng patches.

So it looks like it's a problem specifically with the Windows Server 2016 Essentials edition and 8.2rc1 and later.

Unfortunately that's the edition that I'm licensed for and use. It's failed on both an Intel 12700k and a Xeon E3-1275v3 with the same blue screen message.

xisco

After doing a yum update on one of my 8.3 RC1 server and rebooting, there were no ethernet cards detected.
It happened from time to time in older versions of xcp-ng but after rebooting some times and doing emergency network reset worked fine before. This time it didn't.
Finally I reinstalled RC2 from ISO.
I have two HPE (ProLiant ML350 Gen10), and the master is the one showing this problem, the other one doesn't show this problem.
Doing a yum update in the secondary server promoted to master worked as a charm.
Any ideas?

stormi

@xisco I'd wager it's not directly related to the updates, but rather to the reboot process and something (a bug? a hardware issue?) triggered by your specific setup.

When you say no ethernet cards detected, do you mean even at the PCI level?

bleader

We have 2 packages updated for the first 8.3 security update, a bit late to be part of the final ISO but they will be available at release time. For people willing to test them and provide feedback, see the announcement below.

New security update candidates (xen, intel-microcode)

A new XSA was published on September 24th 2024.
Intel published a microcode update on the September 10th 2024.

XSA-462 a malicious HVM or PVH guest can trigger a DoS of the host.

SECURITY UPDATES

xen-*:
* Fix XSA-462 - x86: Deadlock in vlapic_error(). The handling of x86's APIC (Advanced Programmable Interrupt Controller) allows a guest to configure an illegal vector to handle error interrupts. This causes the vlapic_error() to recurse, this is protected, but the lock used for this protection will try to be taken recursiveley, leading to a deadlock.
intel-microcode:
    * Latest Intel microcode update, still named IPU 2024.3, including security updates for:
        * INTEL-SA-01103
        * INTEL-SA-01097

Test on XCP-ng 8.3

yum clean metadata --enablerepo=xcp-ng-candidates
yum update "xen-*" intel-microcode --enablerepo=xcp-ng-candidates
reboot

The usual update rules apply: pool coordinator first, etc.

Versions:

xen: xen-4.17.5-3.xcpng8.3
intel_microcode: intel-microcode-20240815-1.xcpng8.3

What to test

Normal use and anything else you want to test.

Test window before official release of the update

Until 8.3 release.

ThierryC01

@bleader Update successful, no issue so far. There is just a message appearing when starting the update: "Delta RPMs disabled because /usr/bin/applydeltarpm not installed."

All VM working for now.