XCP-ng 8.3 betas and RCs feedback 🚀

gduperrey

New Security Update Candidates (Xen and AMD CPUs) for Zenbleed

Xen is being updated to mitigate hardware vulnerabilities in AMD CPUs.

Upstream (Xen project) advisory: XSA-433
- Citrix Hypervisor Security Update for CVE-2023-20593

This issue affects systems running AMD Zen 2 CPUs. Under specific microarchitectural circumstances, it may allow an attacker to potentially access sensitive information.

As this flaw can be critical for AMD Zen 2 users, we integrated the patch into our 8.3. You can read about this vulnerability on our blog here. This update includes the latest bugfix of this patch from upstream. You can read about it here on the blog.

Test on XCP-ng 8.3

From an up to date host:

yum clean metadata --enablerepo=xcp-ng-testing
yum update "xen-*" amd-microcode --enablerepo=xcp-ng-testing
reboot

Versions:

xen-*: xen-4.13.5-10.42.3.xcpng8.3
amd-microcode: amd-microcode-20220930-2.1.xcpng8.3

What to test

Normal use and anything else you want to test. The closer to your actual use of XCP-ng, the better.

Test window before official release of the updates

None defined, but early feedback is always better than late feedback, which is in turn better than no feedback

stormi

@gb-123 said in XCP-ng 8.3 beta :

Bugs Found :
The iso creates the partitions: 1,2,3,5,6 ( Partition No. 4 seems to be missing ), this is when NO SR is created at the time of install.

Hi. I think this is on purpose, in the installer code coming initially from XenServer, so that partitions with a given number always serve the same purpose.

stormi

I'm publishing new updates to the base repository of XCP-ng 8.3:

Security fixes for AMD
Debian 12 VM template
Removal of the old and unused experimental EXT4 SR driver. Don't jump: the main EXT SR driver still uses EXT4. I'm talking of the old experimental driver we added back then when the default EXT driver would use EXT3 only. This experimental driver has been deprecated since XCP-ng 8.1.
smartmontools updated to version 7 which brings JSON output
A fix for live migration support in IPv6-only pools

lyan

Hi, I saw we are still using 4.13.5-10.42.1 version of Xen in beta, however, I noticed that there was plan to use 4.17.x by https://en.wikipedia.org/wiki/XCP-ng. I am wondering what is the roadmap of xen in xcp-ng eventually? Do we have upgrade plan for Xen?

POleszkiewicz

Great work,

BTW, would it be possible to add nvme-cli to the installer image? It would be nice if we could actually attach NVMeOF at install time and install to NVMeOF volumes (While keeping /boot either on USB/SD locally or on iSCSI), this way we could easily provision a cluster of diskless hosts, while keeping system storage redundancy by using MD RAID between two NVMEoF volumes located on different target hosts.

(with some manual work to attach NVMeOF before mounting root)

olivierlambert

@lyan Yes, for a version 9.0, because a major number change means we can make huge bump in kernel and Xen versions

olivierlambert

@POleszkiewicz Interesting, might worth doing a PoC with this to see if it works correctly. Can you open an issue on the main XCP Github repo?

stormi

To pave the way, you can also modify the installation ISO with https://github.com/xcp-ng/xcp/tree/master/scripts/iso-remaster: add nvme-cli inside install.img, rebuild the ISO, and see how the installer behaves.

lyan

@olivierlambert nice, out of curiosity, what is the rough timeline for the 9.0 release,

laurentm

Hello, I have encountered a crash with kernel panic and loop reboot on a working 8.2 install. XCP-NG was on a RAID1 SSD partition and datas on a RAID 10 SSD partition. I found no error on the SSDs (which are Enterprise quality) and had no idea how to get a simple way to have the VMs up and running if I wiped and completely reinstall XCP-NG.

So, I did a quick and dirty repair remotely through the DELL R420 IDRAC : upgrading the broken 8.2 with the 8.3 beta ISO and it worked perfectly.
Nevertheless, IDRAC is showing 192 Gb ram and XCP-NG 8.3 see only 144 Gb .
I suspect a RAM issue unless there is something wrong in the beta release.

My concern is: how can I move the VMs to another DELL R820 server which should be running 8.2 (since running a beta on production is a very bad practice) ?
Should I have to export in XVA files the VMs or should 8.2 could accept motion move ? or backup and restore with XenOrchestra ?

Thanks for your advices.

Kind regards,

Laurent

stormi

Live migration from 8.3 to 8.2 is not possible. You can't migrate to a lower release. Warm migration using XenOrchestra is possible.

See this blog post which explains the principle: https://xen-orchestra.com/blog/warm-migration-with-xen-orchestra/

laurentm

@stormi Thanks for the advice. I was pretty sure of no downgrade ability but I did not think to all of the XOA capabilities.

By the way, is there a release scheduled for the final 8.3 iso ?

stormi

@laurentm The schedule follows XenServer's schedule, with an extra delay for adaptation work, and there isn't any strongly defined ETA for it yet. We hope it will be before the end of the year.

Andrew

@stormi You also can't export XVA 8.3 and import into 8.2.... OVA did work.

Thunder-Cloud

Have I understood this correct when I think that:
XCP-NG 8.2 is same "source" as Citrix Hypervisor 8.2?
XCP-NG 8.3 is same "source" as the XenServer 8.0?

stormi

@Thunder-Cloud this is not far from accurate, if by "same source" you mean based on. Not everything from Citrix Hypervisor / XenServer is taken identical in XCP-ng, so we don't share 100% of the source code. There are various proprietary components we removed or replaced, and we also have our specific additions.

Also, the name "XenServer 8.0" is not accurate : it's called simply XenServer 8.

john.c

@stormi It's currently getting stuck on finishing startup on XCP-ng 8.3 beta 1. I have completed the installation of the 8.3 beta via netinstall media. Configured for dual ipv4 and ipv6 network stack.

However it seems to be taking a really long time to bring up the management network or not bringing it up. Alternatively something else is preventing it from finishing the start up.

If attempt to confirm the ethernet port for the management network, it's currently timing out during the attempt.

management connection issue - xcp-ng 8.3 beta1.jpg

management connection issue - xcp-ng 8.3 beta1 - message displayed.jpg

dell poweredge r620 system inventory.png

Can anyone please help me deduce which part of the finishing start-up is getting stuck?

gb.123

@john-c

Can you try installing ipv4 only and alternatively ipv6 only to see if it works ?
(Just wanted to confirm something)

laurentm

@john-c Maybe you could add an intel NIC if you have a spare one.
I like DELL servers but not much the Broadcom they provide to save some bucks.

bc-23

Hi,

I have a issue starting a PV VM on a fresh installed XCP-ng 8.3 server.
The VM was created from a template I exported from our XCP-ng 8.2 cluster and imported into the new 8.3 server.
The template creates a empty PV VM containing the PV boot information to do a network installation.

The error message I get is:
xenopsd internal error: VM = fb7977de-aa28-273b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error

In the xensource.log I don't see much more information:

Sep 28 08:20:12 X xapi: [error||26203 |Async.VM.start R:5c82647ea60e|xenops] Re-raising as INTERNAL_ERROR [ xenopsd internal error: VM
 = fb7977de-aa28-273b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error  ]
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] Async.VM.start R:5c82647ea60e failed with exception Server_error(INTERNAL_ERROR, [ 
xenopsd internal error: VM = fb7977de-aa28-273b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error  ])
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] Raised Server_error(INTERNAL_ERROR, [ xenopsd internal error: VM = fb7977de-aa28-27
3b-7e07-90a8c8639559; domid = 9; Bootloader.Bad_error  ])
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] 1/39 xenopsd-xc Raised at file ocaml/xenopsd/xc/xenops_server_xen.ml, line 2201
Sep 28 08:20:12 X xapi: [error||26203 ||backtrace] 2/39 xenopsd-xc Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 2
4
...

I skipped the remaining 36 lines from the backtrace, as this only seems to be the ocaml stack trace I it doesn't some to contain any additional relevant information.

When I compare two newly created VMs based on the PV template in the 8.2 and 8.3 environment, the look equal.
The PV elements from vm-param-list on both VMs looks like:

xe vm-param-list uuid=<UUID> | grep PV
                             PV-kernel ( RW): 
                            PV-ramdisk ( RW): 
                               PV-args ( RW): preseed/url=<install specific information>
                        PV-legacy-args ( RW): 
                         PV-bootloader ( RW): eliloader
                    PV-bootloader-args ( RW): 
                    PV-drivers-version (MRO): <not in database>
    PV-drivers-up-to-date ( RO) [DEPRECATED]: <not in database>
                   PV-drivers-detected ( RO): <not in database>

I see a difference on the bios-strings parameter, which is empty in 8.2 but contains the following in 8.3:

bios-strings (MRO): bios-vendor: Xen; bios-version: ; system-manufacturer: Xen; system-product-name: HVM domU; system-version: ; system-serial-number: ; baseboard-manufacturer: ; baseboard-product-name: ; baseboard-version: ; baseboard-serial-number: ; baseboard-asset-tag: ; baseboard-location-in-chassis: ; enclosure-asset-tag: ; hp-rombios: ; oem-1: Xen; oem-2: MS_VM_CERT/SHA1/bdbeb6e0a816d43fa6d3fe8aaef04c2bad9d3e3d

Do you have a hint what could case this error, or where I could find additional information, as the error message does not contain a lot of information.

Thanks.