XCP-ng 8.3 betas and RCs feedback 🚀

archw

@stormi
Cool...look forward to it!

gb.123

Is there any real performance difference between installing XCP-NG host on BIOS mode vs installing it on UEFI mode? (Not the VMs, I mean the host itself)

Is there any benefit of installing xcp-ng using uefi, considering we can still have uefi VMs even if we install the host in BIOS mode... correct ?

bufanda

@gb-123 I don't see the relevance of performance. It's in both cases the same software running it's just diffrent how a boot is handled and UEFI has some more security features to ensure the OS running on the hardware is authentic where as in BIOS malware can easily overtake the OS or tamper with the OS.

In normal operation you wouldn't see performance diffrence in my opinion.

jivanpal

@gb-123 Using BIOS for a hypervisor generally complicates things related to disk and partition management, since a BIOS machine boots from an MBR-formatted disk, and MBR does not directly support drives greater than 2TiB. For a hypervisor, the main benefits of UEFI are:

support for booting from GPT-formatted drives (which thus can be larger than 2TiB);
power management (which is generally better in a UEFI-native OS than one being booted in CSM (a.k.a Legacy BIOS) mode; and
Secure Boot, which you can use to ensure that the OS being booted is cryptographically signed by an OS vendor that you trust.

See here for a more detailed comparison of BIOS and UEFI. The comments about Fast Boot aren't relevant here.

I personally have XCP-ng installed in UEFI mode and then run all of my VMs in BIOS mode, because I want those benefits of UEFI on the host system, but don't want all of my guests/VMs' boot disks to have the overhead of an EFI System Partition.

stormi

Also note that support for installing on legacy BIOS firmware will likely be removed in a future release, so better select UEFI by default. In XCP-ng, this will still be supported, but there's a deprecation notice in the installer.

gb.123

@bufanda @jivanpal

Thank you soooo much for your valuable answers !

@stormi

Thanks for letting me know about the deprecation. I missed seeing it in XCP-Ng 8.3 Beta 2 while I was testing it using the ISO install method.
I believe there is no 'upgrade' path from BIOS to UEFI? ( I would need to completely re-install the host again... right ?)

Does XCP-ng have the option of secure boot when installing from iso? (I am not talking about the Guest VMs here)

jivanpal

@gb-123

I believe there is no 'upgrade' path from BIOS to UEFI? ( I would need to completely re-install the host again... right ?)

That is correct, and the installation documentation mentions this:

WARNING
NEVER switch from UEFI to BIOS (or vice-versa) after you installed XCP-ng. Stick to the mode that you chose during the install.

Does XCP-ng have the option of secure boot when installing from iso?

Not yet, though that feature seems to be tracked on GitHub here and mentions a talk discussing the scope of the problem (YouTube video). Unfortunately, it looks like there hasn't been any progress on this in the last 2 or 3 years.

stormi

I just published the last big update before the release of the XCP-ng 8.3 Release Candidate. The last pre-release before the final release.

Now is the best time to test thoroughly.

The changes contain both the result of the work of XCP-ng developers, and a rebase on the latest updates of the XenServer 8 release when appropriate.

Main highlights:

Rebased on XenServer 8 + updates published since the release
We enriched the API in order to let Xen Orchestra give you feedback about the status of Secure Boot in your UEFI VMs and updated the Secure Boot documentation
- Xen Orchestra feature 1: Warn before UEFI VM creation if Secure Boot is on but the pool is not setup for SB
- Xen Orchestra feature 2: Display accurate Secure Boot status and allow to fix a VM's UEFI certs
- Both will be included in the next Xen Orchestra release, which is currently planned for the end of this week.
Security updates: curl, Intel and AMD microcode, OpenSSL, openvswitch, sudo, xen
Installer: support for upgrading from 8.3 alpha/betas is back.
More python tools ported to python3.
Alternate "debugging" kernel updated to version 4.19.316.
Optional netdata package updated to versionn 1.44.3.
New optional driver package mlx4-modules-alt which provides the LTS 4.9 version of the MLX4 drivers, used by older ConnectX cards.
XO Lite updated to version 0.2.3:
- [Pool/Dashboard] In the Storage usage card, DVDs are no longer taken into account
- [Header] Add link to XOA when XOA is detected on the pool
- Add german translation (based on the contribution made by @borzel.

Packages details (detailed changelog available by following the links)

Reboot after update

probain

@stormi
Update: This no longer gives errors after yum autoremove.
I'm guessing that this might be a holdover from me upgrading in place (yum) from 8.2?

*Tried updating, both through XO Source, and via yum upgrade.

Results in error:*

Transaction Summary
========================================================================================================================
Install   1 Package  (+4 Dependent packages)
Upgrade  88 Packages

Total size: 164 M
Is this ok [y/d/N]: y
Downloading packages:
Running transaction check
Running transaction test


Transaction check error:
  file /lib/firmware/intel-ucode/06-8f-05 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-8f-06 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-8f-07 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-8f-08 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-97-02 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-97-05 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-9a-03 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-9a-04 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-b7-01 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-be-00 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-bf-02 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-bf-05 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-cf-01 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64
  file /lib/firmware/intel-ucode/06-cf-02 from install of intel-microcode-20240419-1.xcpng8.3.noarch conflicts with file from package microcode_ctl-2:2.1-26.xs28.1.xcpng8.2.x86_64

Error Summary
-------------

[16:24 xcp ~]#

Update: I fixed it with yum autoremove

[16:29 xcp ~]# yum autoremove
Loaded plugins: fastestmirror
Resolving Dependencies
--> Running transaction check
---> Package libqb.x86_64 0:1.0.1-6.el7 will be erased
---> Package libxslt.x86_64 0:1.1.28-5.el7 will be erased
---> Package microcode_ctl.x86_64 2:2.1-26.xs28.1.xcpng8.2 will be erased
---> Package python2-bitarray.x86_64 0:0.8.3-2.el7 will be erased
---> Package python2-scapy.noarch 0:2.4.5-3.xcpng8.3 will be erased
--> Finished Dependency Resolution
--> Finding unneeded leftover dependencies
Found and removing 0 unneeded dependencies

Dependencies Resolved

=================================================================================================================================================================================================================
 Package                                           Arch                                    Version                                                   Repository                                             Size
=================================================================================================================================================================================================================
Removing:
 libqb                                             x86_64                                  1.0.1-6.el7                                               @install/$releasever                                  193 k
 libxslt                                           x86_64                                  1.1.28-5.el7                                              @install/$releasever                                  486 k
 microcode_ctl                                     x86_64                                  2:2.1-26.xs28.1.xcpng8.2                                  @xcp-ng-updates                                        12 M
 python2-bitarray                                  x86_64                                  0.8.3-2.el7                                               @xcp-ng-base                                          210 k
 python2-scapy                                     noarch                                  2.4.5-3.xcpng8.3                                          @xcp-ng-base                                           11 M

Transaction Summary
=================================================================================================================================================================================================================
Remove  5 Packages

Installed size: 25 M
Is this ok [y/N]: y
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Erasing    : 2:microcode_ctl-2.1-26.xs28.1.xcpng8.2.x86_64                                                                                                                                                 1/5
  Erasing    : python2-scapy-2.4.5-3.xcpng8.3.noarch                                                                                                                                                         2/5
  Erasing    : python2-bitarray-0.8.3-2.el7.x86_64                                                                                                                                                           3/5
  Erasing    : libxslt-1.1.28-5.el7.x86_64                                                                                                                                                                   4/5
  Erasing    : libqb-1.0.1-6.el7.x86_64                                                                                                                                                                      5/5
  Verifying  : python2-scapy-2.4.5-3.xcpng8.3.noarch                                                                                                                                                         1/5
  Verifying  : libqb-1.0.1-6.el7.x86_64                                                                                                                                                                      2/5
  Verifying  : libxslt-1.1.28-5.el7.x86_64                                                                                                                                                                   3/5
  Verifying  : 2:microcode_ctl-2.1-26.xs28.1.xcpng8.2.x86_64                                                                                                                                                 4/5
  Verifying  : python2-bitarray-0.8.3-2.el7.x86_64                                                                                                                                                           5/5

Removed:
  libqb.x86_64 0:1.0.1-6.el7      libxslt.x86_64 0:1.1.28-5.el7      microcode_ctl.x86_64 2:2.1-26.xs28.1.xcpng8.2      python2-bitarray.x86_64 0:0.8.3-2.el7      python2-scapy.noarch 0:2.4.5-3.xcpng8.3

Complete!

stormi

@probain I left this conflict on purpose because update via yum is not supported from 8.2 to 8.3.

flakpyro

@stormi

Since updating one of our linux based virtual appliances (SingleWire Infromacast) no longer boots. It begins booting then hits

"kexec_core: Starting new kernel" then powers off.

xl dmesg shows the following:

(XEN) [15750.951614] d11v0 Triple fault - invoking HVM shutdown action 3
(XEN) [15750.951615] *** Dumping Dom11 vcpu#0 state: ***
(XEN) [15750.951617] ----[ Xen-4.17.4-3  x86_64  debug=n  Not tainted ]----
(XEN) [15750.951617] CPU:    11
(XEN) [15750.951618] RIP:    0010:[<ffffffffb3000089>]
(XEN) [15750.951618] RFLAGS: 0000000000010006   CONTEXT: hvm guest (d11v0)
(XEN) [15750.951619] rax: ffffffffb3000089   rbx: 0000000000100800   rcx: 00000000000000a0
(XEN) [15750.951620] rdx: 000000004a411000   rsi: 000000000fbf3000   rdi: 000000006072a000
(XEN) [15750.951621] rbp: 000000000d000000   rsp: 000000004a403f58   r8:  0000000016000000
(XEN) [15750.951621] r9:  000000004a410d28   r10: 000000004a72d000   r11: 000000000000000e
(XEN) [15750.951622] r12: 0000000000000000   r13: 0000000000000000   r14: 0000000000000000
(XEN) [15750.951622] r15: 0000000000000000   cr0: 0000000080000011   cr4: 00000000000000a0
(XEN) [15750.951623] cr3: 000000006072a000   cr2: ffffffffb3000089
(XEN) [15750.951623] fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) [15750.951624] ds: 0018   es: 0018   fs: 0000   gs: 0000   ss: 0018   cs: 0010

Any ideas? Was working on the latest batch of updates before the ones released today.

EDIT:

Running a yum downgrade xen-hypervisor xen-dom0-libs xen-dom0-tools xen-tools xen-libs followed by a reboot gets the VM booting again.

stormi

@flakpyro Interesting. I will build intermediary releases of Xen to help narrow down to the changes that caused this. Will you be available to test them?

flakpyro

@stormi Of course, anything i can do to help just let me know. We have a number of systems we can use to test builds on.

andyhhp

@flakpyro

This is ultimately a bug in Linux. There was a range of Linux kernels which did something unsafe on kexec which worked most of the time but only by luck. (Specifically - holding a 64bit value in a register while passing through 32bit mode, and expecting it to still be intact later; both Intel and AMD identify this as having model specific behaviour and not to rely on it).

A consequence of a security fix in Xen (https://xenbits.xen.org/xsa/advisory-454.html) makes it reliably fail when depended upon in a VM.

Linux fixed the bug years ago, but one distro managed to pick it up.

Ideally, get SingleWire to fix their kernel. Failing that, adjust the VM's kernel command line to take any ,low or ,high off the crashkernel= line, because that was the underlying way to tickle the bug IIRC.

The property you need to end up with is that /proc/iomem shows the Crash kernel range being below the 4G boundary, because the handover logic from one kernel to the other simply didn't work correctly if the new kernel was above 4G.

flakpyro

@andyhhp Thanks, from what i had found on google i suspected this was a bug. Their disto is high customized too, here is what grub looks like:

Talking to the person who maintains this i learned there is a 14.24 version of the software out that seems to work, it looks like they fixed it in a later release. I was hoping to get it booting as to buy that person time to perform the upgrade. In the mean time i have rolled back Xen to 4.17.3.

andyhhp

@flakpyro If Singlewire have already fixed the bug, then just do what is is necessary to update the VM and be done with it.

That screenshot of grub poses far more questions than it answered, and I doubt we want to get into any of them.

flakpyro

@andyhhp I will pass that on thanks! I believe you're right in the long run this is something we are better off redeploying on their new appliance.

olivierlambert

Thanks for the report @flakpyro and thanks for the detailed answer @andyhhp !

gskger

@stormi Update of my XCP-ng 8.3 test server through XO from source (88 patches) went really well and after a reboot (not sure if that was needed), my VMs started normaly . I will report back if something comes up. Looking forward to the RC

XCP-ng-JustGreat

Applied recent 87 updates to 3-node home-lab pool running XCP-ng 8.3 using XO from source on the latest commit. The update worked perfectly and a mix of existing Linux and Windows VMs are running normally after the update.