XCP-ng 8.3 public alpha 🚀
-
Thanks @cocoon
So, I just pushed two updates to the 8.3 repositories, that you can get with a simple
yum update:xcp-featured, which enables partial vTPM supportguest-templates-json-*, which adds VM templates for RHEL 9 and derivatives.
-
OK, TPM is visible in Windows Server 2022 but no secure boot atm.
Bios says it is disabled. Installed it in a virtual XCP-NG VM on ESXi.
-
@cocoon Did you enable Secure Boot on the VM, and did you also install SecureBoot certificates on your pool? (and if you installed the certificates to the pool after you first started the VM - with or without SB, you also need to install them to the VM by putting it in user mode:
varstore-sb-state user {VM-UUID}varstore-sb-state {VM-UUID} user). -
@stormi said in XCP-ng 8.3 public alpha
:@cocoon Did you enable Secure Boot on the VM, and did you also install SecureBoot certificates on your pool? (and if you installed the certificates to the pool after you first started the VM - with or without SB, you also need to install them to the VM by putting it in user mode:
varstore-sb-state user {VM-UUID}).Ah yes, thanks, that works
the command is just slightly different:varstore-sb-state {VM-UUID} user
-
If you want to push the vTPM test further, you can activate bitlocker in a Windows VM and see if after a reboot the drives are decrypted without having to enter passcodes manually.
-
@stormi That was my plan, Bitlocker feature is already installed, will test this yes
-
Small Announcement / Testing Guide
There is a new and easy way for you to help testing XCP-ng 8.3: a few test scripts from the Xen project which deserve to be run on a wide variety of hosts. You don't need to run them on every host if they are truly identical, but it's good to run them on a wide range of hardware.
The first one is XTF (stands for Xen Test Framework)
Be aware that some of the tests may sometimes cause the host to crash.
XTF
Enable HVM FEP on the host. This is not mandatory but if you don't, several tests that require it will be skipped:
/opt/xensource/libexec/xen-cmdline --set-xen hvm_fep rebootBuild XTF
yum install gcc git -y git clone git://xenbits.xen.org/xtf.git cd xtf make -j8(Optional, protects your host from a crash if its hardware is vulnerable to XSA-304) Switch EPT superpages to secure mode:
xl set-parameters ept=no-exec-spRun the tests
# self test ./xtf-runner selftest -q --host # all tests # -q stands for quiet. Remove one or both if you want to see details. ./xtf-runner -aqq --host # check return code. Should be "3" which means "no failures but some tests were skipped": echo $?Switch back EPT superpages to fast mode, if needed
xl set-parameters ept=exec-spThere will be a few SKIPPED tests, but there shouldn't be many.
Known skipped tests:
test-hvm32-umip test-hvm64-umip: skipped if the CPU is not recent enough to support UMIP.test-pv64-xsa-167: always skippedtest-pv64-xsa-182: skipped in default configuration.
You can ignore them.
xen-dom0-tests
The testsuite is very limited in Xen 4.13, but let's still run what's available.
Install:
yum install xen-dom0-testsRun
/usr/libexec/xen/bin/test-cpu-policy # check return code. Must be 0, otherwise this means there was a failure. echo $? -
@stormi I am afraid my playlab host (Dell Optiplex 9010, Intel i5-3550 CPU) is to old to contribute to the XCP-ng 8.3 alpha testing, but I run the XTF test with this results:
[h01]# ./xtf-runner selftest -q --host Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESSfollowed by
[h01]# ./xtf-runner -aqq --host Combined test results: test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP [h01]# echo $? 3 [h01]#I have some HP Elitedesk 600 G6 mini coming in to test as a small virtualization lab (i5-10500T, 6 cores, 12 threads, 2.3 GHz base clock, 64GB RAM). Not the target infrastructure but will add to the range of hardware.
-
@olivierlambert said in XCP-ng 8.3 public alpha
:It's an alpha not because it's unstable
Just that we can make some updates before the official release. Happy to see it works well!A thought. Is there a planned list of stuff that we can vote on to have on the next release?
-
Hi,
No/not yet/our backlog is already too full to make many choices for something that should be out in a matter of months.
I mean, changing a color somewhere is OK, but anything else is not in the realm of possible for now.
-
@stormi My first HP Elitedesk 600 G6 Mini arrived (i5-10500T, 6 cores, 12 threads, 2.3 GHz base clock, 64GB RAM, 256GB NVME) and XCP-ng 8.3 alpha installed and updated right away (but I have the efi: EFI_MEMMAP not enabled message with a boot delay).
The XTF test on this non-enterprise hardware returns
[M01]# ./xtf-runner selftest -q --host Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESSand
[M01]# ./xtf-runner -aqq --host Combined test results: test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP [M01]# echo $? 3 [M01]#Have not done anything else with this unit yet.
-
Sorry for being very late to this show, but I would like to thank everybody involved in getting XCP-ng ready for using on modern 10th and 11th gen Intel NUCs! Great job!
So, to replace (in the near future) the old and large power guzzling lab servers in my office, I've just ordered 2x Intel NUC 11 with i5-1135G7 CPU and the intel 2.5 gbit I225 NIC.
Looking forward to testing them and getting ready for XCP-ng 8.3!
-
@stormi I have 8.3 alpha+updates running on an HP DL360 G8 and it's working as expected since currently it's mostly an update to 8.2... I did an ISO install/update and it went ok. I did have to re-install some packages that were installed before on 8.2 and fix options that were removed during the 8.3 install but it keep most of XCP/Network/VM config.
-
@stormi said in XCP-ng 8.3 public alpha
:Small Announcement / Testing Guide
There is a new and easy way for you to help testing XCP-ng 8.3: a few test scripts from the Xen project which deserve to be run on a wide variety of hosts. You don't need to run them on every host if they are truly identical, but it's good to run them on a wide range of hardware.
The first one is XTF (stands for Xen Test Framework)
XTF
Enable HVM FEP on the host. This is not mandatory but if you don't, several tests that require it will be skipped:
/opt/xensource/libexec/xen-cmdline --set-xen hvm_fep rebootBuild XTF
yum install gcc git -y git clone git://xenbits.xen.org/xtf.git cd xtf make -j8Run the tests
# self test ./xtf-runner selftest -q --host # all tests # -q stands for quiet. Remove one or both if you want to see details. ./xtf-runner -aqq --host # check return code. Should be "3" which means "no failures but some tests were skipped": echo $?There will be a few SKIPPED tests, but there shouldn't be many.
Known skipped tests:
test-hvm32-umip test-hvm64-umip: skipped if the CPU is not recent enough to support UMIP.test-pv64-xsa-167: always skippedtest-pv64-xsa-182: skipped in default configuration.
You can ignore them.
xen-dom0-tests
The testsuite is very limited in Xen 4.13, but let's still run what's available.
Install:
yum install xen-dom0-testsRun
/usr/libexec/xen/bin/test-cpu-policy # check return code. Must be 0, otherwise this means there was a failure. echo $?I'm still interested in as many users as possible running this on their hardware.
-
@stormi
Hello, here the results on my server:Test server: Dell Poweredge R730 2 x Xenon E5-2698v4 512GB RAM ./xtf-runner selftest -q --host Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS ./xtf-runner -aqq --host Combined test results: test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP Echo Result 3 /usr/libexec/xen/bin/test-cpu-policy CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok Echo Result 0 -
Here's the results from one of my test lab servers:
Test server: HP MicroServer CPU: AMD Athlon(tm) II Neo N36L RAM: 8GB xtf-runner selftest -q --host Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS xtf-runner -aqq --host Combined test results: test-pv64-cpuid-faulting SKIP test-pv64-pv-fsgsbase SKIP test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP /usr/libexec/xen/bin/test-cpu-policy CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok echo $? 0 -
Intel NUC10i5FNH Intel i5-10210U CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 3Intel NUC11PAHi5 Intel i5-1135G7 CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-hvm64-lbr-tsx-vmentry CRASH test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 6 (XEN) [ 93.051330] d13v0 Unknown Host LBR MSRs (XEN) [ 93.051334] domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_msr_write_intercept+0x4c2/0x510 (XEN) [ 93.051335] Domain 13 (vcpu#0) crashed on cpu#7: (XEN) [ 93.051336] ----[ Xen-4.13.4-9.29.1 x86_64 debug=n Tainted: H ]---- (XEN) [ 93.051337] CPU: 7 (XEN) [ 93.051338] RIP: 0008:[<000000000010446e>] (XEN) [ 93.051338] RFLAGS: 0000000000000046 CONTEXT: hvm guest (d13v0) (XEN) [ 93.051340] rax: 0000000000000001 rbx: 000000000010d000 rcx: 00000000000001d9 (XEN) [ 93.051340] rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) [ 93.051341] rbp: 0000000000000000 rsp: 0000000000119fe8 r8: 000000000000000a (XEN) [ 93.051341] r9: 000000000000001e r10: 0000000000000049 r11: 000000000000001e (XEN) [ 93.051342] r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) [ 93.051343] r15: 0000000000000000 cr0: 0000000080010011 cr4: 0000000000000020 (XEN) [ 93.051343] cr3: 000000000010d000 cr2: 0000000000000000 (XEN) [ 93.051344] fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) [ 93.051345] ds: 0033 es: 0033 fs: 0033 gs: 0033 ss: 0000 cs: 0008 (XEN) [ 95.347733] d38v0 Bad rIP 800000000000 for mode 8 (XEN) [ 97.443996] d70v0 Hit #DB in Xen context: e008:ffff82d0bfffb200 [ffff82d0bfffb200], stk e010:8000000000112ff0, dr6 ffff0ff0 (XEN) [ 97.568064] d72v0 Hit #DB in Xen context: e008:ffff82d080371ba0 [int3], stk 0000:ffff83003368ff78, dr6 ffff2ff0ASUS MINIPC PN63-S1 Intel i7-11370H CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-hvm64-lbr-tsx-vmentry CRASH test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 6 (XEN) [ 356.488438] d27v0 Unknown Host LBR MSRs (XEN) [ 356.488442] domain_crash called from arch/x86/hvm/vmx/vmx.c#vmx_msr_write_intercept+0x4c2/0x510 (XEN) [ 356.488443] Domain 27 (vcpu#0) crashed on cpu#7: (XEN) [ 356.488444] ----[ Xen-4.13.4-9.29.1 x86_64 debug=n Tainted: H ]---- (XEN) [ 356.488445] CPU: 7 (XEN) [ 356.488446] RIP: 0008:[<000000000010446e>] (XEN) [ 356.488447] RFLAGS: 0000000000000046 CONTEXT: hvm guest (d27v0) (XEN) [ 356.488448] rax: 0000000000000001 rbx: 000000000010d000 rcx: 00000000000001d9 (XEN) [ 356.488449] rdx: 0000000000000000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) [ 356.488449] rbp: 0000000000000000 rsp: 0000000000119fe8 r8: 000000000000000a (XEN) [ 356.488450] r9: 000000000000001e r10: 0000000000000049 r11: 000000000000001e (XEN) [ 356.488451] r12: 0000000000000000 r13: 0000000000000000 r14: 0000000000000000 (XEN) [ 356.488451] r15: 0000000000000000 cr0: 0000000080010011 cr4: 0000000000000020 (XEN) [ 356.488452] cr3: 000000000010d000 cr2: 0000000000000000 (XEN) [ 356.488453] fsb: 0000000000000000 gsb: 0000000000000000 gss: 0000000000000000 (XEN) [ 356.488454] ds: 0033 es: 0033 fs: 0033 gs: 0033 ss: 0000 cs: 0008 (XEN) [ 358.951619] d52v0 Bad rIP 800000000000 for mode 8 (XEN) [ 361.254829] d84v0 Hit #DB in Xen context: e008:ffff82d0bfffb200 [ffff82d0bfffb200], stk e010:8000000000112ff0, dr6 ffff0ff0 (XEN) [ 361.397060] d86v0 Hit #DB in Xen context: e008:ffff82d080371ba0 [int3], stk 0000:ffff831080c27f78, dr6 ffff2ff0HP ML310e G8 Intel E3-1240 V2 CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 3Supermicro X9DRL-7F Intel E5-2643 CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-pv64-cpuid-faulting SKIP test-pv64-pv-fsgsbase SKIP test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 3HP EliteDesk 800 G3 Intel i7-7700 CPU CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 3ASRock 4x4 BOX-5000 AMD Ryzen 5 5600U Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-pv64-cpuid-faulting SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 3HP DL360p Gen8 Intel E5-2680 CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-pv64-cpuid-faulting SKIP test-pv64-pv-fsgsbase SKIP test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 3 -
@Andrew I pushed a test build of Xen in the testing repository for XCP-ng 8.3, which will hopefully solve the crash situations the tests found on your two servers.
Quoting @andyhhp about what the crash message means: "The guest has tried to turn on LBR, but I don't know what that means on this CPU" where LBR stands for Last Branch Record and is used for debugging and in anti-cheat software in video games.
yum clean metadata --enablerepo=xcp-ng-testing yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing rebootIf I understood correctly, the patches add support for more recent CPUs (Sapphire Rapids) and at the same time make it so that now when guests attempt to turn LBR on, this does nothing (but at least doesn't crash).
-
NUC 11 (i5-1135G7) running
xen-hypervisor.x86_64 0:4.13.4-10.36.0.sapphirerapids.1.xcpng8.3no longer gives the crash error. The test runs but it still generates a Xen message:# ./xtf-runner -aqq --host ; echo Result: $? Combined test results: test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP Result: 3 (XEN) [ 119.229055] d127v0 Bad rIP 800000000000 for mode 8 (XEN) [ 121.294326] d159v0 Hit #DB in Xen context: e008:ffff82d0bfffe080 [ffff82d0bfffe080], stk e010:8000000000112ff0, dr6 ffff0ff0 (XEN) [ 121.418332] d161v0 Hit #DB in Xen context: e008:ffff82d080369ba0 [int3], stk 0000:ffff830896a5ff78, dr6 ffff2ff0 -
@Andrew Those are normal.
Bad rIPis actually an error introduced in XSA-170 because someone misread the Intel manual. I've been trying to delete it upstream for years now. Its been so long that Intel nearly released a feature which would have required us to delete that check, and I successfully persuaded the Intel documentation team to add a footnote clarifying the statement which was misinterpreted during XSA-170.At some point in my copious free never, I should restart the argument to delete it upstream...
The other two are logging from the XSA-260 fix. There's an error(/misfeature) in the x86 architecture and those would have been privilege escalations before the fix was in place. I decided when fixing XSA-260 that such attempts shouldn't be entirely silent, hence the one-liner. That particular printk() is actually common with other debugging routines, so can occur during regular development.
