XCP-ng 8.3 public alpha 🚀
-
@Andrew I pushed a test build of Xen in the testing repository for XCP-ng 8.3, which will hopefully solve the crash situations the tests found on your two servers.
Quoting @andyhhp about what the crash message means: "The guest has tried to turn on LBR, but I don't know what that means on this CPU" where LBR stands for Last Branch Record and is used for debugging and in anti-cheat software in video games.
yum clean metadata --enablerepo=xcp-ng-testing yum update xen-dom0-libs xen-dom0-tools xen-hypervisor xen-libs xen-tools --enablerepo=xcp-ng-testing rebootIf I understood correctly, the patches add support for more recent CPUs (Sapphire Rapids) and at the same time make it so that now when guests attempt to turn LBR on, this does nothing (but at least doesn't crash).
-
NUC 11 (i5-1135G7) running
xen-hypervisor.x86_64 0:4.13.4-10.36.0.sapphirerapids.1.xcpng8.3no longer gives the crash error. The test runs but it still generates a Xen message:# ./xtf-runner -aqq --host ; echo Result: $? Combined test results: test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP Result: 3 (XEN) [ 119.229055] d127v0 Bad rIP 800000000000 for mode 8 (XEN) [ 121.294326] d159v0 Hit #DB in Xen context: e008:ffff82d0bfffe080 [ffff82d0bfffe080], stk e010:8000000000112ff0, dr6 ffff0ff0 (XEN) [ 121.418332] d161v0 Hit #DB in Xen context: e008:ffff82d080369ba0 [int3], stk 0000:ffff830896a5ff78, dr6 ffff2ff0 -
@Andrew Those are normal.
Bad rIPis actually an error introduced in XSA-170 because someone misread the Intel manual. I've been trying to delete it upstream for years now. Its been so long that Intel nearly released a feature which would have required us to delete that check, and I successfully persuaded the Intel documentation team to add a footnote clarifying the statement which was misinterpreted during XSA-170.At some point in my copious free never, I should restart the argument to delete it upstream...
The other two are logging from the XSA-260 fix. There's an error(/misfeature) in the x86 architecture and those would have been privilege escalations before the fix was in place. I decided when fixing XSA-260 that such attempts shouldn't be entirely silent, hence the one-liner. That particular printk() is actually common with other debugging routines, so can occur during regular development.
-
Your Xen guru badge is well earned @andyhhp
-
@olivierlambert said in XCP-ng 8.3 public alpha
:Your Xen guru badge is well earned @andyhhp
"purveyor of general grumpiness"
-
Tried on my Intel NUC11ATKC4 with N5105 and 32GB of RAM.
I know it's an edge case, nevertheless here the results:
[13:17 xcp-ng-1 xtf]# ./xtf-runner selftest -q --host Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS[13:19 xcp-ng-1 xtf]# ./xtf-runner -aq --host Combined test results: test-hvm32-cpuid-faulting SUCCESS test-hvm32pae-cpuid-faulting SUCCESS test-hvm32pse-cpuid-faulting SUCCESS test-hvm64-cpuid-faulting SUCCESS test-pv64-cpuid-faulting SUCCESS test-hvm64-fpu-exception-emulation SUCCESS test-hvm32-invlpg~hap SUCCESS test-hvm32-invlpg~shadow SUCCESS test-hvm32pae-invlpg~hap SUCCESS test-hvm32pae-invlpg~shadow SUCCESS test-hvm64-invlpg~hap SUCCESS test-hvm64-invlpg~shadow SUCCESS test-hvm64-lbr-tsx-vmentry CRASH test-hvm32-livepatch-priv-check SUCCESS test-hvm64-livepatch-priv-check SUCCESS test-pv64-livepatch-priv-check SUCCESS test-hvm32pae-memop-seg SUCCESS test-hvm64-memop-seg SUCCESS test-pv64-memop-seg SUCCESS test-hvm32pae-nmi-taskswitch-priv SUCCESS test-pv64-pv-fsgsbase SUCCESS test-pv64-pv-iopl~hypercall SUCCESS test-pv64-pv-iopl~vmassist SUCCESS test-hvm32-swint-emulation SUCCESS test-hvm32pae-swint-emulation SUCCESS test-hvm32pse-swint-emulation SUCCESS test-hvm64-swint-emulation SUCCESS test-hvm32-umip SUCCESS test-hvm64-umip SUCCESS test-hvm32-xsa-122 SUCCESS test-hvm32pae-xsa-122 SUCCESS test-hvm32pse-xsa-122 SUCCESS test-hvm64-xsa-122 SUCCESS test-pv64-xsa-122 SUCCESS test-hvm32-xsa-123 SUCCESS test-pv64-xsa-167 SKIP test-hvm64-xsa-168 SUCCESS test-hvm64-xsa-170 SUCCESS test-hvm64-xsa-173 SUCCESS test-pv64-xsa-182 SKIP test-hvm32-xsa-186 SUCCESS test-hvm64-xsa-186 SUCCESS test-hvm32-xsa-188 SUCCESS test-hvm32pae-xsa-188 SUCCESS test-hvm32pse-xsa-188 SUCCESS test-hvm64-xsa-188 SUCCESS test-pv64-xsa-188 SUCCESS test-hvm32-xsa-191 SUCCESS test-hvm32-xsa-192 SUCCESS test-pv64-xsa-193 SUCCESS test-hvm64-xsa-195 SUCCESS test-hvm64-xsa-196 SUCCESS test-hvm32-xsa-200 SUCCESS test-hvm32-xsa-203 SUCCESS test-hvm64-xsa-204 SUCCESS test-pv64-xsa-212 SUCCESS test-pv64-xsa-213 SUCCESS test-hvm64-xsa-221 SUCCESS test-pv64-xsa-221 SUCCESS test-pv64-xsa-224 SUCCESS test-pv64-xsa-227 SUCCESS test-hvm64-xsa-231 SUCCESS test-pv64-xsa-231 SUCCESS test-hvm64-xsa-232 SUCCESS test-pv64-xsa-232 SUCCESS test-pv64-xsa-234 SUCCESS test-hvm32-xsa-239 SUCCESS test-pv64-xsa-255 SUCCESS test-pv64-xsa-259 SUCCESS test-pv64-xsa-260 SUCCESS test-hvm64-xsa-261 SUCCESS test-pv64-xsa-265 SUCCESS test-hvm64-xsa-269 SUCCESS test-hvm64-xsa-277 SUCCESS test-hvm64-xsa-278 SUCCESS test-pv64-xsa-279 SUCCESS test-pv64-xsa-286 SUCCESS test-pv64-xsa-296 SUCCESS test-pv64-xsa-298 SUCCESS test-hvm64-xsa-308 SUCCESS test-pv64-xsa-316 SUCCESS test-hvm32-xsa-317 SUCCESS test-hvm32pae-xsa-317 SUCCESS test-hvm32pse-xsa-317 SUCCESS test-hvm64-xsa-317 SUCCESS test-pv64-xsa-317 SUCCESS test-pv64-xsa-333 SUCCESS test-pv64-xsa-339 SUCCESS test-hvm32pae-xsa-consoleio-write SUCCESS[13:21 xcp-ng-1 xtf]# /usr/libexec/xen/bin/test-cpu-policy CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok[13:21 xcp-ng-1 xtf]# ./xtf-runner -aqq --host ; echo Result: $? Combined test results: test-hvm64-lbr-tsx-vmentry CRASH test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP Result: 6 -
@rRobbie Thanks. This look like the same crash as discussed just above, with a candidate for a fix provided, if you want to test it. It should be included in the next update of the Xen packages anyway.
-
@stormi Here's results from one of my lab servers. I'll run on some newer hardware next week.
Asus KGPE-D16, Opteron 6328, 128GB ram.
Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-pv64-cpuid-faulting SKIP test-pv64-pv-fsgsbase SKIP test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 3CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 -
@ravenet said in XCP-ng 8.3 public alpha
:Opteron 6328
Wow! It's been a while I didn't heard about those ones
11 years old CPU, and still usable, impressive -
Here's two more old servers that I have still running:
Supermicro H8DGU AMD Opteron 6380 CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok Result 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-pv64-cpuid-faulting SKIP test-pv64-pv-fsgsbase SKIP test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP Result 3HP DL360 G5 Intel E5450 CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok Result 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Executing 'xl create -p tests/invlpg/test-hvm32-invlpg~hap.cfg' Parsing config from tests/invlpg/test-hvm32-invlpg~hap.cfg libxl: error: libxl_create.c:604:libxl__domain_make: domain creation fail: Invalid argument libxl: error: libxl_create.c:1102:initiate_domain_create: cannot make domain: -3 Error: Failed to create VM Result 1I guess the old G5 is getting a little too old to keep running... but it does still work.
-
@stormi
Here's a Ryzen 1700 on Asus TUF B550Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-pv64-cpuid-faulting SKIP test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP 3CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok 0 -
@Andrew Intel E5450, that's very retro.
It's also first-gen VT-x and doesn't have HAP, which is why the test that is looking explicitly for HAP doesn't work.
As a stopgap, remove
hapfrom theVARY-CFG := hap shadowline intests/invlpg/Makefileand rebuild. In the meantime I'll try to figure out a nice way to cope with this. -
@stormi @andyhhp Yes, it is very retro.... it was my first XCP-ng host (because it was free).
HP DL360 G5 Intel E5450 HAP test disabled CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok Result 0 Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS Combined test results: test-pv64-cpuid-faulting SKIP test-pv64-pv-fsgsbase SKIP test-hvm32-umip SKIP test-hvm64-umip SKIP test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP Result 3 -
@stormi Hello.
Just installed the latest patches to the XCP-NG host and did the test again.
Not sure the fix was already included in the packages... So just for your info
Intel NUC11ATKC4 with N5105 and 32GB of RAM
[21:55 xcp-ng-1 xtf]# yum upgrade Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org No packages marked for update [21:55 xcp-ng-1 xtf]# ./xtf-runner -aqq --host ; echo Result: $? Combined test results: test-hvm64-lbr-tsx-vmentry CRASH test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP Result: 6 -
@rRobbie It's not included yet.
-
So, a few days ago, we pushed the following updates to XCP-ng 8.3:
guest-templates-json-*: adds Ubuntu Jammy template.- XAPI updated to version 22.34. Among other things:
- the host's webpage, when accessed on port 80, now automatically redirects to HTTPS on port 443 rather than displaying a 403 Forbidden error. Note that in a future update, port 80 may be completely closed instead.
- I haven't checked yet, but I think this removes the immortal pending
xapi eventstask that a previous update added, while keeping the perk of the previous change, which removed a lot of spurious "INVALID_SESSION" errors from xensource.log (almost anyone having had a look at the logs has seen them).
xcp-featured: was simply rebuilt for the updated XAPImicrosemi-smartpqivendor driver updated to2.1.20_035sudopatched for security issue
No need to re-run the Xen tests if you ran them previously, as Xen wasn't updated in this round of updates. If you haven't yet, then the "let's cover the largest range of hardware possible" action continues.
-
A test was recently pushed to XTF's repository which may crash a host whose hardware is vulnerable to XSA-304 (hardware vulnerability related to EPT superpages).
There are two modes in Xen regarding EPT superpages, fast vs secure, and XCP-ng defaults to fast due to the big performance hit secure would have by default on affected hardware. The vulnerability may allow an HVM guest to crash the host, and this is what the test demonstrates.
I've updated the small Xen testing guide above with instructions to switch to secure mode before running the tests, if you want to avoid the crash in case your host is vulnerable.
-
Here's the results on my new Intel NUC11 Pool (tested on one of the two identical NUCs)
Host: Intel Nuc11TNK
CPU: i5-1135G7
RAM: 64GB G.SKILL Ripjaws F4-3200C22D-64GRS
SSD: Samsung 980 1TB
NIC: Intel i225 running at 2500 mbit
UEFI BIOS: TNTGL357.0064.2022.0217.1550[00:51 xcpng33 xtf]# ./xtf-runner -aqq --host ; echo Result: $? Combined test results: test-hvm64-lbr-tsx-vmentry CRASH test-pv64-xsa-167 SKIP test-pv64-xsa-182 SKIP Result: 6 [00:53 xcpng33 xtf]# ./xtf-runner selftest -q --host Combined test results: test-hvm32-selftest SUCCESS test-hvm32pae-selftest SUCCESS test-hvm32pse-selftest SUCCESS test-hvm64-selftest SUCCESS test-pv64-selftest SUCCESS [00:53 xcpng33 xtf]# ./xtf-runner -aq --host Combined test results: test-hvm32-cpuid-faulting SUCCESS test-hvm32pae-cpuid-faulting SUCCESS test-hvm32pse-cpuid-faulting SUCCESS test-hvm64-cpuid-faulting SUCCESS test-pv64-cpuid-faulting SUCCESS test-hvm64-fpu-exception-emulation SUCCESS test-hvm32-invlpg~hap SUCCESS test-hvm32-invlpg~shadow SUCCESS test-hvm32pae-invlpg~hap SUCCESS test-hvm32pae-invlpg~shadow SUCCESS test-hvm64-invlpg~hap SUCCESS test-hvm64-invlpg~shadow SUCCESS test-hvm64-lbr-tsx-vmentry CRASH test-hvm32-livepatch-priv-check SUCCESS test-hvm64-livepatch-priv-check SUCCESS test-pv64-livepatch-priv-check SUCCESS test-hvm32pae-memop-seg SUCCESS test-hvm64-memop-seg SUCCESS test-pv64-memop-seg SUCCESS test-hvm32pae-nmi-taskswitch-priv SUCCESS test-pv64-pv-fsgsbase SUCCESS test-pv64-pv-iopl~hypercall SUCCESS test-pv64-pv-iopl~vmassist SUCCESS test-hvm32-swint-emulation SUCCESS test-hvm32pae-swint-emulation SUCCESS test-hvm32pse-swint-emulation SUCCESS test-hvm64-swint-emulation SUCCESS test-hvm32-umip SUCCESS test-hvm64-umip SUCCESS test-hvm32-xsa-122 SUCCESS test-hvm32pae-xsa-122 SUCCESS test-hvm32pse-xsa-122 SUCCESS test-hvm64-xsa-122 SUCCESS test-pv64-xsa-122 SUCCESS test-hvm32-xsa-123 SUCCESS test-pv64-xsa-167 SKIP test-hvm64-xsa-168 SUCCESS test-hvm64-xsa-170 SUCCESS test-hvm64-xsa-173 SUCCESS test-pv64-xsa-182 SKIP test-hvm32-xsa-186 SUCCESS test-hvm64-xsa-186 SUCCESS test-hvm32-xsa-188 SUCCESS test-hvm32pae-xsa-188 SUCCESS test-hvm32pse-xsa-188 SUCCESS test-hvm64-xsa-188 SUCCESS test-pv64-xsa-188 SUCCESS test-hvm32-xsa-191 SUCCESS test-hvm32-xsa-192 SUCCESS test-pv64-xsa-193 SUCCESS test-hvm64-xsa-195 SUCCESS test-hvm64-xsa-196 SUCCESS test-hvm32-xsa-200 SUCCESS test-hvm32-xsa-203 SUCCESS test-hvm64-xsa-204 SUCCESS test-pv64-xsa-212 SUCCESS test-pv64-xsa-213 SUCCESS test-hvm64-xsa-221 SUCCESS test-pv64-xsa-221 SUCCESS test-pv64-xsa-224 SUCCESS test-pv64-xsa-227 SUCCESS test-hvm64-xsa-231 SUCCESS test-pv64-xsa-231 SUCCESS test-hvm64-xsa-232 SUCCESS test-pv64-xsa-232 SUCCESS test-pv64-xsa-234 SUCCESS test-hvm32-xsa-239 SUCCESS test-pv64-xsa-255 SUCCESS test-pv64-xsa-259 SUCCESS test-pv64-xsa-260 SUCCESS test-hvm64-xsa-261 SUCCESS test-pv64-xsa-265 SUCCESS test-hvm64-xsa-269 SUCCESS test-hvm64-xsa-277 SUCCESS test-hvm64-xsa-278 SUCCESS test-pv64-xsa-279 SUCCESS test-pv64-xsa-286 SUCCESS test-pv64-xsa-296 SUCCESS test-pv64-xsa-298 SUCCESS test-hvm64-xsa-304 SUCCESS test-hvm64-xsa-308 SUCCESS test-pv64-xsa-316 SUCCESS test-hvm32-xsa-317 SUCCESS test-hvm32pae-xsa-317 SUCCESS test-hvm32pse-xsa-317 SUCCESS test-hvm64-xsa-317 SUCCESS test-pv64-xsa-317 SUCCESS test-pv64-xsa-333 SUCCESS test-pv64-xsa-339 SUCCESS test-hvm32pae-xsa-consoleio-write SUCCESS [00:54 xcpng33 xtf]# /usr/libexec/xen/bin/test-cpu-policy CPU Policy unit tests Testing CPU vendor identification: Testing CPUID serialise success: Testing CPUID deserialise failure: Testing CPUID out-of-range clearing: Testing MSR serialise success: Testing MSR deserialise failure: Testing policy compatibility success: Testing policy compatibility failure: Done: all ok -
@Strebor Thanks! You likely have the same crash as what was discussed earlier in the thread (same test failing), with instructions to install a patched Xen to see if it fixes it for you.
-
@stormi You're all welcome! Thanks so much for making this possible!
I will install the patch next week when I'm back in the office and send the updated results.
