XCP-ng 8.0.0 Beta now available!

s_mcleod

Some great news with the following setup, my XCP-ng VM is passing spectre-meltdown-checker including the latest zombieload etc....

I have never seen any XenServer or XCP-ng VM pass all these tests before.

XCP-ng 8.0 beta
Xeon E5-2690v4
Hyperthreading Disabled (In BIOS)
VM: CentOS 7.6
VM: All updates installed
VM: Kernel 5.1.4-1 (elrepo kernel-ml)

spectre-meltdown-checker output:

[root@pm-samm-dev-01 ~]# ./spectre-meltdown-checker.sh
Spectre and Meltdown mitigation detection tool v0.41

Checking for vulnerabilities on current system
Kernel is Linux 5.1.4-1.el7.elrepo.x86_64 #1 SMP Wed May 22 08:12:02 EDT 2019 x86_64
CPU is Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates IBRS capability:  YES  (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available:  YES
    * CPU indicates IBPB capability:  YES  (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates STIBP capability:  YES  (Intel STIBP feature bit)
  * Speculative Store Bypass Disable (SSBD)
    * CPU indicates SSBD capability:  YES  (Intel SSBD)
  * L1 data cache invalidation
    * FLUSH_CMD MSR is available:  YES
    * CPU indicates L1D flush capability:  YES  (L1D flush feature bit)
  * Microarchitecture Data Sampling
    * VERW instruction is available:  YES  (MD_CLEAR feature bit)
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO):  NO
  * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO):  NO
  * CPU/Hypervisor indicates L1D flushing is not necessary on this system:  NO
  * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA):  NO
  * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDC_NO):  NO
  * CPU supports Software Guard Extensions (SGX):  NO
  * CPU microcode is known to cause stability problems:  NO  (model 0x4f family 0x6 stepping 0x1 ucode 0xb000036 cpuid 0x406f1)
  * CPU microcode is the latest known available version:  YES  (latest version is 0xb000036 dated 2019/03/02 according to builtin MCExtractor DB v110 - 2019/05/11)
* CPU vulnerability to the speculative execution attack variants
  * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass):  YES
  * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection):  YES
  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load):  YES
  * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read):  YES
  * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass):  YES
  * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault):  NO
  * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault):  YES
  * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault):  YES
  * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)):  YES
  * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)):  YES
  * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)):  YES
  * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory (MDSUM)):  YES

CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass'
* Mitigated according to the /sys interface:  YES  (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec:  UNKNOWN  (missing 'perl' binary, please install it)
* Kernel has the Red Hat/Ubuntu patch:  NO
* Kernel has mask_nospec64 (arm64):  UNKNOWN  (missing 'perl' binary, please install it)
* Checking count of LFENCE instructions following a jump in kernel...  NO  (only 5 jump-then-lfence instructions found, should be >= 30 (heuristic))
> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)

CVE-2017-5715 aka 'Spectre Variant 2, branch target injection'
* Mitigated according to the /sys interface:  YES  (Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: disabled, RSB filling)
* Mitigation 1
  * Kernel is compiled with IBRS support:  YES
    * IBRS enabled and active:  YES  (for firmware code only)
  * Kernel is compiled with IBPB support:  YES
    * IBPB enabled and active:  YES
* Mitigation 2
  * Kernel has branch predictor hardening (arm):  NO
  * Kernel compiled with retpoline option:  YES
    * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
> STATUS:  NOT VULNERABLE  (Full retpoline + IBPB are mitigating the vulnerability)

CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI):  YES
  * PTI enabled and active:  YES
  * Reduced performance impact of PTI:  YES  (CPU supports INVPCID, performance impact of PTI will be greatly reduced)
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

CVE-2018-3640 aka 'Variant 3a, rogue system register read'
* CPU microcode mitigates the vulnerability:  YES
> STATUS:  NOT VULNERABLE  (your CPU microcode mitigates the vulnerability)

CVE-2018-3639 aka 'Variant 4, speculative store bypass'
* Mitigated according to the /sys interface:  YES  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
* Kernel supports disabling speculative store bypass (SSB):  YES  (found in /proc/self/status)
* SSB mitigation is enabled and active:  YES  (per-thread through prctl)
* SSB mitigation currently active for selected processes:  NO  (no process found using SSB mitigation through prctl)
> STATUS:  NOT VULNERABLE  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)

CVE-2018-3615 aka 'Foreshadow (SGX), L1 terminal fault'
* CPU microcode mitigates the vulnerability:  N/A
> STATUS:  NOT VULNERABLE  (your CPU vendor reported your CPU model as not vulnerable)

CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTE Inversion)
* Kernel supports PTE inversion:  YES  (found in kernel image)
* PTE inversion enabled and active:  YES
> STATUS:  NOT VULNERABLE  (Mitigation: PTE Inversion)

CVE-2018-3646 aka 'Foreshadow-NG (VMM), L1 terminal fault'
* Information from the /sys interface:
* This system is a host running a hypervisor:  NO
* Mitigation 1 (KVM)
  * EPT is disabled:  N/A  (the kvm_intel module is not loaded)
* Mitigation 2
  * L1D flush is supported by kernel:  YES  (found flush_l1d in /proc/cpuinfo)
  * L1D flush enabled:  UNKNOWN  (unrecognized mode)
  * Hardware-backed L1D flush supported:  YES  (performance impact of the mitigation will be greatly reduced)
  * Hyper-Threading (SMT) is enabled:  NO
> STATUS:  NOT VULNERABLE  (this system is not running a hypervisor)

CVE-2018-12126 aka 'Fallout, microarchitectural store buffer data sampling (MSBDS)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

CVE-2018-12130 aka 'ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

CVE-2018-12127 aka 'RIDL, microarchitectural load port data sampling (MLPDS)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

CVE-2019-11091 aka 'RIDL, microarchitectural data sampling uncacheable memory (MDSUM)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

> SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK CVE-2018-12126:OK CVE-2018-12130:OK CVE-2018-12127:OK CVE-2019-11091:OK

s_mcleod

As per https://xcp-ng.org/forum/post/12028

It looks like vDSO (available since Xen 4.8 onwards and can greatly improve highly multithreaded VM workload performance):

Isn't enabled under XCP-ng 8.0 by default
When you enable it, it doesn't seem to work (see https://github.com/sammcj/vdso-test-results/blob/master/xeon/Xeon E5-2690-v4/XCP-ng-8.0-beta1-vDSO-tsc-2019-05-23-vm-kernel-5.1.4.el7.md)
The default clocksource for VMs is still xen although it's recommended to use tsc under Xen virtualisation these days. [1] [2]

[1] https://news.ycombinator.com/item?id=13813936
[2] https://aws.amazon.com/premiumsupport/knowledge-center/manage-ec2-linux-clock-source/

borzel

@s_mcleod issue with the next button is an upstream bug and will be fixed in XCP-ng Center 8.0.1

https://github.com/xcp-ng/xenadmin/issues/156

gorbi13G created this issue in xcp-ng/xenadmin

closed XCP-ng Center 8.0.0 (build 8.0.0.24) live/offline virtual migration/move #156

s_mcleod

@borzel good catch!

Thanks,

olivierlambert

So VM.suspend is failing right? I wonder if it's reproducible in CH 8.0, I think we saw the same thing here in XCP-ng.

I'll update our few hosts running XS 7.6 to CH 8.0 next week, so we'll be able to compare

cocoon

Thanks for your hard work!
Just did my first upgrade with the ISO from XCP-NG 7.6 (VirtualBox 6.08 VM). Upgrade went without a problem.

My ext4 datastore was detached after the upgrade and could not be attached, because the "sm-additional-drivers" are not installed by default (as was written in your WHAT'S NEW section).
After "yum install sm-additional-drivers" and a toolstack restart it was working again.
Also installing the container package with "yum install xscontainer" went without problems.

I did import a coreos.vhd with XCP-NG Center 8, converted the VM from HVM to PV with XCP-NG Center and I could start it.
I can confirm the problem with suspending the VM, I get the same error (xenopsd End_of_file).

apayne

It's a little strange to see older but capable CPUs dropped from the list. The old Opteron 2356 I have didn't make the cut, even though it works just fine in 7.6. I'm still going to try out 8.0 anyways.

I understand that vendors don't want to "extend support forever" but it's silly when you have 10+ year old hardware that runs fine, and the only limitation really comes down to "hardware feature XYZ is a requirement". But so far, I've not seen the actual minimum CPU requirement; which makes me a bit suspicious about "it won't run".

DustinB

@apayne said in XCP-ng 8.0.0 Beta now available!:

Opteron 2356

That's a 12 year old CPU so I don't see any issue with dropping it.

apayne

@DustinB said in XCP-ng 8.0.0 Beta now available!:

That's a 12 year old CPU so I don't see any issue with dropping it.

As I said, 10+ year old hardware; and I also said, I get that vendors want to draw lines in the sand so they don't end up supporting everything under the sun. It's good business sense to limit expenditures to equipment that is commonly used.

But my (poorly articulated from the last posting) point remains: there isn't a known or posted reason why the software forces me to drop the CPU. Citrix just waved their hands and said "these don't work anymore". Well, I suspect it really does work, and this is just the side-effect of a vendor cost-cutting decision for support that has nothing to do with XCP-ng, but unfortunately impacts it anyways. So it's worth a try, and if it fails, so be it - at least there will be a known reason why, instead of the Citrix response of "nothing to see here, move along..."

XCP-ng is a killer deal, probably THE killer deal when viewed through the lens of a home lab.
That makes it hard to justify shelling out money for Windows Hyper-V or VMWare when there is a family to feed and rent to pay. Maybe that explains why I am so keen on seeing if Citrix really did make changes that prevent it from running.

Fail or succeed, either way, it'll be more information to be contributed back to the community here, and something will be learned. That's a positive outcome all the way around.

olivierlambert

Regarding Citrix choice: I think they can't really publicly communicate about the reasons. It might be related to security issues and some CPU vendors not upgrading microcodes anymore.

That's why it should work technically speaking, but Citrix won't be liable for any security breach due to an "old" CPU.

Regarding XCP-ng: as long as it just works, and you don't pay for support, I think you'll be fine for your home lab

Please report if you have any issue with 8.0 on your old hardware, we'll try to help as far as we could.

cocoon

My experience with "XOA Quick Deploy"

It failed.

Because I need to use a proxy, I opened an issue here:
https://github.com/xcp-ng/xcp/issues/193

cocoon created this issue in xcp-ng/xcp

open XOA Quick Deploy with Proxy #193

AllooTikeeChaat

Installed fine on one of my Intel X56XX based system and the system boots into XCP 8.0 although the zfs local storage wasn't mounted and a repair fails as well.

modprobe zfs fails with "module zfs not found"

Is there any updated ZFS documentation for 8.0 that would help?

stormi

zfs is not part of the default installation, so you need to install it manually using yum from our repositories. Documentation on the wiki has not been updated yet for 8.0: https://github.com/xcp-ng/xcp/wiki/ZFS-on-XCP-ng
Now the zfs packages are directly in our main repositories, no need to add extra --enablerepo options. Just yum install zfs.

I've just built an updated zfs package (new major version 0.8.0 instead of the 0.7.3 that was initially available in the repos for the beta) in the hope that it solves some of the issues we had with previous versions (such as VDI export or the need to patch some packages to remove the use of O_DIRECT). I'm just waiting for the main mirror to sync to test that it installs fine.

AllooTikeeChaat

@stormi Thanks for the quick reply .. tried the yum install zfs and it errors with the following:

stormi

That's why I told that I'm waiting for the main mirror to sync
And the build machine is having a "I'm feeling all slow" moment.

AllooTikeeChaat

@stormi Ok..... silly me .. will wait for you to give the aok.

stormi

@AllooTikeeChaat You can now try.

AllooTikeeChaat

@stormi still broken .. same error

stormi

@AllooTikeeChaat yum considers that your medata are recent enough. Ask it to clean them: yum clean all.

AllooTikeeChaat

@stormi yum clean all worked. Installed aok and modprobe zfs doesn't complain so that's all good.

A repair of the ZFS SR fails with the same error and "zpool status: no pools available" and "zfs list no datasets available".

I'll open a new post if I can't get it working rather than replying to this one.

Update: fixed the issue ..