Posts made by s_mcleod

s_mcleod

I’ve fairly certain I’ve had this problem since xenserver 6.x

s_mcleod

@olivierlambert I’m sorry I’m not more available to bring more testing to the table this week, I am AFK.

s_mcleod

@MajorTom Sorry about that I've been (and still am AFK):

Corrected link: https://github.com/sammcj/benchmark_results/blob/master/xcpng/8/hyperthreading_impact/test_2_sysbench_prime.md

s_mcleod

Just FYI - I have performed CPU and PGBench benchmarks on XCP-ng 8 beta 1, both with Hyperthreading enabled and disabled when running two identical VMs under different types of low, medium and heavy CPU load.

Results are available here: https://github.com/sammcj/benchmark_results/tree/master/xcpng/8/hyperthreading_impact

TLDR;

Significant performance decrease (38.7725%) when running multithreaded Sysbench CPU benchmarks in parallel on two VMs when hyperthreading is disabled.
Significant performance decrease (16.96%) when running PGBench under 'normal' load benchmarks in parallel on two VMs when hyperthreading is disabled.
No significant performance decrease when running Phoronix Test Suite's Pybench and OpenSSL benchmarks in parallel on two VMs when hyperthreading is disabled.

s_mcleod

@borzel good catch!

Thanks,

s_mcleod

As per https://xcp-ng.org/forum/post/12028

It looks like vDSO (available since Xen 4.8 onwards and can greatly improve highly multithreaded VM workload performance):

Isn't enabled under XCP-ng 8.0 by default
When you enable it, it doesn't seem to work (see https://github.com/sammcj/vdso-test-results/blob/master/xeon/Xeon E5-2690-v4/XCP-ng-8.0-beta1-vDSO-tsc-2019-05-23-vm-kernel-5.1.4.el7.md)
The default clocksource for VMs is still xen although it's recommended to use tsc under Xen virtualisation these days. [1] [2]

[1] https://news.ycombinator.com/item?id=13813936
[2] https://aws.amazon.com/premiumsupport/knowledge-center/manage-ec2-linux-clock-source/

s_mcleod

Some great news with the following setup, my XCP-ng VM is passing spectre-meltdown-checker including the latest zombieload etc....

I have never seen any XenServer or XCP-ng VM pass all these tests before.

XCP-ng 8.0 beta
Xeon E5-2690v4
Hyperthreading Disabled (In BIOS)
VM: CentOS 7.6
VM: All updates installed
VM: Kernel 5.1.4-1 (elrepo kernel-ml)

spectre-meltdown-checker output:

[root@pm-samm-dev-01 ~]# ./spectre-meltdown-checker.sh
Spectre and Meltdown mitigation detection tool v0.41

Checking for vulnerabilities on current system
Kernel is Linux 5.1.4-1.el7.elrepo.x86_64 #1 SMP Wed May 22 08:12:02 EDT 2019 x86_64
CPU is Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates IBRS capability:  YES  (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available:  YES
    * CPU indicates IBPB capability:  YES  (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates STIBP capability:  YES  (Intel STIBP feature bit)
  * Speculative Store Bypass Disable (SSBD)
    * CPU indicates SSBD capability:  YES  (Intel SSBD)
  * L1 data cache invalidation
    * FLUSH_CMD MSR is available:  YES
    * CPU indicates L1D flush capability:  YES  (L1D flush feature bit)
  * Microarchitecture Data Sampling
    * VERW instruction is available:  YES  (MD_CLEAR feature bit)
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO):  NO
  * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO):  NO
  * CPU/Hypervisor indicates L1D flushing is not necessary on this system:  NO
  * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA):  NO
  * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDC_NO):  NO
  * CPU supports Software Guard Extensions (SGX):  NO
  * CPU microcode is known to cause stability problems:  NO  (model 0x4f family 0x6 stepping 0x1 ucode 0xb000036 cpuid 0x406f1)
  * CPU microcode is the latest known available version:  YES  (latest version is 0xb000036 dated 2019/03/02 according to builtin MCExtractor DB v110 - 2019/05/11)
* CPU vulnerability to the speculative execution attack variants
  * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass):  YES
  * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection):  YES
  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load):  YES
  * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read):  YES
  * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass):  YES
  * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault):  NO
  * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault):  YES
  * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault):  YES
  * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)):  YES
  * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)):  YES
  * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)):  YES
  * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory (MDSUM)):  YES

CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass'
* Mitigated according to the /sys interface:  YES  (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec:  UNKNOWN  (missing 'perl' binary, please install it)
* Kernel has the Red Hat/Ubuntu patch:  NO
* Kernel has mask_nospec64 (arm64):  UNKNOWN  (missing 'perl' binary, please install it)
* Checking count of LFENCE instructions following a jump in kernel...  NO  (only 5 jump-then-lfence instructions found, should be >= 30 (heuristic))
> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)

CVE-2017-5715 aka 'Spectre Variant 2, branch target injection'
* Mitigated according to the /sys interface:  YES  (Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: disabled, RSB filling)
* Mitigation 1
  * Kernel is compiled with IBRS support:  YES
    * IBRS enabled and active:  YES  (for firmware code only)
  * Kernel is compiled with IBPB support:  YES
    * IBPB enabled and active:  YES
* Mitigation 2
  * Kernel has branch predictor hardening (arm):  NO
  * Kernel compiled with retpoline option:  YES
    * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
> STATUS:  NOT VULNERABLE  (Full retpoline + IBPB are mitigating the vulnerability)

CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI):  YES
  * PTI enabled and active:  YES
  * Reduced performance impact of PTI:  YES  (CPU supports INVPCID, performance impact of PTI will be greatly reduced)
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

CVE-2018-3640 aka 'Variant 3a, rogue system register read'
* CPU microcode mitigates the vulnerability:  YES
> STATUS:  NOT VULNERABLE  (your CPU microcode mitigates the vulnerability)

CVE-2018-3639 aka 'Variant 4, speculative store bypass'
* Mitigated according to the /sys interface:  YES  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
* Kernel supports disabling speculative store bypass (SSB):  YES  (found in /proc/self/status)
* SSB mitigation is enabled and active:  YES  (per-thread through prctl)
* SSB mitigation currently active for selected processes:  NO  (no process found using SSB mitigation through prctl)
> STATUS:  NOT VULNERABLE  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)

CVE-2018-3615 aka 'Foreshadow (SGX), L1 terminal fault'
* CPU microcode mitigates the vulnerability:  N/A
> STATUS:  NOT VULNERABLE  (your CPU vendor reported your CPU model as not vulnerable)

CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTE Inversion)
* Kernel supports PTE inversion:  YES  (found in kernel image)
* PTE inversion enabled and active:  YES
> STATUS:  NOT VULNERABLE  (Mitigation: PTE Inversion)

CVE-2018-3646 aka 'Foreshadow-NG (VMM), L1 terminal fault'
* Information from the /sys interface:
* This system is a host running a hypervisor:  NO
* Mitigation 1 (KVM)
  * EPT is disabled:  N/A  (the kvm_intel module is not loaded)
* Mitigation 2
  * L1D flush is supported by kernel:  YES  (found flush_l1d in /proc/cpuinfo)
  * L1D flush enabled:  UNKNOWN  (unrecognized mode)
  * Hardware-backed L1D flush supported:  YES  (performance impact of the mitigation will be greatly reduced)
  * Hyper-Threading (SMT) is enabled:  NO
> STATUS:  NOT VULNERABLE  (this system is not running a hypervisor)

CVE-2018-12126 aka 'Fallout, microarchitectural store buffer data sampling (MSBDS)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

CVE-2018-12130 aka 'ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

CVE-2018-12127 aka 'RIDL, microarchitectural load port data sampling (MLPDS)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

CVE-2019-11091 aka 'RIDL, microarchitectural data sampling uncacheable memory (MDSUM)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

> SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK CVE-2018-12126:OK CVE-2018-12130:OK CVE-2018-12127:OK CVE-2019-11091:OK

s_mcleod

I'll do whatever testing I can with the limited resources I / my team has at the moment.

I'll update this comment with any tests I perform.

Hardware

HPE BL460c Gen9, Xeon E5-2690v4, 256GB, Latest firmware (22/05/19), HT disabled
Local storage: 2x Crucial MX300 SSDs in hardware RAID 1
Network cards: Dual port HPE 560M, Dual port HPE 560FLB (both Intel 82599)

Software

XCP-ng Centre 8.0 (stable) was used throughout testing.
Client VM is CentOS 7.6

Test 1: ISO upgrade from fresh XCP-ng 7.6 install

Fresh Install XCP-ng 7.6.
Configure only the management interface, just left it as DHCP.
Install XCP-ng 7.6 updates and reboot.
Reboot off XCP-ng 8 Beta ISO and select upgrade from 7.6.
Upgrade completed, rebooted server.
Connected with XCP-ng Centre.
Tried to copy VM cross-pool from XCP-ng 7.6 to the new standalone XCP-ng 8 beta host
- Looked like it was going to work, but wanted to setup network bonding and VLANs first so cancelled.
Created a 2x 10Gbit, 20Gbit LACP bond on the non-management NICs. (~~Failed Test~~)
- After 1-3 minutes the bond showed up, but in a disconnected stated..
- Waited 10 minutes, no change, nothing in xe-task list.
- Rebooted host.
- Tried deleting and re-creating the bond.
- Bond still showing as disconnected: (~~Failed Test~~)
- ```
  # ovs-appctl bond/show bond0
      ---- bond0 ----
  bond_mode: balance-slb
  bond may use recirculation: no, Recirc-ID : -1
  bond-hash-basis: 0
  updelay: 31000 ms
  downdelay: 200 ms
  next rebalance: 6113 ms
  lacp_status: negotiated
  active slave mac: 00:00:00:00:00:00(none)

  slave eth0: disabled
    may_enable: false

  slave eth1: disabled
    may_enable: false
```
- Switches (Juniper EX4550) show:
  - Physical links: up
  - Bond: down (this should be up)
  - Received state: defaulted (this should be current), this means that the switches "did not receive an aggravation control PDU within the LACP timeout * 2".
  - MUC state: detached (should be collecting distributing).
- Note: This is the same config as our 7.6 pools as it was with XenServer as pools as well.
- Completely powered off server, then powered on again - this strangely enough Fixed the issue!
- Leaving for now, continuing other testing.
Created a 2x 10Gbit, Active/Passive bond on the management NICs.
- Bond created and connected within 30 seconds.
Tried to copy VM cross-pool from XCP-ng 7.6 to the new standalone XCP-ng 8 beta host (Failed Test)
- The Next button was greyed out this time, but I couldn't see any issues. (Failed Test)
- Closed and re-opened XCP-ng Centre, tried again - this time Next was available when trying to copy cross-pool, but I selected cancel to test again.
- This time - the Next button was greyed out again, but I couldn't see any issues. (Failed Test)
- Leaving for now, continuing other testing.
Added a SMB based ISO SR.
Created a CentOS 7 VM, installed from the latest CentOS 7 ISO on the ISO SR to local storage.
VM booted, installed xen tools, rebooted all fine.
No networking in the CentOS 7 VM due to the XCP-ng bond0 not functioning, so could not test in-VM functions any further.
Suspended VM failed. (Failed Test)
Start VM successfully.
Snapshot VM successfully.
Delete snapshot successfully.
Shutdown VM successfully.

From https://github.com/xcp-ng/xcp/wiki/Test-XCP

Text XCP - Results

Verify installation.
Verify connectivity with your interfaces - Pass, but required server to be powered off and on again rather than a reboot to work, will test again.
Verify connectivity to Shared Storages. - Cannot test, at test location there is no spare SAN LUN I can use at present.
Verify creation a new Linux VM (install guest tools).
~~Verify creation a new Windows VM (install guest tools)~~ - We don't run Windows
Verify basic VM functionality (start, reboot, suspend, shutdown)
Verify migration of a VM from an host to another. - Cannot test, at test location there is no second XCP-ng 8.0 host available at present.
Verify migration of a VM from an old host to (this) release one -
FAIL
Verify migration of a VM from a newest host to the old one (this test should be fail).
Check your logs for uncommon info or warnings. - Yet to test.

Performance Test

Compare speed of write/read of disks in the old and in the new release - Yet to test.
Compare speed of interfaces in the old and in the new release - Yet to test.

s_mcleod

Howdy all,

I'm Sam McLeod, an engineer and long time DevOps proponent based in Melbourne, Australia.
I work with an Australian Non-Profit and Charitable Organisation called Infoxchange.

I'm currently moving us from XenServer 7.2 to XenServer 7.6.

I was (still am) on pretty good terms with several people at Citrix's XenServer teams but I was pretty disgusted by the change in free licensing with the release of 7.3 and I'm also the Sam McLeod replying to David here: https://xenserver.org/blog/entry/xenserver-7-3-changes-to-the-free-edition.html

Twitter: https://twitter.com/s_mcleod
Website: https://smcleod.net
GitLab: https://gitlab.com/sammcj
GitHub: https://github.com/sammcj
HN: https://news.ycombinator.com/user?id=mrmondo
Goodreads: https://www.goodreads.com/user/show/5836466-sam
LinkedIN: https://www.linkedin.com/in/sammcj/