XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    1. Home
    2. s_mcleod
    3. Posts
    • Profile
    • Following 0
    • Followers 0
    • Topics 0
    • Posts 9
    • Groups 0

    Posts

    Recent Best Controversial
    • RE: XCP-ng 8.0.0 Beta now available!

      I’ve fairly certain I’ve had this problem since xenserver 6.x

      posted in News
      s_mcleodS
      s_mcleod
    • RE: XCP-ng 8.0.0 Beta now available!

      @olivierlambert I’m sorry I’m not more available to bring more testing to the table this week, I am AFK.

      posted in News
      s_mcleodS
      s_mcleod
    • RE: XCP-ng 8.0.0 Beta now available!

      @MajorTom Sorry about that I've been (and still am AFK):

      Corrected link: https://github.com/sammcj/benchmark_results/blob/master/xcpng/8/hyperthreading_impact/test_2_sysbench_prime.md

      posted in News
      s_mcleodS
      s_mcleod
    • RE: XCP-ng 8.0.0 Beta now available!

      Just FYI - I have performed CPU and PGBench benchmarks on XCP-ng 8 beta 1, both with Hyperthreading enabled and disabled when running two identical VMs under different types of low, medium and heavy CPU load.

      Results are available here: https://github.com/sammcj/benchmark_results/tree/master/xcpng/8/hyperthreading_impact

      TLDR;

      • Significant performance decrease (38.7725%) when running multithreaded Sysbench CPU benchmarks in parallel on two VMs when hyperthreading is disabled.

      • Significant performance decrease (16.96%) when running PGBench under 'normal' load benchmarks in parallel on two VMs when hyperthreading is disabled.

      • No significant performance decrease when running Phoronix Test Suite's Pybench and OpenSSL benchmarks in parallel on two VMs when hyperthreading is disabled.

      posted in News
      s_mcleodS
      s_mcleod
    • RE: XCP-ng 8.0.0 Beta now available!

      @borzel good catch!

      Thanks,

      posted in News
      s_mcleodS
      s_mcleod
    • RE: XCP-ng 8.0.0 Beta now available!

      As per https://xcp-ng.org/forum/post/12028

      It looks like vDSO (available since Xen 4.8 onwards and can greatly improve highly multithreaded VM workload performance):

      1. Isn't enabled under XCP-ng 8.0 by default
      2. When you enable it, it doesn't seem to work (see https://github.com/sammcj/vdso-test-results/blob/master/xeon/Xeon E5-2690-v4/XCP-ng-8.0-beta1-vDSO-tsc-2019-05-23-vm-kernel-5.1.4.el7.md)
      3. The default clocksource for VMs is still xen although it's recommended to use tsc under Xen virtualisation these days. [1] [2]

      [1] https://news.ycombinator.com/item?id=13813936
      [2] https://aws.amazon.com/premiumsupport/knowledge-center/manage-ec2-linux-clock-source/

      posted in News
      s_mcleodS
      s_mcleod
    • RE: XCP-ng 8.0.0 Beta now available!

      Some great news with the following setup, my XCP-ng VM is passing spectre-meltdown-checker including the latest zombieload etc....

      I have never seen any XenServer or XCP-ng VM pass all these tests before.

      • XCP-ng 8.0 beta
      • Xeon E5-2690v4
      • Hyperthreading Disabled (In BIOS) 😭
      • VM: CentOS 7.6
      • VM: All updates installed
      • VM: Kernel 5.1.4-1 (elrepo kernel-ml)

      spectre-meltdown-checker output:

      [root@pm-samm-dev-01 ~]# ./spectre-meltdown-checker.sh
Spectre and Meltdown mitigation detection tool v0.41

Checking for vulnerabilities on current system
Kernel is Linux 5.1.4-1.el7.elrepo.x86_64 #1 SMP Wed May 22 08:12:02 EDT 2019 x86_64
CPU is Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz

Hardware check
* Hardware support (CPU microcode) for mitigation techniques
  * Indirect Branch Restricted Speculation (IBRS)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates IBRS capability:  YES  (SPEC_CTRL feature bit)
  * Indirect Branch Prediction Barrier (IBPB)
    * PRED_CMD MSR is available:  YES
    * CPU indicates IBPB capability:  YES  (SPEC_CTRL feature bit)
  * Single Thread Indirect Branch Predictors (STIBP)
    * SPEC_CTRL MSR is available:  YES
    * CPU indicates STIBP capability:  YES  (Intel STIBP feature bit)
  * Speculative Store Bypass Disable (SSBD)
    * CPU indicates SSBD capability:  YES  (Intel SSBD)
  * L1 data cache invalidation
    * FLUSH_CMD MSR is available:  YES
    * CPU indicates L1D flush capability:  YES  (L1D flush feature bit)
  * Microarchitecture Data Sampling
    * VERW instruction is available:  YES  (MD_CLEAR feature bit)
  * Enhanced IBRS (IBRS_ALL)
    * CPU indicates ARCH_CAPABILITIES MSR availability:  NO
    * ARCH_CAPABILITIES MSR advertises IBRS_ALL capability:  NO
  * CPU explicitly indicates not being vulnerable to Meltdown (RDCL_NO):  NO
  * CPU explicitly indicates not being vulnerable to Variant 4 (SSB_NO):  NO
  * CPU/Hypervisor indicates L1D flushing is not necessary on this system:  NO
  * Hypervisor indicates host CPU might be vulnerable to RSB underflow (RSBA):  NO
  * CPU explicitly indicates not being vulnerable to Microarchitectural Data Sampling (MDC_NO):  NO
  * CPU supports Software Guard Extensions (SGX):  NO
  * CPU microcode is known to cause stability problems:  NO  (model 0x4f family 0x6 stepping 0x1 ucode 0xb000036 cpuid 0x406f1)
  * CPU microcode is the latest known available version:  YES  (latest version is 0xb000036 dated 2019/03/02 according to builtin MCExtractor DB v110 - 2019/05/11)
* CPU vulnerability to the speculative execution attack variants
  * Vulnerable to CVE-2017-5753 (Spectre Variant 1, bounds check bypass):  YES
  * Vulnerable to CVE-2017-5715 (Spectre Variant 2, branch target injection):  YES
  * Vulnerable to CVE-2017-5754 (Variant 3, Meltdown, rogue data cache load):  YES
  * Vulnerable to CVE-2018-3640 (Variant 3a, rogue system register read):  YES
  * Vulnerable to CVE-2018-3639 (Variant 4, speculative store bypass):  YES
  * Vulnerable to CVE-2018-3615 (Foreshadow (SGX), L1 terminal fault):  NO
  * Vulnerable to CVE-2018-3620 (Foreshadow-NG (OS), L1 terminal fault):  YES
  * Vulnerable to CVE-2018-3646 (Foreshadow-NG (VMM), L1 terminal fault):  YES
  * Vulnerable to CVE-2018-12126 (Fallout, microarchitectural store buffer data sampling (MSBDS)):  YES
  * Vulnerable to CVE-2018-12130 (ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)):  YES
  * Vulnerable to CVE-2018-12127 (RIDL, microarchitectural load port data sampling (MLPDS)):  YES
  * Vulnerable to CVE-2019-11091 (RIDL, microarchitectural data sampling uncacheable memory (MDSUM)):  YES

CVE-2017-5753 aka 'Spectre Variant 1, bounds check bypass'
* Mitigated according to the /sys interface:  YES  (Mitigation: __user pointer sanitization)
* Kernel has array_index_mask_nospec:  UNKNOWN  (missing 'perl' binary, please install it)
* Kernel has the Red Hat/Ubuntu patch:  NO
* Kernel has mask_nospec64 (arm64):  UNKNOWN  (missing 'perl' binary, please install it)
* Checking count of LFENCE instructions following a jump in kernel...  NO  (only 5 jump-then-lfence instructions found, should be >= 30 (heuristic))
> STATUS:  NOT VULNERABLE  (Mitigation: __user pointer sanitization)

CVE-2017-5715 aka 'Spectre Variant 2, branch target injection'
* Mitigated according to the /sys interface:  YES  (Mitigation: Full generic retpoline, IBPB: conditional, IBRS_FW, STIBP: disabled, RSB filling)
* Mitigation 1
  * Kernel is compiled with IBRS support:  YES
    * IBRS enabled and active:  YES  (for firmware code only)
  * Kernel is compiled with IBPB support:  YES
    * IBPB enabled and active:  YES
* Mitigation 2
  * Kernel has branch predictor hardening (arm):  NO
  * Kernel compiled with retpoline option:  YES
    * Kernel compiled with a retpoline-aware compiler:  YES  (kernel reports full retpoline compilation)
> STATUS:  NOT VULNERABLE  (Full retpoline + IBPB are mitigating the vulnerability)

CVE-2017-5754 aka 'Variant 3, Meltdown, rogue data cache load'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTI)
* Kernel supports Page Table Isolation (PTI):  YES
  * PTI enabled and active:  YES
  * Reduced performance impact of PTI:  YES  (CPU supports INVPCID, performance impact of PTI will be greatly reduced)
* Running as a Xen PV DomU:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: PTI)

CVE-2018-3640 aka 'Variant 3a, rogue system register read'
* CPU microcode mitigates the vulnerability:  YES
> STATUS:  NOT VULNERABLE  (your CPU microcode mitigates the vulnerability)

CVE-2018-3639 aka 'Variant 4, speculative store bypass'
* Mitigated according to the /sys interface:  YES  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)
* Kernel supports disabling speculative store bypass (SSB):  YES  (found in /proc/self/status)
* SSB mitigation is enabled and active:  YES  (per-thread through prctl)
* SSB mitigation currently active for selected processes:  NO  (no process found using SSB mitigation through prctl)
> STATUS:  NOT VULNERABLE  (Mitigation: Speculative Store Bypass disabled via prctl and seccomp)

CVE-2018-3615 aka 'Foreshadow (SGX), L1 terminal fault'
* CPU microcode mitigates the vulnerability:  N/A
> STATUS:  NOT VULNERABLE  (your CPU vendor reported your CPU model as not vulnerable)

CVE-2018-3620 aka 'Foreshadow-NG (OS), L1 terminal fault'
* Mitigated according to the /sys interface:  YES  (Mitigation: PTE Inversion)
* Kernel supports PTE inversion:  YES  (found in kernel image)
* PTE inversion enabled and active:  YES
> STATUS:  NOT VULNERABLE  (Mitigation: PTE Inversion)

CVE-2018-3646 aka 'Foreshadow-NG (VMM), L1 terminal fault'
* Information from the /sys interface:
* This system is a host running a hypervisor:  NO
* Mitigation 1 (KVM)
  * EPT is disabled:  N/A  (the kvm_intel module is not loaded)
* Mitigation 2
  * L1D flush is supported by kernel:  YES  (found flush_l1d in /proc/cpuinfo)
  * L1D flush enabled:  UNKNOWN  (unrecognized mode)
  * Hardware-backed L1D flush supported:  YES  (performance impact of the mitigation will be greatly reduced)
  * Hyper-Threading (SMT) is enabled:  NO
> STATUS:  NOT VULNERABLE  (this system is not running a hypervisor)

CVE-2018-12126 aka 'Fallout, microarchitectural store buffer data sampling (MSBDS)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

CVE-2018-12130 aka 'ZombieLoad, microarchitectural fill buffer data sampling (MFBDS)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

CVE-2018-12127 aka 'RIDL, microarchitectural load port data sampling (MLPDS)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

CVE-2019-11091 aka 'RIDL, microarchitectural data sampling uncacheable memory (MDSUM)'
* Mitigated according to the /sys interface:  YES  (Mitigation: Clear CPU buffers; SMT Host state unknown)
* CPU supports the MD_CLEAR functionality:  YES
* Kernel supports using MD_CLEAR mitigation:  YES  (md_clear found in /proc/cpuinfo)
* Kernel mitigation is enabled and active:  YES
* SMT is either mitigated or disabled:  NO
> STATUS:  NOT VULNERABLE  (Mitigation: Clear CPU buffers; SMT Host state unknown)

> SUMMARY: CVE-2017-5753:OK CVE-2017-5715:OK CVE-2017-5754:OK CVE-2018-3640:OK CVE-2018-3639:OK CVE-2018-3615:OK CVE-2018-3620:OK CVE-2018-3646:OK CVE-2018-12126:OK CVE-2018-12130:OK CVE-2018-12127:OK CVE-2019-11091:OK
      posted in News
      s_mcleodS
      s_mcleod
    • RE: XCP-ng 8.0.0 Beta now available!

      I'll do whatever testing I can with the limited resources I / my team has at the moment.

      I'll update this comment with any tests I perform.

      Hardware

      • HPE BL460c Gen9, Xeon E5-2690v4, 256GB, Latest firmware (22/05/19), HT disabled
      • Local storage: 2x Crucial MX300 SSDs in hardware RAID 1
      • Network cards: Dual port HPE 560M, Dual port HPE 560FLB (both Intel 82599)

      Software

      • XCP-ng Centre 8.0 (stable) was used throughout testing.
      • Client VM is CentOS 7.6

      Test 1: ISO upgrade from fresh XCP-ng 7.6 install

      1. Fresh Install XCP-ng 7.6.
      2. Configure only the management interface, just left it as DHCP.
      3. Install XCP-ng 7.6 updates and reboot.
      4. Reboot off XCP-ng 8 Beta ISO and select upgrade from 7.6.
      5. Upgrade completed, rebooted server.
      6. Connected with XCP-ng Centre.
      7. Tried to copy VM cross-pool from XCP-ng 7.6 to the new standalone XCP-ng 8 beta host
        • Looked like it was going to work, but wanted to setup network bonding and VLANs first so cancelled.
      8. Created a 2x 10Gbit, 20Gbit LACP bond on the non-management NICs. (Failed Test)
        • After 1-3 minutes the bond showed up, but in a disconnected stated..
        • Waited 10 minutes, no change, nothing in xe-task list.
        • Rebooted host.
        • Tried deleting and re-creating the bond.
        • Bond still showing as disconnected: (Failed Test)
        •   # ovs-appctl bond/show bond0
      ---- bond0 ----
  bond_mode: balance-slb
  bond may use recirculation: no, Recirc-ID : -1
  bond-hash-basis: 0
  updelay: 31000 ms
  downdelay: 200 ms
  next rebalance: 6113 ms
  lacp_status: negotiated
  active slave mac: 00:00:00:00:00:00(none)

  slave eth0: disabled
    may_enable: false

  slave eth1: disabled
    may_enable: false
        • Networking.jpg Networking with VLAN.jpg NICs.jpg
        • Switches (Juniper EX4550) show:
          • Physical links: up
          • Bond: down (this should be up)
          • Received state: defaulted (this should be current), this means that the switches "did not receive an aggravation control PDU within the LACP timeout * 2".
          • MUC state: detached (should be collecting distributing).
        • Note: This is the same config as our 7.6 pools as it was with XenServer as pools as well.
        • Completely powered off server, then powered on again - this strangely enough Fixed the issue!
        • Leaving for now, continuing other testing.
      9. Created a 2x 10Gbit, Active/Passive bond on the management NICs.
        • Bond created and connected within 30 seconds.
      10. Tried to copy VM cross-pool from XCP-ng 7.6 to the new standalone XCP-ng 8 beta host (Failed Test)
        • The Next button was greyed out this time, but I couldn't see any issues. (Failed Test)
        • Closed and re-opened XCP-ng Centre, tried again - this time Next was available when trying to copy cross-pool, but I selected cancel to test again.
        • This time - the Next button was greyed out again, but I couldn't see any issues. (Failed Test)
        • cross-pool-migration-greyed-out.jpg
        • Leaving for now, continuing other testing.
      11. Added a SMB based ISO SR.
      12. Created a CentOS 7 VM, installed from the latest CentOS 7 ISO on the ISO SR to local storage.
      13. VM booted, installed xen tools, rebooted all fine.
      14. No networking in the CentOS 7 VM due to the XCP-ng bond0 not functioning, so could not test in-VM functions any further.
      15. Suspended VM failed. (Failed Test)
        • suspend fail.jpg
      16. Start VM successfully.
      17. Snapshot VM successfully.
      18. Delete snapshot successfully.
      19. Shutdown VM successfully.

      From https://github.com/xcp-ng/xcp/wiki/Test-XCP

      Text XCP - Results

      • Verify installation.
      • Verify connectivity with your interfaces - Pass, but required server to be powered off and on again rather than a reboot to work, will test again.
      • Verify connectivity to Shared Storages. - Cannot test, at test location there is no spare SAN LUN I can use at present.
      • Verify creation a new Linux VM (install guest tools).
      • Verify creation a new Windows VM (install guest tools) - We don't run Windows
      • Verify basic VM functionality (start, reboot, suspend, shutdown)
      • Verify migration of a VM from an host to another. - Cannot test, at test location there is no second XCP-ng 8.0 host available at present.
      • Verify migration of a VM from an old host to (this) release one -
        FAIL
      • Verify migration of a VM from a newest host to the old one (this test should be fail).
      • Check your logs for uncommon info or warnings. - Yet to test.

      Performance Test

      • Compare speed of write/read of disks in the old and in the new release - Yet to test.
      • Compare speed of interfaces in the old and in the new release - Yet to test.
      posted in News
      s_mcleodS
      s_mcleod
    • RE: Introduce yourself!

      Howdy all,

      I'm Sam McLeod, an engineer and long time DevOps proponent based in Melbourne, Australia.
      I work with an Australian Non-Profit and Charitable Organisation called Infoxchange.

      I'm currently moving us from XenServer 7.2 to XenServer 7.6.

      I was (still am) on pretty good terms with several people at Citrix's XenServer teams but I was pretty disgusted by the change in free licensing with the release of 7.3 and I'm also the Sam McLeod replying to David here: https://xenserver.org/blog/entry/xenserver-7-3-changes-to-the-free-edition.html

      • Twitter: https://twitter.com/s_mcleod
      • Website: https://smcleod.net
      • GitLab: https://gitlab.com/sammcj
      • GitHub: https://github.com/sammcj
      • HN: https://news.ycombinator.com/user?id=mrmondo
      • Goodreads: https://www.goodreads.com/user/show/5836466-sam
      • LinkedIN: https://www.linkedin.com/in/sammcj/
      posted in Off topic
      s_mcleodS
      s_mcleod