New server not performing (disk, memory?)

arvid

Hello!

I have migrated a VM to a new server running XCP-ng and I am not seeing as big of a jump in performance as I would have liked to.

The VM was previously hosted on a desktop PC running VMware ESXi 6.5:

CPU: intel i7-3930K
RAM: 32 GB, DDR-3?
Storage: SATA SSD

The new host is running XCP-ng 8.2 and has the following specs:

System: Asrock rack 1U4LW-X570/2L2T
CPU: Ryzen 9 5950X
RAM: 128 GB ram (4x32 DDR4-2666 MHz)
Storage: 2x corsair force SSD (PCIe 3.0 nvme) in software raid-1 (created during XCP-ng installation)

The VM is a compile machine running windows 8. When we moved to the new system we increased the number of vCPU from 8 to 32 and the amount of RAM from 8 GB to 12 GB. (I also tried the original configuration and saw the same performance.)

All in all, we saw only a 5%-10% shorter compile times and I was hoping for more. We have also compared the performance to windows machines with AMD Ryzen 9 3950X, 32 GB of DDR4-3600 MHz and various nvme storage. And there the performance is up to 50% higher than what we see in the VM now.

We have run some benchmark programs and it seems that memory and disk IO is where the VM is lagging behind the workstations.

Is there really this much overhead or is there something I can do to tweak the performance? Any suggestions appreciated.

olivierlambert

I suppose you have all the tools installed in your Windows machine?

It's hard to answer without knowing exactly what are you requirement: single thread or multithread load? RAM speed or RAM qty dependent? (or both?). IOPS load? small or big blocks? Seq or random access?

arvid

@olivierlambert Thanks for the reply. I have installed the tools and it looks ok I think:

[13:28 XHOST ~]# xe vm-param-get param-name=PV-drivers-detected uuid=...
true
[13:28 XHOST ~]# xe vm-param-get param-name=PV-drivers-version uuid=...
major: 0; minor: 0; micro: 0; build: 0
[13:28 XHOST ~]# xe vm-param-get param-name=has-vendor-device uuid=...
false

Regarding the load, I am really unsure. It should be a period of multithreaded compilation, followed by single threaded linking. But we don't see 100% cpu utilization, it fluctuated between 30%-70% (as reported by windows task mgr on the VM).

Are there any tools I can use to monitor this host or VM to see which resources it is lacking? I've looked at windows task manager in the VM, and the output of iostat on the dom0, and it doesn't look like anything is maxing out.

arvid

We have seen some smaller increases in performance when playing around with the commands xenpm set-scaling-governor and xenpm set-max-cstate. But still not as high as expected.

I wonder if maybe the cpu isn't fully supported? When I search xen hypervisor cpu compatibility I only see server cpus listed. If I run grep flags /proc/cpuinfo | uniq I see way less than I expect.

flags           : fpu de tsc msr pae mce cx8 apic mca cmov pat clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy abm sse4a misalignsse 3dnowprefetch bpext ibpb vmmcall fsgsbase bmi1 avx2 bmi2 erms rdseed adx clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 clzero xsaveerptr arat vaes vpclmulqdq rdpid

olivierlambert

I'm not aware about any CPU problems despite it's not a "server" one it should work. Have you tried to disable all CPU spectre/meltdown mitigations?

arvid

@olivierlambert Thanks. I tried that right now, booting with mitigations=off and saw no improvements unfortunately.

Before:

[09:06 VIRTUAL33 ~]# grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/itlb_multihit:Not affected
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/mds:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Vulnerable
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: usercopy/swapgs barriers and __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full AMD retpoline, IBPB: conditional, STIBP: disabled, RSB filling
/sys/devices/system/cpu/vulnerabilities/srbds:Not affected
/sys/devices/system/cpu/vulnerabilities/tsx_async_abort:Not affected

After:

[12:18 VIRTUAL33 ~]# grep . /sys/devices/system/cpu/vulnerabilities/*
/sys/devices/system/cpu/vulnerabilities/l1tf:Not affected
/sys/devices/system/cpu/vulnerabilities/meltdown:Not affected
/sys/devices/system/cpu/vulnerabilities/spec_store_bypass:Vulnerable
/sys/devices/system/cpu/vulnerabilities/spectre_v1:Mitigation: __user pointer sanitization
/sys/devices/system/cpu/vulnerabilities/spectre_v2:Mitigation: Full AMD retpoline, IBPB: conditional, STIBP: disabled, RSB filling