xcp-ng CPU low performance issue

enes.selcuk

Hi,
Dell r740 server SILVER 4210 processor headlight, but after installing xcp-ng operating system hypervisor operating system and virtualized windows and linux environments can not reach the CPU max value. I have checked all the performance settings in the BIOS, but unfortunately there is no solution. Note: it can reach maximum CPU value when it is checked by installing windows operating system directly on the current server.

Intel Xeon Silver 4216
22M Cache, 2,10 GHz
xcp-ng 8.0.0

I tested all the next version.
https://support.citrix.com/article/CTX200390#.VMjt9xm-SpE.twitter
https://wiki.xenproject.org/wiki/Xenpm_command
http://blog.blazis.com/how-to-enable-cpu-frequency-scaling-on-xenserver/
https://support.citrix.com/article/CTX200390

Danp

How are you testing? Also, might help if you told us the results.

enes.selcuk

@Danp said in xcp-ng CPU low performance issue:

How are you testing? Also, might help if you told us the results.

[19:10 xcp-ng-lmtguftk ~]# sudo xenpm get-cpufreq-para
cpu id : 0
affected_cpus : 0
cpuinfo frequency : max [2101000] min [800000] cur [2101000]
scaling_driver : acpi-cpufreq
scaling_avail_gov : userspace performance powersave ondemand
current_governor : performance
scaling_avail_freq : *2101000 2100000 2000000 1900000 1800000 1700000 1600000 150 0000 1400000 1300000 1200000 1100000 1000000 900000 800000
scaling frequency : max [2101000] min [800000] cur [2101000]
turbo mode : enabled

max 2.10 ghz but The processor can go up to 3.2 GHz under normal conditions.

xenpm start & sleep 5; killall -s SIGINT xenpm

The maximum speed of the task manager on windows is 2.10ghz.

tuxen

@enes-selcuk

What's the output of:

# xenpm start 1 | grep -i "avg freq"

enes.selcuk

@tuxen said in xcp-ng CPU low performance issue:

xenpm start 1 | grep -i "avg freq"

[06:13 xcp-ng-lmtguftk ~]# xenpm start 1 | grep -i "avg freq"
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz
Avg freq 64 KHz

enes.selcuk

ll performance settings in the server bios settings are as follows.

System Profile Current Value Pending Value
CPU Power Management System DBPM (DAPC)
Memory Frequency Maximum Performance
Turbo Boost Enabled
C1E Enabled
C States Enabled
Write Data CRC Disabled
Memory Patrol Scrub Standard
Memory Refresh Rate 1x
Uncore Frequency Dynamic
Energy Efficient Policy Balanced Performance
Number of Turbo Boost Enabled Cores for Processor 1 All
Number of Turbo Boost Enabled Cores for Processor 2 All
Monitor/Mwait Enabled
Workload Profile Not Available
CPU Interconnect Bus Link Power Management Enabled
PCI ASPM L1 Link Power Management Enabled

CPU Power Management Some options include = Performans Per Watt (OS ), Performans Per Watt (DAPC ), Performance, Workstation Performans, Custom
Workload Profile Some options include = HPC Profile =,Low Latency Optimized Prodile, Virtualization Optimized Performans Profile, Virtualization Optimized Performans Per Watt Profile, Database Optimized Performans Profile, Database Optimized Performans Per Watt Profile, SDS Optimized Performans Profile and SDS Optimized Performans Per Watt Profile
I have tried the above options one by one but the result has not changed.

tuxen

The incorrect clock results mean that Xen isn't in charge of frequency scaling management. Set the CPU Power Management to Performance Per Watt (OS) and run the previous xenpm, this time with a watch for real-time:

# watch 'xenpm start 1 | grep -i "avg freq"'

Start a VM boot storm (or a stress test inside one or more VMs) in order to generate some CPU load.

tuxen

@enes-selcuk

This is one of our production servers with the CPU Power Management config I posted earlier. It's a Dell R730 - Xeon E5-2630v3 @ 2.40 Ghz. As you can see, some cores are boosting to 3.0-3.1GHz. The maximum for this CPU model is 3.2 GHz:

  Avg freq	2737140	KHz
  Avg freq	2737140	KHz
  Avg freq	2689120	KHz
  Avg freq	2665110	KHz
  Avg freq	2737140	KHz
  Avg freq	3097290	KHz
  Avg freq	3073280	KHz
  Avg freq	2713130	KHz
  Avg freq	2665110	KHz
  Avg freq	2641100	KHz
  Avg freq	2761150	KHz
  Avg freq	2665110	KHz
  Avg freq	2689120	KHz
  Avg freq	2617090	KHz
  Avg freq	2881200	KHz
  Avg freq	2785160	KHz
  Avg freq	2641100	KHz
  Avg freq	2713130	KHz
  Avg freq	2689120	KHz
  Avg freq	2689120	KHz
  Avg freq	2689120	KHz
  Avg freq	2785160	KHz
  Avg freq	2833180	KHz
  Avg freq	2761150	KHz
  Avg freq	2761150	KHz
  Avg freq	2905210	KHz
  Avg freq	2713130	KHz
  Avg freq	3025260	KHz
  Avg freq	2761150	KHz
  Avg freq	2713130	KHz
  Avg freq	2833180	KHz
  Avg freq	2617090	KHz

enes.selcuk

@tuxen said in xcp-ng CPU low performance issue:

Start a VM boot storm (or a stress test inside one or more VMs) in order to generate some CPU load.

Thank you very much for your information. I tried and saw the following results in real time.Every 2,0s: xenpm start 1 | grep -i "avg freq" Thu Nov 28 13:27:00 2019

Avg freq 2521200 KHz
Avg freq 2689280 KHz
Avg freq 2836350 KHz
Avg freq 2689280 KHz
Avg freq 2731300 KHz
Avg freq 2689280 KHz
Avg freq 2773320 KHz
Avg freq 2752310 KHz
Avg freq 2689280 KHz
Avg freq 2689280 KHz
Avg freq 2710290 KHz
Avg freq 2710290 KHz
Avg freq 2689280 KHz
Avg freq 2710290 KHz

ruskofd

Try to disable C-States, it could also improve a lot of operations, including storage.

enes.selcuk

@ruskofd I turned off the C1E and C States features. Problem is solved. But SSD performance is very low compared to windows machine.

fibrewire

bumping this as it is relevant for my datacenter. Don't want to forget it!

olivierlambert

Feel free to submit a PR in our official doc, we might have a place for "Best practice" or something like that

stevewest15

@enes-selcuk Did you find a setting that works best w/ Dell servers?

I have a Dell R640's which I'll be using for LAMP/LEMP vm servers and was wondering what are the best settings to use in Dell Bios and if there are any changes I need to make on the xcp-ng host?

Dual Intel(R) Xeon(R) Gold 5218 CPU @ 2.30GHz
Level 2 Cache 16x1 MB
Level 3 Cache 22 MB
Number of Cores 16

Dell Bios Setting:
System Profile: Performance Per Watt (OS)
CPU Power Management: OS DBPM
Memory Frequency: Maximum Performance
Turbo Boost: Enabled
C1E: Enabled
C States: Enabled
Write Data CRC: Disabled
Memory Patrol Scrub: Standard
Memory Refresh Rate: 1x
Uncore Frequency : Dynamic
Energy Efficient Policy: Balanced Performance
Number of Turbo Boost Enabled Cores for Processor 1: All
Number of Turbo Boost Enabled Cores for Processor 2: All
Monitor/Mwait: Enabled
Workload Profile: Not Available
CPU Interconnect Bus Link Power Management: Enabled
PCI ASPM L1 Link Power Management: Enabled

Thank you!

SW