Please help me with the answer

MaratKzn

Re: Monitoring actual current CPU frequency in guest OS

DustinB

@MaratKzn repost with the content visible, not going to zoom out to try and read whatever is buried in that link.

MaratKzn

I have installed XCP-ng 8.2.1 on some testing HW (Lenovo SR650). I created Ubuntu 20.04 VM on it and am running some performance tests.

I am able to monitor CPU frequency in Hypervisor via xenpm command:

[13:50 test-xcp-ng ~]# xenpm get-cpufreq-average | grep 'frequency: '
average cpu frequency: 1134540
average cpu frequency: 2227060
average cpu frequency: 1449690
average cpu frequency: 2248070
average cpu frequency: 1428680
average cpu frequency: 1869890
average cpu frequency: 2143020
average cpu frequency: 1764840
average cpu frequency: 2290090
average cpu frequency: 1554740
average cpu frequency: 2584230
average cpu frequency: 1050500
average cpu frequency: 2206050
average cpu frequency: 1071510
average cpu frequency: 1890900
average cpu frequency: 1050500
average cpu frequency: 1134540
average cpu frequency: 1113530
average cpu frequency: 1113530
average cpu frequency: 1092520
average cpu frequency: 1113530
average cpu frequency: 1113530
average cpu frequency: 1134540
average cpu frequency: 1092520
average cpu frequency: 1113530
average cpu frequency: 1113530
average cpu frequency: 1155550
average cpu frequency: 1134540
average cpu frequency: 1155550
average cpu frequency: 1155550
average cpu frequency: 1176560
average cpu frequency: 1176560

But /proc/cpuinfo in Ubuntu VM shows only fixed CPU frequency:

root@test-vm:~# cat /proc/cpuinfo | grep -i mhz
cpu MHz : 2095.211
cpu MHz : 2095.211
cpu MHz : 2095.211
cpu MHz : 2095.211

Adding also lscpu output (again same MHz as above):

root@test-vm:~# lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 48 bits virtual
CPU(s): 4
On-line CPU(s) list: 0-3
Thread(s) per core: 1
Core(s) per socket: 1
Socket(s): 4
NUMA node(s): 1
Vendor ID: GenuineIntel
CPU family: 6
Model: 85
Model name: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
Stepping: 7
CPU MHz: 2095.211
BogoMIPS: 4190.29
Hypervisor vendor: Xen
Virtualization type: full
L1d cache: 128 KiB
L1i cache: 128 KiB
L2 cache: 4 MiB
L3 cache: 88 MiB
NUMA node0 CPU(s): 0-3
Vulnerability Itlb multihit: KVM: Vulnerable
Vulnerability L1tf: Mitigation; PTE Inversion
Vulnerability Mds: Mitigation; Clear CPU buffers; SMT Host state unknown
Vulnerability Meltdown: Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl cpuid pni pclmulqdq sss
e3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti intel_ppin ssbd ibrs ibp
b stibp fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt clwb xsaveopt xsavec xgetbv1 xsaves pku ospke md_clear flush_l1d

In some cloud providers (e.g. AWS EC2), CPU frequency that is visible in /proc/cpuinfo looks like current CPU frequency (looks "live" - changing on every run):

root@aws-instance:~# cat /proc/cpuinfo | grep -i mhz
cpu MHz : 3499.906
cpu MHz : 3500.627
cpu MHz : 3499.477
cpu MHz : 3501.732
cpu MHz : 3499.307
cpu MHz : 3499.868
cpu MHz : 3500.009
cpu MHz : 3499.829
root@aws-instance:~# cat /proc/cpuinfo | grep -i mhz
cpu MHz : 3500.867
cpu MHz : 3500.318
cpu MHz : 3500.907
cpu MHz : 3499.985
cpu MHz : 3500.236
cpu MHz : 3500.225
cpu MHz : 3499.921
cpu MHz : 3501.104
root@aws-instance:~# cat /proc/cpuinfo | grep -i mhz
cpu MHz : 3500.423
cpu MHz : 3500.150
cpu MHz : 3500.372
cpu MHz : 3500.026
cpu MHz : 3501.071
cpu MHz : 3500.911
cpu MHz : 3500.316
cpu MHz : 3500.048

I would like to monitor current CPU frequency in my Ubuntu VM that is running in XCP-ng hypervisor, similar to what I can see in AWS EC2 instance. Is that possible?

DustinB

@MaratKzn I'm not sure that you can do this through the CLI (you probably can), but you can see how any given VM is behaving through XO.

MaratKzn

@DustinB said in Please help me with the answer:

I'm not sure that you can do this through the CLI (you probably can), but you can see how any given VM is behaving through XO.

In XOA, I can only see CPU utilization, not CPU core frequency

DustinB

@MaratKzn said in Please help me with the answer:

@DustinB said in Please help me with the answer:

I'm not sure that you can do this through the CLI (you probably can), but you can see how any given VM is behaving through XO.

In XOA, I can only see CPU utilization, not CPU core frequency

Why do you think the frequency would change from whatever the physical hardware has?

MaratKzn

@DustinB

The frequency on the XEN host and in the VM on that host are different

For example

CPU: AMD EPYC 7702P
Base Clock 2.0GHz
Max. Boost Clock Up to 3.35GHz

On the XEN host:

[16:57 edge05 ~]# xenpm start 1|grep "Avg freq"
Avg freq 2540000 KHz
Avg freq 3200000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3280000 KHz
Avg freq 3320000 KHz
Avg freq 3340000 KHz
Avg freq 3320000 KHz
Avg freq 3340000 KHz
Avg freq 3320000 KHz
Avg freq 3240000 KHz
Avg freq 3140000 KHz
Avg freq 3340000 KHz
Avg freq 3300000 KHz
Avg freq 3240000 KHz
Avg freq 3340000 KHz
Avg freq 3280000 KHz
Avg freq 3320000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3260000 KHz
Avg freq 3340000 KHz
Avg freq 3280000 KHz
Avg freq 3340000 KHz
Avg freq 3320000 KHz
Avg freq 3280000 KHz
Avg freq 3280000 KHz
Avg freq 3240000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3300000 KHz
Avg freq 3340000 KHz
Avg freq 3320000 KHz
Avg freq 3160000 KHz
Avg freq 3080000 KHz
Avg freq 3340000 KHz
Avg freq 3100000 KHz
Avg freq 3340000 KHz
Avg freq 3240000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3320000 KHz
Avg freq 3040000 KHz
Avg freq 3100000 KHz
Avg freq 3320000 KHz
Avg freq 3220000 KHz
Avg freq 3220000 KHz
Avg freq 3060000 KHz
Avg freq 3320000 KHz
Avg freq 3340000 KHz
Avg freq 3300000 KHz
Avg freq 3320000 KHz
Avg freq 3180000 KHz
Avg freq 3320000 KHz
Avg freq 3240000 KHz
Avg freq 3240000 KHz
Avg freq 3160000 KHz
Avg freq 3140000 KHz
Avg freq 3160000 KHz
Avg freq 3260000 KHz
Avg freq 3260000 KHz
Avg freq 3200000 KHz
Avg freq 3180000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3080000 KHz
Avg freq 3000000 KHz
Avg freq 3000000 KHz
Avg freq 3340000 KHz
Avg freq 3100000 KHz
Avg freq 3200000 KHz
Avg freq 3220000 KHz
Avg freq 3160000 KHz
Avg freq 3120000 KHz
Avg freq 3220000 KHz
Avg freq 3200000 KHz
Avg freq 3200000 KHz
Avg freq 3200000 KHz
Avg freq 3200000 KHz
Avg freq 3300000 KHz
Avg freq 3320000 KHz
Avg freq 3340000 KHz
Avg freq 3160000 KHz
Avg freq 3340000 KHz
Avg freq 3240000 KHz
Avg freq 3340000 KHz
Avg freq 3180000 KHz
Avg freq 3340000 KHz
Avg freq 3140000 KHz
Avg freq 3080000 KHz
Avg freq 3260000 KHz
Avg freq 3240000 KHz
Avg freq 3220000 KHz
Avg freq 3180000 KHz
Avg freq 3240000 KHz
Avg freq 3240000 KHz
Avg freq 3260000 KHz
Avg freq 3280000 KHz
Avg freq 3060000 KHz
Avg freq 3020000 KHz
Avg freq 3200000 KHz
Avg freq 3200000 KHz
Avg freq 3180000 KHz
Avg freq 3180000 KHz
Avg freq 3200000 KHz
Avg freq 3200000 KHz
Avg freq 3240000 KHz
Avg freq 3180000 KHz
Avg freq 3320000 KHz
Avg freq 3320000 KHz
Avg freq 3240000 KHz
Avg freq 3200000 KHz
Avg freq 3280000 KHz
Avg freq 3280000 KHz
Avg freq 3020000 KHz
Avg freq 3020000 KHz
Avg freq 3180000 KHz
Avg freq 3200000 KHz
Avg freq 3040000 KHz
Avg freq 3320000 KHz
Avg freq 3320000 KHz
Avg freq 3340000 KHz
Avg freq 3360000 KHz
Avg freq 3340000 KHz

On the VM on this host:
watch "grep 'cpu MHz' /proc/cpuinfo"

Every 2.0s: grep 'cpu MHz' /proc/cpuinfo virtual.ml.srv.mq: Fri Feb 2 16:14:32 2024

cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127

Thank you!

planedrop

You have the tools installed in the VM correct?

I'm also curious, is there a reason you need to see correct CPU frequency in the VM guest? CPU usage % should still be accurate in terms of whatever is assigned to the VM. But not sure why frequency would be needed.

Still very curious if this is possible though so will help however I can, would be nice to have regardless.

MaratKzn

@planedrop

You have the tools installed in the VM correct?
If you mean xen-tools, the VM has the latest version installed (7.30.0-11)

I'm also curious, is there a reason you need to see correct CPU frequency in the VM guest? CPU usage % should still be accurate in terms of whatever is assigned to the VM. But not sure why frequency would be needed.

I have a suspicion that the VM is running at base CPU frequency and not using "Turbo mode"

watch "grep 'cpu MHz' /proc/cpuinfo"

Every 2.0s: grep 'cpu MHz' /proc/cpuinfo virtual.ml.srv.mq: Fri Feb 2 16:14:32 2024

cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127
cpu MHz : 2000.127

Even though "Turbo mode" works on the host itself.

[16:57 edge05 ~]# xenpm start 1|grep "Avg freq"
Avg freq 2540000 KHz
Avg freq 3200000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3340000 KHz
Avg freq 3280000 KHz
Avg freq 3320000 KHz
Avg freq 3340000 KHz
Avg freq 3320000 KHz
Avg freq 3340000 KHz
Avg freq 3320000 KHz
Avg freq 3240000 KHz
Avg freq 3140000 KHz
Avg freq 3340000 KHz
Avg freq 3300000 KHz
Avg freq 3240000 KHz
Avg freq 3340000 KHz

As a consequence, I am losing CPU performance on the VM

DustinB

@MaratKzn Do you have Guest Tools installed within the VM?

MaratKzn

Yes, VM has the latest version Guest Tools installed (7.30.0-11)

bleader

Hello, not sure I can actually help, but I'll try to bring some info.

As far as I know, Xen handles the cpu scaling by default, as you checked with xenpm. Therefore I would guess it does not expose the p-states to the guest. When reading /proc/cpuinfo the kernel will call show_cpuinfo() function will call arch_freq_get_on_cpu() which ends up checking support for p-states on fallback to a simpler version that will in our case probably always report the base clock. At least that's my understanding, I went through this quickly and may have missed bits, but sound "logical" to me.
I think the guest tools won't help you there, they report information from host to guest only, and as mentionned in the first point the guest frequency reporting is not live frequency.
The only way I see how to confirm your VM is using turbo clocks would be to pin your VM vcpus to fixed pcpus, load the VM and check these pcpu with xenpm as you were doing.
It is interesting to me you're seeing turbo clocks there at all, from my tests for some other stuff, on epyc I actually never saw the clock go above the base clock, did you do anything specific to enable it? I guess BIOS setup to enable turbo, and maybe enable it with xenpm?