Nehalem cpu power management

olivierlambert

Is this message coming from Xen or Dom0 kernel?

bogikornel

[18:25 home-xcp ~]# xl dmesg |grep C-
(XEN) [000000871c2971a8] Disable C3 and C6 C states on Nehalem processors due to error.
(XEN) [ 2.652059] mwait-idle: max C state 1 reached
(XEN) [ 2.698434] mwait-idle: max C state 1 reached

olivierlambert

Try this: https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#max_cstate-x86

And there's also https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#preferred-cstates-x86

bogikornel

I've looked through the xen documentation/forum, tried the parmeters, but nothing helped.

I'm thinking of putting an old 6.x xenserver on, because it doesn't have the higher c-state disabled

olivierlambert

Maybe @andyhhp knows which parameter to use

bogikornel

@andyhhp Can you help me ?

andSmv

Hello @bogikornel

The Intel_errata_workarounds and probe_c3_errata routines are executed unconditionally (there's no boot parameter to disable these checks)

The only way through would be to recompile your Xen. I'm not sure that this is a good idea because Intel documentation talks about Unpredictable System Behaviour.

If that is really what you need, I can help you with this (or do this for you), so let me know...

bogikornel

@andSmv thanks for the reply
The server is a home test, power consumption is more important than stability, so I'll try to recompile xen.I guess it will be enough to recompile only version 4.13.4 ?

PS.: I just looked at the source code and I don't see that there is a switch to turn it off. I will modify mwait-idle.c to see if it works

PS2: Done :

[09:50 home-xcp ~]# xenpm get-cpuidle-states 0
Max possible C-state: C9

cpu id               : 0
total C-states       : 5
idle time(ms)        : 134037
C0                   : transition [                5351]
                       residency  [                6507 ms]
C1                   : transition [                  13]
                       residency  [                  15 ms]
C2                   : transition [                  71]
                       residency  [                 120 ms]
C3                   : transition [                 701]
                       residency  [                6956 ms]
C4                   : transition [                4566]
                       residency  [              120437 ms]
pc3                  : [               13371 ms]
pc6                  : [               23594 ms]
cc3                  : [               14446 ms]
cc6                  : [               63719 ms]

The intel.c had to be modified to be correct. Let's see how stable it will be.

olivierlambert

Interesting Keep us posted!

bogikornel

@olivierlambert
In the original xen 4.13.4 source code I modified 2 files, one is mwait-idle.c :

--- mwait-idle.c.orig   2023-01-14 17:30:29.635655771 +0100
+++ mwait-idle.c        2023-01-14 09:06:04.274001499 +0100
@@ -864,8 +864,6 @@

 static const struct idle_cpu idle_cpu_nehalem = {
        .state_table = nehalem_cstates,
-       .auto_demotion_disable_flags = NHM_C1_AUTO_DEMOTE | NHM_C3_AUTO_DEMOTE,
-       .disable_promotion_to_c1e = 1,
 };

 static const struct idle_cpu idle_cpu_atom = {

the other is intel.c

--- intel.c.orig        2023-01-14 09:42:27.751861245 +0100
+++ intel.c     2023-01-14 09:43:05.036530787 +0100
@@ -326,8 +326,8 @@
          x86_match_cpu(models) )
     {
         printk(XENLOG_WARNING
-              "Disabling C-states C3 and C6 due to CPU errata\n");
-        max_cstate = 1;
+              "Nehalem errata deleted ...\n");
+        max_cstate = 9;
     }
 }

so cstate is not restricted, but many patches are missing that xcp-ng provides. At the moment windows VMs won't even start.

I am now trying to build a xen with patches by xcp-ng, although if someone could help me with the above modifications, I would appreciate it.

PS: Rebuilt the xen-hypervisor-4.13.4-10.36.xcpng8.3.x86_64 package with the above modification, and removed the detect-nehalem-c-state.patch

andyhhp

This is unconditional for a reason. The CSTATE errata in Nehalem are crippling - IIRC a core going in an out of a deep C state does not maintain cache coherency correctly, resulting in arbitrary memory corruption.

You really do care about not hitting this errata, even on a test/hobby server.