Nehalem cpu power management

bogikornel

My server has a Xeon X5650 Nehalem CPU. I get this during boot:
"Disabling C-states C3 and C6 on Nehalem Processors due to errata"

Also:
[18:23 home-xcp ~]# xenpm get-cpuidle-states 0
Max possible C-state: C1

Can I override this ? I.e. should C-state be higher to consume less power ? The stability is not interesting, as it would be a test/hobby server.

olivierlambert

Sadly, Intel doesn't provide any support nor updates for the microcode of this CPU generation. So, if it's disabled, it's because of an issue in the microcode with higher c-states.

So either use it "as is", or get a more recent CPU if you want more efficiency/less power usage.

bogikornel

@olivierlambert This cannot be overridden by a kernel boot parameter ? Because the reverse is possible.

olivierlambert

Is this message coming from Xen or Dom0 kernel?

bogikornel

@olivierlambert Yes, xl dmesg:

[18:25 home-xcp ~]# xl dmesg |grep C-
(XEN) [000000871c2971a8] Disable C3 and C6 C states on Nehalem processors due to error.
(XEN) [ 2.652059] mwait-idle: max C state 1 reached
(XEN) [ 2.698434] mwait-idle: max C state 1 reached

olivierlambert

Try this: https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#max_cstate-x86

And there's also https://xenbits.xen.org/docs/unstable/misc/xen-command-line.html#preferred-cstates-x86

bogikornel

I've looked through the xen documentation/forum, tried the parmeters, but nothing helped.

I'm thinking of putting an old 6.x xenserver on, because it doesn't have the higher c-state disabled

olivierlambert

Maybe @andyhhp knows which parameter to use

bogikornel

@andyhhp Can you help me ?

andSmv

Hello @bogikornel

The Intel_errata_workarounds and probe_c3_errata routines are executed unconditionally (there's no boot parameter to disable these checks)

The only way through would be to recompile your Xen. I'm not sure that this is a good idea because Intel documentation talks about Unpredictable System Behaviour.

If that is really what you need, I can help you with this (or do this for you), so let me know...

bogikornel

@andSmv thanks for the reply
The server is a home test, power consumption is more important than stability, so I'll try to recompile xen.I guess it will be enough to recompile only version 4.13.4 ?

PS.: I just looked at the source code and I don't see that there is a switch to turn it off. I will modify mwait-idle.c to see if it works

PS2: Done :

[09:50 home-xcp ~]# xenpm get-cpuidle-states 0
Max possible C-state: C9

cpu id               : 0
total C-states       : 5
idle time(ms)        : 134037
C0                   : transition [                5351]
                       residency  [                6507 ms]
C1                   : transition [                  13]
                       residency  [                  15 ms]
C2                   : transition [                  71]
                       residency  [                 120 ms]
C3                   : transition [                 701]
                       residency  [                6956 ms]
C4                   : transition [                4566]
                       residency  [              120437 ms]
pc3                  : [               13371 ms]
pc6                  : [               23594 ms]
cc3                  : [               14446 ms]
cc6                  : [               63719 ms]

The intel.c had to be modified to be correct. Let's see how stable it will be.

olivierlambert

Interesting Keep us posted!

bogikornel

@olivierlambert
In the original xen 4.13.4 source code I modified 2 files, one is mwait-idle.c :

--- mwait-idle.c.orig   2023-01-14 17:30:29.635655771 +0100
+++ mwait-idle.c        2023-01-14 09:06:04.274001499 +0100
@@ -864,8 +864,6 @@

 static const struct idle_cpu idle_cpu_nehalem = {
        .state_table = nehalem_cstates,
-       .auto_demotion_disable_flags = NHM_C1_AUTO_DEMOTE | NHM_C3_AUTO_DEMOTE,
-       .disable_promotion_to_c1e = 1,
 };

 static const struct idle_cpu idle_cpu_atom = {

the other is intel.c

--- intel.c.orig        2023-01-14 09:42:27.751861245 +0100
+++ intel.c     2023-01-14 09:43:05.036530787 +0100
@@ -326,8 +326,8 @@
          x86_match_cpu(models) )
     {
         printk(XENLOG_WARNING
-              "Disabling C-states C3 and C6 due to CPU errata\n");
-        max_cstate = 1;
+              "Nehalem errata deleted ...\n");
+        max_cstate = 9;
     }
 }

so cstate is not restricted, but many patches are missing that xcp-ng provides. At the moment windows VMs won't even start.

I am now trying to build a xen with patches by xcp-ng, although if someone could help me with the above modifications, I would appreciate it.

PS: Rebuilt the xen-hypervisor-4.13.4-10.36.xcpng8.3.x86_64 package with the above modification, and removed the detect-nehalem-c-state.patch

andyhhp

This is unconditional for a reason. The CSTATE errata in Nehalem are crippling - IIRC a core going in an out of a deep C state does not maintain cache coherency correctly, resulting in arbitrary memory corruption.

You really do care about not hitting this errata, even on a test/hobby server.