Reading temperature of Intel CPUs by (carefully) making a hole in the Matrix

Hello, I'm Teddy Astie working on the Xen hypervisor for XCP-ng, and will present how to read Intel CPU temperature (technical aspects) and how it got implemented to XCP-ng.

It's a well known fact that CPUs consume electricity, and by Joule effect, produces heat. As a consequence, CPUs can get warm and it can be useful (or crucial) to know how warm it is (for various reasons). This can be done through built-in sensors of the CPU, which can be queried from software.

Most operating systems including Linux have facilities to read such sensors (through coretemp using sensors command), which generally work, but things get trickier with Xen, more so on Intel platforms.

CPU temperature on Intel : Thermal Status MSR

CPU temperature on Intel platforms is exposed as a part of internal registers named MSRs (Model Specific Register), in particular one named IA32_THERM_STATUS. Each individual core has its own register, that needs to be read from the said core using the dedicated rdmsr (Read MSR) instruction.

This register gives several informations on the thermal sensor, in particular, the "digital readout". Note that this is not exactly the CPU temperature but "how far we are from maximum temperature threshold (tjmax)". To know this tjmax value, we also need to read another MSR named MSR_TEMPERATURE_TARGET for "Temperature Target".

Once we have all this information the final CPU temperature (in °C) can be computed as
temp = tjmax - "Digital Readout"

CPU package (socket-wide) temperature can be read using the same approach but taking the "Digital Readout" from MSR_PACKAGE_THERM_STATUS MSR instead.

The 'in the matrix' problem

But under Xen, most of the system is running as some form of virtual machine (Dom0), introducing various challenges :

• A Dom0 vCPU is not a physical CPU, for instance, vCPU 0 of Dom0 doesn't always* run on physical CPU 0 (unless dom0_vcpus_pin is set)
  • we can't get coherent temperature readings from this point of view
  • there may be less Dom0 vCPUs than physical CPUs, breaking the Linux coretemp model
• Dom0 doesn't see this hardware capability, hence coretemp fails to load
• And there is no direct access to hardware registers (rdmsr is restricted)

(Carefully) breaking out of the matrix

In order to read CPU temperatures, we need a way to get access to such registers, but in a controlled way :

• we want a way to read these MSR for a specific physical CPU, to compute the final temperature
• in a way practical for the user
• and not too invasive for the hypervisor

Such interface already exists through XENPF_resource_op hypercall, which is a interface to read a specific "platform resource", which includes CPU's MSR from a known list.

What is missing here is to teach Xen of these registers of interest, which is just a matter of updating the list.

Once done, what remains is a program that reads these registers through hypercalls, compute CPU temperatures from it; which is made available as xenpm get-core-temp.

Compute and GPU | XCP-ng Documentation

This section is dedicated to compute related things, like GPU/vGPU or PCI passthrough, nested virtualization or advanced Xen features.

XCP-ng Documentation

What about AMD CPUs ?

So far, this applies to Intel CPUs, and AMD CPUs are very different.

Contrary to Intel CPUs, AMD CPUs don't have their temperature exposed as CPU MSRs but as a part of the Northbridge registers (System Management Unit subsystem), which are reachable from Dom0 code through usual means.

These can be accessed through regular PCI MMIO means (hence, directly reachable from Dom0), and doesn't require specific support, and as a result works through regular k10temp driver or the more recent zenpower project.

Though, some recent AMD CPUs may require patches to k10temp driver to be recognised properly.