Non-server CPU compatibility - Ryzen and Intel

stormi

@olivierlambert Yes: https://nextcloud.vates.fr/index.php/s/5GHSMojntLKT5z5

With SHA256SUM being 172e295f561dc567251302a1a7670aa5cc07d495fec67428a25e3e837ff1a4a4

olivierlambert

Ah great, please test it @alex821982 that might be exactly what you seek

alex821982

@stormi
@olivierlambert

Thank you, I will try to check in the coming days

alex821982

From this ISO, the installation went fine
I will test further...

olivierlambert

Great The ISO is "just" a refreshed 8.2 with more updated baked into it, so you shouldn't have any problem to use XCP-ng after that!

Sam

@olivierlambert will it be fixed all of the IOMMU?

I'm still very running very stable 8.3, but I'm not using right now pcie passthroguh and SR-IOV. But I'm planning to in the next month or so.

Also some issue I've got from the internal 2,5gb nic. Realtek one.

Not having much trouble with the ASUS board so far.
I woule like to move to 2x48GB, 4x32, or 4x48gb. Affordable performance/price ratio.

olivierlambert

For consumer grade hardware, 8.3 is better: it's also easier to integrate updates without risking stuff unlike for an LTS.

The IOMMU thing is already fixed in 8.3 and recent 8.2

tmservo433

@olivierlambert Using Asrock Rack boards, I have had zero p roblems on this; that said, I am -VERY- eager to throw a new Threadripper (7000) into an XCP-NG setup; I am constantly looking at ways to significantly bump multi-run SQL Services as well as our custom app; and just the idea of passing through sets of ML cards on all those PCI-E lanes for some off-site workstation payloads is drool worthy. Ramp that up with 1TB or so of DDR5? I know -exactly- where that can go.

olivierlambert

A part of my lab uses "regular" but recent Ryzen CPUs with decent DD5, the results are pretty nice So I can imagine how great could be 7000 series Threadrippers

stormi

@olivierlambert said in Non-server CPU compatibility - Ryzen and Intel:

Great The ISO is "just" a refreshed 8.2 with more updated baked into it, so you shouldn't have any problem to use XCP-ng after that!

It also was created with new ISO generation scripts and hasn't undergone QA testing, so it's not ready yet to be an official release. No particular issues are expected, but no promises either until it receives the QA stamp.

gecant

May I ask a question?

Long time XCP-ng user on Xeon CPUs, I am now considering one AMD Ryzen 9 7950X3D for a new XCP-ng 8.3 setup.
(All running VMs will be Linux)

The question is:
Is XCP-ng kernel (currently 4.19 on XCP-ng 8.3) able to support the boosted CPU frequencies of the CPU? That is up to 5.7 GHz.
Or is the hypervisor kernel irrelevant?

To my knowledge (also from several bare-metal setups) proper frequency boost for Zen4 is possible only by using newer kernels (actually enabling "amd_pstate" driver is needed).

Is the kernel inside the VM that counts here? I mean enabling "amd_pstate" driver inside the VM it enough to reach the 5.7GHz freqs?
Or is the hypervisor kernel (4.19) a limitation to reach those high frequencies?

Thank you.

olivierlambert

IIRC, boost is the real of Xen, not the Dom0 kernel (because Xen deals with memory and CPU, not the Dom0, which is a VM). Sometimes the frontier is blurry and it's not trivial to know (or decide when you dev it) who should handle what. I'll ask around to be sure about boost responsibility (even if I'm pretty sure it's Xen)

gecant

@olivierlambert said in Non-server CPU compatibility - Ryzen and Intel:

I'll ask around to be sure about boost responsibility (even if I'm pretty sure it's Xen)

Thank you!

I guess the amd_pstate driver is not backported on 4.19 kernel that XCP-ng uses, right?

If anyone with a Zen4 CPU can check the CPU frequencies that VMs are able to reach by default, I guess this would be useful.

redakula

@gecant said in Non-server CPU compatibility - Ryzen and Intel:

If anyone with a Zen4 CPU can check the CPU frequencies that VMs are able to reach by default, I guess this would be useful.

I have a Ryzen 9 7900 which reports this:

[14:50 xenserver ~]# xenpm get-cpufreq-para 0
cpu id               : 0
affected_cpus        : 0
cpuinfo frequency    : max [3700000] min [3000000] cur [3000000]
scaling_driver       : powernow
scaling_avail_gov    : userspace performance powersave ondemand
current_governor     : ondemand
  ondemand specific  :
    sampling_rate    : max [10000000] min [10000] cur [20000]
    up_threshold     : 80
scaling_avail_freq   : 3700000 *3000000
scaling frequency    : max [3700000] min [3000000] cur [3000000]
turbo mode           : enabled

But i might have disabled boost in BIOS for power saving - i can check later

Also very interested in details on the support of newer Ryzens. From what i understand power management for new zen architectures is fairly recent in the kernel so i do wonder how well supported these cpu's are currently.
From xenpm output it does not appear to scale below 3GHz - my previous intel cpu would scale to 800MHz.

gecant

@redakula Indeed, low power consuption on "amd_pstate" driver allows much lower frequencies down to 400 MHz.

Can you also please try to see the output of command xenpm start 1|grep "Avg freq" under some CPU load?
This allows to see the CPU frequency regardless of the available scaling frequencies.

For example, you can run in one VM stress-ng -c 4 to have 4 CPUs at full load and while this is running see the output of xenpm start 1|grep "Avg freq" on your Dom0 to see the CPU frequencies achieved under that stress.

Thank you.

redakula

@gecant said in Non-server CPU compatibility - Ryzen and Intel:

@redakula Indeed, low power consuption on "amd_pstate" driver allows much lower frequencies down to 400 MHz.

That is what i suspected from the p-state driver being heavily developed in kernel 6.5 and 6.9... Hope we will get support soon in XCP-NG.
The 7900 is ridiculously overkill for most of my homelab use so each watt saved helps

Can you also please try to see the output of command xenpm start 1|grep "Avg freq" under some CPU load?
This allows to see the CPU frequency regardless of the available scaling frequencies.

For example, you can run in one VM stress-ng -c 4 to have 4 CPUs at full load and while this is running see the output of xenpm start 1|grep "Avg freq" on your Dom0 to see the CPU frequencies achieved under that stress.

Thank you.

Seems to scale above base frequency (3,7GHz for the 7900) just fine - but does not go below 3GHz. 5GHz is pretty close to the max boost of 5,4GHz.
System is running 8.3 Beta with xen 4.17.
This is running stress on a few cores:

[17:25 xenserver ~]# xenpm start 1|grep "Avg freq"
  Avg freq      2960000 KHz
  Avg freq      2960000 KHz
  Avg freq      2960000 KHz
  Avg freq      2923000 KHz
  Avg freq      2960000 KHz
  Avg freq      2960000 KHz
  Avg freq      2960000 KHz
  Avg freq      2960000 KHz
  Avg freq      2960000 KHz
  Avg freq      2960000 KHz
  Avg freq      3922000 KHz
  Avg freq      3108000 KHz
  Avg freq      5069000 KHz
  Avg freq      5069000 KHz
  Avg freq      5069000 KHz
  Avg freq      5069000 KHz
  Avg freq      2886000 KHz
  Avg freq      2886000 KHz
  Avg freq      5069000 KHz
  Avg freq      5069000 KHz
  Avg freq      2923000 KHz
  Avg freq      3034000 KHz
  Avg freq      5069000 KHz
  Avg freq      5069000 KHz

gecant

@redakula By looking at your xenpm start 1|grep "Avg freq" output, this 5 GHz seems normal for your Ryzen 9 7900, because I can see at least 8 CPU threads already running on high frequency.
To get to 5.4 GHz you need less CPUs on high frequencies, you see the more CPUs are boosted at the same time, the lower the max frequency they can reach. For about 8-9 CPUs already boosted, 5 GHz seems reasonable.

So from your output I believe your CPUs are reaching the boosted/high frequencies without problem.

Thank you for your tests here on this.

redakula

@gecant

Yep - i just tried stressing a single core and got to 5.4GHz.

But would be nice to get power management also for these cpu's as they are pretty awesome

tmservo433

Largely depends on your homelab. I'm currently in my homelab using a 3970X Threadripper But, I got that board/chip free after we did a client upgrade to the new Threadrippers, and there isn't a single amount of money I could save on energy to match that savings.

That said, think about what you need to use your homelab configuration for; I often find that the availability of RAM and total available cores. But I guess different needs matter too..