Non-server CPU compatibility - Ryzen and Intel

nhoty

@olivierlambert Sure! I came to xcp-ng watching that video from @lawrencesystems , and asked him if he knows if am5 sockets from ryzen were working with xcp-ng he advises me to ask the forum.
So I was hoping that someone has tried a new Ryzen CPU on their builds.

olivierlambert

Ah, you mean on latest and greatest desktop AMD platform? I think you asked in the right place, we'll see if someone in the community did

Andrew

@nhoty I have an AMD Ryzen 5 5600U setup for testing in a small box. Not exactly the same but it works well. I would think the Ryzen 9 7900 in the right board would be good... For a server use ECC memory and just try to keep the CPU cool!

As for Intel, I run some i5 and i7 desktop CPUs and they work faster than older server CPUs. Just note that Intel desktop CPUs normally skip the ECC options.

olivierlambert

FYI, I ordered a small configuration to test it here. I'll keep you posted.

dj423

Is it ok to mix hosts with Intel and Ryzen CPU in the same pool, or does that even matter? As pertaining to migrating VM instances between them? Thinking of refreshing my hosts to more modern CPU/RAM - but I don't want to induce issues of compatibility.
Thank you

olivierlambert

There's a fully automated system that will reduce all features to the lowest capable CPU of the pool, so there's no compatibility problems (a big thanks to @andyhhp for that)

nhoty

@Andrew Thank you I will try that, I'm only waiting for Asrock Rack Motherboard B650D4U (https://www.asrockrack.com/general/productdetail.asp?Model=B650D4U#Specifications ) so I can get ECC compatibility. Hope that they will release it soon.

andyhhp

You cannot mix Intel and AMD CPUs in the same pool. They're simply not compatible enough for a VM to survive a migration between the two, so we explicitly disallow it. (You can't even --force a pool join to this effect.)

dj423

@andyhhp
Can we migrate VM's over to another pool with the different CPU type?
Thank you

Andrew

@dj423 Yes. You can do a cold (VM power off) migration between different CPU types.

olivierlambert

Warm migration is even a great solution in those cases (Intel to AMD or vice versa)

olivierlambert

Just installed XCP-ng 8.3 on a Zen 4 platform:

Asus Prime B650M-Wifi
Ryzen 5 7600 (Zen 4)
32GiB DDR5

Just had to enable x2API in the BIOS, everything worked out of the box (including the 2.5G Realtek NIC).

mgales

With the Ryzen 9 7900X, an Asus B650E series motherboard, and 32 GiB DDR5, the results I've gotten aren't favorable. I was able to install xcp-ng-8.3.testing-2023.02.15-12.19-install.iso without enabling x2API in the BIOS, and XCP-ng comes up fine. While xo-ce (running on the same box) comes up right away, it takes about 8-10 minutes before it acquires it's IP address (both XCP-ng and xo-ce use DHCP to get their IPs from a DHCP/DNS).

After xo-ce is up and accessible, Windows 10 works well (install from ISO, boot & run performance is good). OSs from various Linux distros are horrendously slow. It took about 2 hrs to install Linux Mint 21.1 from an ISO. It takes several minutes, with any of the Linux ISOs I've tried (CentOS Stream 9.1, Ubuntu 22.04.1, Linux Mint 21.1) for a 2nd screen to appear after the initial install screen (which comes up fine).

The install of Mint took about 2 hrs to complete.
VM configuration: 8-CPUs, 8 GiB RAM, 22 GiB hard drive.

Linux Mint OS comes up, but with similar slow boot time (29 minutes before LM logo, CPU usage at 100% on Stats tab during that 29 minutes, but oddly everything else at 0. Then I noticed CPU usage dropped and both Network and Disk throughput had activity, but still 0B of 8GiB RAM used, and I switched tabs and these 2 messages appear in the console screen:

1. [  1.778229] vbd vbd-5696: 19 xenbus_dev_probe on device/vbd/5696
2. [165.437239] piix4_smbus 0000:00:01.3: SMBus Host Controller not enabled!).

(LM logo appeared on the console window shortly after the 2 messages above)

At 48 minutes (per the VM_Started time on the Logs tab) the arrowhead cursor first appeared (console window), and the login screen appeared at 52 minutes. At 57 minutes the desktop is displayed (still at 0 GiB RAM usage) and system response is extremely slow.

Another Note - these messages appeared on the xo-ce console window:

[ 3684.425914] rcu: INFO: rcu_sched self-detected stall on CPU
[ 3684.425914] rcu: #1-...!: (1 ticks this GP) idle=d22/0/0x1 softirq=3206/3206 fqs=1
[ 3684.425914] rcu: rcu_sched thread starved for 4463 jiffies! g321 f0x0 RCU_GP_WAIT_FQS(5) -> state=0X402 ->cpu=1
[ 3684.425914] rcu: #unless rcu_sched thread gets sufficient CPU time, OOM is now expected behavior.
[ 3684.425914] rcu: RCU grace-period kthread stack dump:
{Nothing after that on the screen}

Side Note: my original Intel XCP-ng box shows all 4 (CPU, Memory, Network and Disk all peaking roughly at the same time that the Linux Mint OS boots, but on this new Ryzen box, 0B Memory at all times on the Stats screen for the Linux Mint VM. The xo-ce VM does show about 712 MiB Memory usage.

I did try a separate boot, with X2API enabled in the motherboard BIOS before booting XCP-ng, but it didn't make any noticeable difference compared to having it set to Default.

Repeating: I didn't experience sluggishness, problems installing, booting or running Windows 10 from an ISO under XCP-ng on this Ryzen box, only Linux distros.

Far Side Notes: I used the same Linux Mint 21.1 ISO from above and installed it directly on a separate hard drive on the same Ryzen box and it works good. I'm running VMware running under Linux Mint 21.1 and Rocky Linux 9 loaded directly on this Ryzen box on separate hard drives and VMware works good when I boot on those drives (XCP-ng/xo-ce is running on a Samsung 970 plus M.2).

I hope some of the information will be useful to the XCP-ng development team.
I'd be glad to test a fix on my Ryzen box if that would help.
It's a home lab box, so starting over isn't an issue.

I'm fairly new to XCP-ng. I'm impressed by the support of the XCP-ng team for home users too.

Addendum - inside the Linux Mint VM, I entered the free command in a terminal window.
Results:

	total	used	free	shared	buff/cache	available
Mem:	8119596	727396	6087584	20004	1304616		7103828
Swap:	1043340	     0	1043340

olivierlambert

What kind of storage are you using?

edit: I will try on my setup, but a least on Debian I had 0 issue and installation was blazing fast. Can you try on Debian and see if you still have issues?

olivierlambert

So I installed Linux Mint 21.1 on my Zen4 setup, using a Debian 11 template, with 4vCPUs and 4GiB RAM and 40GiB virtual disk:

the ISO booted on the OS is less than 1 minute (likely 20 secs)
the installation itself was 5/6 minutes long tops

After the install, booting the OS took around 15 seconds (to get to the login)

Note: I'm using a cheap Kingston 120GiB SSD, not even an NVMe.

edit: I had to do bash /mnt/Linux/install.sh -d debian -m 11 to install the tools, but that's it.

olivierlambert

So it's likely an issue with your physical machine (buggy BIOS? thing not enabled?). I would start by the usual dmesg in the dom0, and also a xl dmesg.

mgales

Thanks @olivierlambert for going to those lengths to recreate the same steps that I had done.
It's good to know that the new generation of Ryzen processors and motherboards should work without issues for most.
I appreciate the time you spent - it's beyond what I expected.

I'll try your suggestions with dmesg, experiment with BIOS settings, and do the install to a regular hard drive for starters. I'm not familiar with dmesg, but I'll read up on that - looks like it could reveal boot issues.

It puzzles me why the only issues I'm having are with Linux VMs on my hardware, but hopefully I can figure out what's causing it. Forgot to mention that I had updated to the latest BIOS update from Asus before doing this install, but it didn't change the behavior from what I was experiencing prior to the update.

If I'm able to get it working, I'll post back here what helped.

Mike ( @mgales)

olivierlambert

Okay keep us posted Since it's work here, it's not obvious on what to do, because not reproducing a problem makes it harder to solve

mgales

BTW, here's the full model number of the motherboard (purchased as part of a combo CPU/RAM/MB package):
ROG STRIX B650E-F Gaming WIFI.
Product URL: https://rog.asus.com/motherboards/rog-strix/rog-strix-b650e-f-gaming-wifi-model/

scot1297

@olivierlambert I have a question for you on the AMD platform you have listed in this thread. I am looking at updating my current setup to a newer amd zen 4 cpu and motherboard. My question is are you able to pass through some of the usb ports from this motherboard or would I still need to use a dedicated pci USB device. I'm asking because currently I have a separate PCI USB device for pass through, but I am hopefull that in my next build I don't need to have a separate USB PCI slot taken and can just use the USB ports on the motherboard?

Thanks for any insight. I do know this is an edge case and most don't need the USB pass through like this, but I have a few USB devices I need to provide to a VM.

Quick Edit, I know you can do just a plan USB passthrough, but I am looking at the PCI level so that on host restarts I dont' have to redo USB passthrough each time. Because right now with my gpu and USB pci card on host restart I dont' have to re-pass through these devices.

Thanks,
Scot