Any hardware recommendations? - consolidating desktop on server
-
I am not using and PCI pass through but I have three of these Ryzen systems and they have working great for over a year.
Parts list here: https://kit.co/lawrencesystems/ryzen-9-xcp-ng-virtualization-server-build-with-25gb
Video on the setup here:
https://youtu.be/AbzKv_3x5S4?si=wI8OCd3-YMDl4jyS -
I have 2 XCP-ng nodes with a Ryzen 9 3900X. Both work great! I also have a 7900X running Linux as my primary desktop.
I would personally avoid the 7950X3D. That particular chip has 2 chips with cores (AMD refers to as CCDs). One of the chips has the extra cache, the other does not. This isn't really a bad thing but you have to be more careful to which cores you "pin" your gaming VM to. The non-3D chips still have 2 CCDs, but they are both equal. This consideration in my opinion is very similar to Intel's usage of "P" cores and "E" cores. It works but you have to pay attention to the specific core assignments. For the GPU, I'd probably go with an Nvidia GPU. Intel is good but it's still first generation. I heard the RX6000 are supposedly good compared to the 5000 for passthrough.
For passthrough, I've only messed around with it in a limited capacity with old hardware (GT710/GT210 for example). I've had mixed luck, Believe it to be issues with reset and GPU BIOS issues - on a 1070Ti. It was just a small experiment so I didn't put much troubleshooting effort.
For Motherboard recommendations. I have been using the Asus ProArt X670e. For PCIe, it has 3 x16 physical slots (16/0/4 or 8/8/4 PCIe lanes). 8 USB-A ports, 2 Thunderbolt 4 ports and 1 USB-C 20G port. For ethernet, it has a 2.5G port (Intel i225) and 10G (Marvell AQtion AQC113CS, also supports 2.5G and 5G). XCP-ng 8.3 will add support for the 2.5G port natively, but there's also a way to get it in 8.2 and @stormi is preparing an updated iso. I believe @Andrew has support for the 10G port. Not sure how PCIe works on it, but I have heard of very few Mobos that have issues with passthrough.
-
@ajpri1998 The i225 driver is being added as a standard install for 8.2 and 8.3. This will allow for at least basic connectivity in many new machines. The existing atlantic driver does not support the newer 10G cards. A new alternate version of the atlantic driver will be available to easily install on XCP as needed.
-
Just wanted to chime in here as I've considered doing this exact thing for years and have never actually managed to be successful enough to go all in.
Not specific to XCP-ng or any issues with it, just GPU passthrough is a huge pain and is likely to create issues down the road so I personally won't do it for my desktop machine as I need that running 100% of the time.
Not trying to put any doubt in here, just that make sure you know what you are doing and know the difficulty that can sometimes occur. I managed to get something similar working on Proxmox once long ago, but it took me literal months of headache and troubleshooting to finally get it functional.
In my mind servers are servers and desktops are desktops, but the idea of putting it all together is still really fun to me so if you do get this working/go through with it, please post about it so we can all hear!
Now spec wise, I agree with others, avoid the 7950X3D for this use case, I have one in my desktop and it's absolutely amazing but I wouldn't use it for a server, may just create some complications (though if you can manage to get it working in a way where the gaming machine/desktop runs on the 3D vCache cores, it'd be a beast).
GPU wise, go with NVidia or AMD IMO, not Intel, there is more information on the 2 other brands when it comes to passthrough, NVidia likes to make it difficult whereas AMD tends to be a bit easier. I've done it with NVidia in the past primarily though.
Maybe also a thought would be to consider a server grade GPU and then use vGPU for this to avoid some of the passthrough headache?
-
I looked at desktop systems for some long while. None really do the needful very cheaply. I wanted to do PCI passthru at least and run a few more substantial vms. This meets the requirements and had pretty much everything
Looks like it was never powered on, not a spec of dust inside. Swapped out the supplied windoze disk with another SSD and so far, XCP likes it very much. I am very impressed with what I obtained for ~$1500 with tax.
-
Thanks for all the input. I ended up purchasing most items on my list. I went with the 7950X and an Nvidia 4070 GPU. I'm still waiting for the RAM to arrive, so I tested the 4070 in my current server. Passthrough is working perfectly. Nvidia drivers and software detected the GPU and installed just fine. I wasn't seeing any additional performance in games with the 4070, and I ran some benchmarks. Interestingly, the system is heavily CPU bottlenecked. I wasn't surprised by the bottleneck itself but by the severity of it. The 4070 wasn't doing much better than the old GTX 970. In the benchmarks, CPU utilization is >80% and GPU <50% but usually between 20-40%. So once I get the Ryzen CPU in there, it will hopefully perform close to similar configurations.
I ended up going with the Supermicro since I've had experience with them, and they have provided me with very good support in the past. I generally avoid "gaming" oriented motherboards. I feel that they are more for show than quality. Some of my research on the Asus motherboards seem to indicate this, but it was just anecdotal information. Plus I'm completely turned off by the Asus gaming BIOS screen, no thanks. AsRock I don't have any experience with. I was debating the AsRock board since it includes 10Gbe, but I would prefer SFP28 anyway. So I'll just drop an SFP28 card in sometime in the future when I upgrade my network switch.
The only feature I would like that is currently missing with Supermicro is the AMD Ryzen "eco-mode." I just learned about this recently. It will throttle the CPU from 170w to either 105w or 65w. And the benchmarks that I found online have indicated only about a 5% or so performance penalty at 65w. I emailed Supermicro about it and they are already looking at adding the feature into one of there future BIOS updates.
@Theoi-Meteoroi said in Any hardware recommendations? - consolidating desktop on server:
I looked at desktop systems for some long while. None really do the needful very cheaply. I wanted to do PCI passthru at least and run a few more substantial vms. This meets the requirements and had pretty much everything
Looks like it was never powered on, not a spec of dust inside. Swapped out the supplied windoze disk with another SSD and so far, XCP likes it very much. I am very impressed with what I obtained for ~$1500 with tax.
Rack mount is a requirement for me. I already have a 4U case, but for $1500 that looks pretty good
@planedrop said in Any hardware recommendations? - consolidating desktop on server:
Just wanted to chime in here as I've considered doing this exact thing for years and have never actually managed to be successful enough to go all in.
Not specific to XCP-ng or any issues with it, just GPU passthrough is a huge pain and is likely to create issues down the road so I personally won't do it for my desktop machine as I need that running 100% of the time.
I had enough success with the old RTX 970 that I figured I'd take the plunge. Also, it's time to replace both my old Hades Canyon NUC and upgrade my server, so this seemed like a good time to go all in. If it ends up not working, no harm done. I needed to upgrade the server anyway. My Macbook Pro is my "desktop," so this helps me clear out some extra hardware clutter. I'll make the NUC my edge router running XCP-NG. I'm currently using a 2010 Mac mini, and I'm surprised I got XCP-NG to work on it at all. But the little mac mini is ready to be put down.
Now spec wise, I agree with others, avoid the 7950X3D for this use case, I have one in my desktop and it's absolutely amazing but I wouldn't use it for a server, may just create some complications (though if you can manage to get it working in a way where the gaming machine/desktop runs on the 3D vCache cores, it'd be a beast).
GPU wise, go with NVidia or AMD IMO, not Intel, there is more information on the 2 other brands when it comes to passthrough, NVidia likes to make it difficult whereas AMD tends to be a bit easier. I've done it with NVidia in the past primarily though.
Thanks for the heads up. I did a little more research, and the 7950X3D does have its issues, and the performance is questionable anyway. I went with a 7950X on Black Friday. I wanted an Intel Arc GPU just to not support the Nvidia/AMD duopoly. GPU prices have come down, but they are still ridiculous. I did a little research on the Intel Arc, and they seem to have a similar reset issue as some AMD cards in passthrough. Nvidia seemed like the safest bet, so I nabbed a 4070 on sale. $500 for a "mid range" card, which was more than I wanted to spend. However, I plopped it into my existing Xeon server, and passthrough works flawlessly.
Maybe also a thought would be to consider a server grade GPU and then use vGPU for this to avoid some of the passthrough headache?
I considered this. If it was only a $50 to $100 price premium over a similarly performant GPU, it would be worth the extra cost to save on the headaches. But vGPU cards are about twice the price. Maybe it would make sense if I needed a GPU to support multiple VMs, but I really only need one GPU for one VM.
@ajpri1998 said in Any hardware recommendations? - consolidating desktop on server:
I would personally avoid the 7950X3D. That particular chip has 2 chips with cores (AMD refers to as CCDs). One of the chips has the extra cache, the other does not. This isn't really a bad thing but you have to be more careful to which cores you "pin" your gaming VM to. The non-3D chips still have 2 CCDs, but they are both equal. This consideration in my opinion is very similar to Intel's usage of "P" cores and "E" cores. It works but you have to pay attention to the specific core assignments.
Thanks for highlighting this. I went with the 7950X. I found some reviewers echoing what you said as well.
@lawrencesystems said in Any hardware recommendations? - consolidating desktop on server:
Parts list here: https://kit.co/lawrencesystems/ryzen-9-xcp-ng-virtualization-server-build-with-25gb
Thanks, there are some components I may pull from this list, like that Mellanox card. I'm not going for pure CPU/GPU performance. A stable system for all my other tasks are more important than the gaming VM, so I went with ECC RAM.
-
The pass-through worked well in my case, attached to a Ubuntu 22.04 VM and the driver loaded ( had to blacklist it because I wanted to use the CUDA toolkit ) fine. The toolkit also installed fine, I kept the OpenGL libs so the GUI still works in XCP. I'd suppose this would also work just fine with Windows. I'll have to test that with a eval ISO.
-
Just a little update on some of the purchases for folks shopping for compatible hardware. The Supermicro motherboard was a bust. Supermicro H13SAE-MF wouldn’t boot XCP-NG. It initially would boot, but after applying a BIOS update, it wouldn’t boot it at all. A couple of other Linux CDs wouldn’t work either, but I did get Ubuntu to boot. And I had other weird issues where the BIOS would hang at POST if I had my USB DVD drive plugged into certain USB ports instead of others.
The BMC worked ok overall. Interestingly, mounting an ISO image via IPMI requires an additional license, so I used physical CDs or my piKVM. Overall the whole experience felt like I was working with a preproduction board and not a finished product. I emailed Supermicro and they told me only UEFI OSes are supported, and I was using UEFI OSes. They also sent me the OS compatibility chart, which you can find here: https://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_H13_B650_AM5-UP.cfm
So the OS list seemed very limited to me, and it doesn’t seem this motherboard is intended for general purpose use. Maybe after a few BIOS updates it will have broader UEFI OS support.
I’m returning the Supermicro board and just ordered the Asrock Rack B650D4U. Seems like a lot of folks have success with AsRock, and I’ll try to report back if this specific mobo works.
-
@Niko1 Some motherboards updated their BIOS for Windows 11 compatibility which may have automatically turned on secure boot, etc, etc.
I can't see any other reason Linux ISO's would all of a sudden fail to boot.
-
@Pyroteq said in Any hardware recommendations? - consolidating desktop on server:
@Niko1 Some motherboards updated their BIOS for Windows 11 compatibility which may have automatically turned on secure boot, etc, etc.
I can't see any other reason Linux ISO's would all of a sudden fail to boot.
I’m not really sure. I assumed there must have been some way to get it to work, but given the other issues and Supermicro highlighting the very limited OS support, returning the motherboard seemed like the most prudent action. I got the AsRock board, and it’s working great so far. I’ll post my build and my AsRock experience once I have a chance to test more
-
Just to close this out, here is my build so far. Everything works great. I’ve tried both moonlight and Steam remote play. Steam works much better with my upgraded setup than it did before, so for ease of use, I’ve been using that. I’ve been streaming in 4k without any issues so far
- AsRock Rack B650D4U
- AMD Ryzen 9 7950X - 16-Core
- 64GB ECC RAM Supermicro (Hynix) 32GB 288-Pin DDR5 4800 (PC5-38400) Server Memory (MEM-DR532MD-EU48)
- ZOTAC GAMING GeForce RTX 4070 Twin Edge DLSS 3 12GB
So far, everything works as well as I could expect for game streaming. The performance hit seems minimal. Now I can get rid of my old windows Hades Canyon NUC.
I have an annoying issue when booting. Everything goes through IPMI until grub boots xcp-ng. I suspect it then redirects video to the physical vga port. I’ll post a separate thread for that since it’s not really related to VM gaming.
-
N Niko1 has marked this topic as solved on
-
@Niko1 Glad you're having a good experience with it!
I have more recently done GPU passthrough and it's been fantastic actually so I take back a bit of what I had said before. For some reason in the past I was having major issues with it and I think I may have had defective hardware (very hard to rule out in this situation though).
