@Niko1

Just to close this out, here is my build so far. Everything works great. I’ve tried both moonlight and Steam remote play. Steam works much better with my upgraded setup than it did before, so for ease of use, I’ve been using that. I’ve been streaming in 4k without any issues so far

• AsRock Rack B650D4U
• AMD Ryzen 9 7950X - 16-Core
• 64GB ECC RAM Supermicro (Hynix) 32GB 288-Pin DDR5 4800 (PC5-38400) Server Memory (MEM-DR532MD-EU48)
• ZOTAC GAMING GeForce RTX 4070 Twin Edge DLSS 3 12GB

So far, everything works as well as I could expect for game streaming. The performance hit seems minimal. Now I can get rid of my old windows Hades Canyon NUC.

I have an annoying issue when booting. Everything goes through IPMI until grub boots xcp-ng. I suspect it then redirects video to the physical vga port. I’ll post a separate thread for that since it’s not really related to VM gaming.

@Niko1

Just to close this out, here is my build so far. Everything works great. I’ve tried both moonlight and Steam remote play. Steam works much better with my upgraded setup than it did before, so for ease of use, I’ve been using that. I’ve been streaming in 4k without any issues so far

• AsRock Rack B650D4U
• AMD Ryzen 9 7950X - 16-Core
• 64GB ECC RAM Supermicro (Hynix) 32GB 288-Pin DDR5 4800 (PC5-38400) Server Memory (MEM-DR532MD-EU48)
• ZOTAC GAMING GeForce RTX 4070 Twin Edge DLSS 3 12GB

So far, everything works as well as I could expect for game streaming. The performance hit seems minimal. Now I can get rid of my old windows Hades Canyon NUC.

I have an annoying issue when booting. Everything goes through IPMI until grub boots xcp-ng. I suspect it then redirects video to the physical vga port. I’ll post a separate thread for that since it’s not really related to VM gaming.

@Pyroteq said in Any hardware recommendations? - consolidating desktop on server:

@Niko1 Some motherboards updated their BIOS for Windows 11 compatibility which may have automatically turned on secure boot, etc, etc.

I can't see any other reason Linux ISO's would all of a sudden fail to boot.

I’m not really sure. I assumed there must have been some way to get it to work, but given the other issues and Supermicro highlighting the very limited OS support, returning the motherboard seemed like the most prudent action. I got the AsRock board, and it’s working great so far. I’ll post my build and my AsRock experience once I have a chance to test more

Just a little update on some of the purchases for folks shopping for compatible hardware. The Supermicro motherboard was a bust. Supermicro H13SAE-MF wouldn’t boot XCP-NG. It initially would boot, but after applying a BIOS update, it wouldn’t boot it at all. A couple of other Linux CDs wouldn’t work either, but I did get Ubuntu to boot. And I had other weird issues where the BIOS would hang at POST if I had my USB DVD drive plugged into certain USB ports instead of others.

The BMC worked ok overall. Interestingly, mounting an ISO image via IPMI requires an additional license, so I used physical CDs or my piKVM. Overall the whole experience felt like I was working with a preproduction board and not a finished product. I emailed Supermicro and they told me only UEFI OSes are supported, and I was using UEFI OSes. They also sent me the OS compatibility chart, which you can find here: https://www.supermicro.com/Aplus/support/resources/OS/OS_Comp_H13_B650_AM5-UP.cfm

So the OS list seemed very limited to me, and it doesn’t seem this motherboard is intended for general purpose use. Maybe after a few BIOS updates it will have broader UEFI OS support.

I’m returning the Supermicro board and just ordered the Asrock Rack B650D4U. Seems like a lot of folks have success with AsRock, and I’ll try to report back if this specific mobo works.

Thanks for all the input. I ended up purchasing most items on my list. I went with the 7950X and an Nvidia 4070 GPU. I'm still waiting for the RAM to arrive, so I tested the 4070 in my current server. Passthrough is working perfectly. Nvidia drivers and software detected the GPU and installed just fine. I wasn't seeing any additional performance in games with the 4070, and I ran some benchmarks. Interestingly, the system is heavily CPU bottlenecked. I wasn't surprised by the bottleneck itself but by the severity of it. The 4070 wasn't doing much better than the old GTX 970. In the benchmarks, CPU utilization is >80% and GPU <50% but usually between 20-40%. So once I get the Ryzen CPU in there, it will hopefully perform close to similar configurations.

I ended up going with the Supermicro since I've had experience with them, and they have provided me with very good support in the past. I generally avoid "gaming" oriented motherboards. I feel that they are more for show than quality. Some of my research on the Asus motherboards seem to indicate this, but it was just anecdotal information. Plus I'm completely turned off by the Asus gaming BIOS screen, no thanks. AsRock I don't have any experience with. I was debating the AsRock board since it includes 10Gbe, but I would prefer SFP28 anyway. So I'll just drop an SFP28 card in sometime in the future when I upgrade my network switch.

The only feature I would like that is currently missing with Supermicro is the AMD Ryzen "eco-mode." I just learned about this recently. It will throttle the CPU from 170w to either 105w or 65w. And the benchmarks that I found online have indicated only about a 5% or so performance penalty at 65w. I emailed Supermicro about it and they are already looking at adding the feature into one of there future BIOS updates.

@Theoi-Meteoroi said in Any hardware recommendations? - consolidating desktop on server:

I looked at desktop systems for some long while. None really do the needful very cheaply. I wanted to do PCI passthru at least and run a few more substantial vms. This meets the requirements and had pretty much everything

Looks like it was never powered on, not a spec of dust inside. Swapped out the supplied windoze disk with another SSD and so far, XCP likes it very much. I am very impressed with what I obtained for ~$1500 with tax.

Rack mount is a requirement for me. I already have a 4U case, but for $1500 that looks pretty good

@planedrop said in Any hardware recommendations? - consolidating desktop on server:

Just wanted to chime in here as I've considered doing this exact thing for years and have never actually managed to be successful enough to go all in.

Not specific to XCP-ng or any issues with it, just GPU passthrough is a huge pain and is likely to create issues down the road so I personally won't do it for my desktop machine as I need that running 100% of the time.

I had enough success with the old RTX 970 that I figured I'd take the plunge. Also, it's time to replace both my old Hades Canyon NUC and upgrade my server, so this seemed like a good time to go all in. If it ends up not working, no harm done. I needed to upgrade the server anyway. My Macbook Pro is my "desktop," so this helps me clear out some extra hardware clutter. I'll make the NUC my edge router running XCP-NG. I'm currently using a 2010 Mac mini, and I'm surprised I got XCP-NG to work on it at all. But the little mac mini is ready to be put down.

Now spec wise, I agree with others, avoid the 7950X3D for this use case, I have one in my desktop and it's absolutely amazing but I wouldn't use it for a server, may just create some complications (though if you can manage to get it working in a way where the gaming machine/desktop runs on the 3D vCache cores, it'd be a beast).

GPU wise, go with NVidia or AMD IMO, not Intel, there is more information on the 2 other brands when it comes to passthrough, NVidia likes to make it difficult whereas AMD tends to be a bit easier. I've done it with NVidia in the past primarily though.

Thanks for the heads up. I did a little more research, and the 7950X3D does have its issues, and the performance is questionable anyway. I went with a 7950X on Black Friday. I wanted an Intel Arc GPU just to not support the Nvidia/AMD duopoly. GPU prices have come down, but they are still ridiculous. I did a little research on the Intel Arc, and they seem to have a similar reset issue as some AMD cards in passthrough. Nvidia seemed like the safest bet, so I nabbed a 4070 on sale. $500 for a "mid range" card, which was more than I wanted to spend. However, I plopped it into my existing Xeon server, and passthrough works flawlessly.

Maybe also a thought would be to consider a server grade GPU and then use vGPU for this to avoid some of the passthrough headache?

I considered this. If it was only a $50 to $100 price premium over a similarly performant GPU, it would be worth the extra cost to save on the headaches. But vGPU cards are about twice the price. Maybe it would make sense if I needed a GPU to support multiple VMs, but I really only need one GPU for one VM.

@ajpri1998 said in Any hardware recommendations? - consolidating desktop on server:

I would personally avoid the 7950X3D. That particular chip has 2 chips with cores (AMD refers to as CCDs). One of the chips has the extra cache, the other does not. This isn't really a bad thing but you have to be more careful to which cores you "pin" your gaming VM to. The non-3D chips still have 2 CCDs, but they are both equal. This consideration in my opinion is very similar to Intel's usage of "P" cores and "E" cores. It works but you have to pay attention to the specific core assignments.

Thanks for highlighting this. I went with the 7950X. I found some reviewers echoing what you said as well.

@lawrencesystems said in Any hardware recommendations? - consolidating desktop on server:

@Niko1

Parts list here: https://kit.co/lawrencesystems/ryzen-9-xcp-ng-virtualization-server-build-with-25gb

Thanks, there are some components I may pull from this list, like that Mellanox card. I'm not going for pure CPU/GPU performance. A stable system for all my other tasks are more important than the gaming VM, so I went with ECC RAM.

Hi there,

I’ve been using XCP-NG 8.3 for a little while now with little issue, and I’ve decided it may be time to upgrade my aging server hardware. I also have a Hades Canyon NUC for gaming, which is also showing its age. One goal is to consolidate my gaming desktop into a VM on my server. I was able to get GPU passthrough to work with my old hardware, and game streaming from the Windows VM was working great (relatively for the hardware), with a few caveats. Gaming isn’t the only thing I want to use my server for. I currently use it for research purposes on various software, OSes, and networks, so I may run a lab environment with multiple VMs at any given time. So if I can consolidate as much as I can to one server, that would be ideal.

My current setup:

• Supermicro (X9SAE)
• Intel Xeon E3-1245, 8 core
• 16 GB RAM
• Nvidia RTX 970
• Removed AMD Radeon 5500, suffered from reset bug

The AMD GPU reset bug caused me several hours of grief until I spent a couple hours on Google and discovered the problem. So no AMD GPUs for my new build. The Nvidia GPU worked with no problems.

8 cores have been just enough for most tasks. But if I’ll be moving my Windows desktop to a VM, that will no longer be the case, so I’ll probably be looking for at least 12 to 16 cores. Also, ECC RAM is still a requirement. I thought I would be limited to Xeon or EPYC processors and forced to make some heavy compromises to try to keep CPU costs reasonable, but AMD Ryzen seems to meet my requirements at a fraction of the cost. The top of the line has 16 cores and supports ECC RAM. I’m considering the AMD Ryzen 9 7950X or 7950X3D. I believe I saw some posts that XCP-NG has some issues with 7950X3D?

Server grade motherboard options that support AMD Ryzen seem to be extremely limited. At a minimum, IOMMU/AMD-V required of course. I believe that’s the minimum to ensure GPU passthrough? I also would Like Resizable BAR support, which is heavily recommended for Intel Arc GPUs. ReBAR will provide the option of being able to use either Intel or Nvidia GPUs since AMD is not an option for GPU passthrough. And I’m heavily inclined to purchase an Intel GPU as my next GPU. ECC RAM support is a must, and IPMI remote management is also a really big nice to have. I have a PiKVM (purchased after a BIOS update hosed the X9SAE’s on board NIC, iSCSI, and Intel AMT), so I can work around not having IPMI. I noticed a new mobo by Supermicro (H13SAE-MF) that seems to check these boxes.

New hardware considered:
Supermicro H13SAE-MF
16 core AMD Ryzen 9 7950X or 7950X3D. Does 7950X3D offer any advantage?
At least 64 GB ECC DDR5 RAM. I’m aiming for 128GB depending on how expensive DDR5 is
Intel or Nvidia GPU

Am I missing anything that may be a showstopper with PCI passthrough, or overall preventing me from using the new setup as described here? Any known compatibility issues with XCP-NG? Have others found any other alternatives? I’ve seen some Asrock recommendations, but do they have good support and reliability? Stability is the priority here, performance is secondary.

Appreciate any thoughts on this configuration and experiences on similar setups

Thanks

Hello, I decided to give the alpha a go, butI must have screwed something up. I upgraded one of my servers from 8.2. Upgrade seemed to go well. But now the console is very laggy and locks up for periods of time. I keep getting errors to the terminal

Broadcast message from systemd-journald@xen-rack (Wed 2023-02-15 20:37:34 EST):

xapi-nbd[2128]: main:   "Failed to log in via xapi's Unix domain socket in 300.000000 seconds")

in /var/log/xensource.log

Feb 15 20:37:13 xen-rack xapi: [debug||0 |bringing up management interface D:45c71627e307|xapi_mgmt_iface] Shutting down the old management interface (if any)
Feb 15 20:37:13 xen-rack xapi: [ info||0 |bringing up management interface D:45c71627e307|xapi_mgmt_iface] Starting new server (listening on all IP addresses)
Feb 15 20:37:13 xen-rack xapi: [ info||0 |bringing up management interface D:45c71627e307|xapi_http] Successfully bound socket to: :::80
Feb 15 20:37:13 xen-rack xapi: [error||0 |bringing up management interface D:45c71627e307|master_connection] Caught Master_connection.Goto_handler
Feb 15 20:37:13 xen-rack xapi: [debug||0 |bringing up management interface D:45c71627e307|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
Feb 15 20:37:13 xen-rack xapi: [error||0 |bringing up management interface D:45c71627e307|master_connection] Connection to master died. I will continue to retry indefinitely (supressing future logging of this message).
Feb 15 20:37:13 xen-rack xapi: [debug||0 |bringing up management interface D:45c71627e307|master_connection] Sleeping 2.000000 seconds before retrying master connection...
Feb 15 20:37:15 xen-rack xapi: [debug||0 |bringing up management interface D:45c71627e307|stunnel] client cert verification 192.168.1.15:443: None
Feb 15 20:37:15 xen-rack xapi: [debug||0 |bringing up management interface D:45c71627e307|master_connection] stunnel: stunnel start\x0A
Feb 15 20:37:15 xen-rack xapi: [debug||0 |bringing up management interface D:45c71627e307|stunnel] Started a client (pid:7816): -> 192.168.1.15:443

Is there something I can do to fix the upgrade?