Non-server CPU compatibility - Ryzen and Intel
-
@Sam @olivierlambert So.. I spent some time in our little storefront here running some tests. Remember, I'm coming from primarily a VMWare background so I played with some VMWare 8 over the weekend as well for my one team that will not switch pretty much regardless due to their deal with Dell. That said, I grabbed hold of some X670 boards and here is what I found out - and I'm going to do some comparisons with different Hyper-V here.
First, I want to say I would -never- recommend an X670 for Hyper-V. Normally a B650E is fine as you are not (or definitely should not) be overclocking or utilizing many of those features. That said, if you're going X670, it's hard to not say: get a 670E vs. a 670. The reason is the "E" means you have 24 PCI 5 lanes vs 4 on the X670. This effectively allows for better later expansion if you are looking at really utilizing NVME 5.0 or solid-state cards if you are looking at really intensive IOPs but also for future proof use. For example; I have one client who does extensive engineering using XCP-NG where both SolidWorks and Mastercam model works are data stored there; the faster we go the better. So we keep their workstations which are currently all Threadripper Pro 5995WX or Threadripper 3990X. One of the issues is understanding the purpose for which you are planning to both virtualize and the means by which you are needing to generate the IOPs and performance. If your end goal is just a home lab to run Plex or something, then your data and performance needs are different than planning to to fulfill in a setting where excellent server speed matters.
So, with that said, I ran some benches on a few boards that are consumer-oriented just to give some feedback that are X670E.
I tried these boards, utilizing the exact same processor (7900X, non-3D) as I had it on hand and did not feel I needed to open a new processor to test as if I established a baseline it'd be fine. 128GB of DDR5-6000 Corsair RAM, and while I could use onboard video with these processors I have mixed feelings about how that works out as it seems across the board in every Hyperviser to result in not the performance I desired.
So, before I get to XCP-NG, I want to comment on what I found in the other products on the market; I do utilize some of these at different points based on the familiarity sometimes with the onsite staff, as I work as primarily an outside consultant and I manage others or I work to implement policies/procedures/setups/etc.
Microsoft Hyper-V 2019: Microsoft's last HyperV core server, as they will not introduce another per their announcement to all of us; there is no HyperV 2022 stand alone product. This product worked and is improved over 2016 by a great margin, especially once you hit 12 or more high-use VMs that are of mixed OS (With Windows as a backbone). It is also one of the ones that really can address mixing Windows modes very, very well, with fewer issues associated with Windows 2003 and before if you are trying to spin up and recover old OS - something I have had to do more than once in the last few years where clients have discovered data "lost" but an old machine is hanging around; being able to image the old machine (I use Acronis) and image back into here.. I can make that click for NT4.0 and Server 2000, which doesn't always work with others. That said, the performance of Microsoft's Hyper-V trails every other product considerably and despite what you'd think about it being accepting of drivers, I ran into more driver issues including just utilizing Hyper-V inside fo a full install of Server2022 on any of these motherboards than most other OS. I was eventually able to "cram" AMD drivers down it's throat, but without them, many required features were broken, performed poorly, and did the experience was not pleasant.
Proxmox: functional from the beginning, the install went smooth, easy getting it going and connecting to TrueNas Scale as a backup target. I do not in general care for the ProxMox interface which reminds me of VMWare 4.0/5.0 web interface if I remember right.. very sparse; I have always appreciated their ZFS attitude, and performance was about equal to XCP-NG. I'm going to admit, I've always been pretty agnostic between the two products. There are things I really like about both, things I wish would change about both. If you like one, stick with it, if you like the other, stick with it. I feel it's a bit like people who got into fights between Ubuntu, Slackware, Redhat. Shit, I grew up in a basement lab running Sun Spark OS (SunOS) so.. yeah. Stick with what you like. When I get to the performance differences on the boards, I want to note that BOTH XCP-NG and ProxMox had damn near identical performance on these boards.
VMWare 8.0... ok, let me say this now: I like the interface. They've really upgraded, and I've used VMWare for years since the beginning. They've learned alot and they continue to implement some functions that I feel are absolutely unmatched. Then again, they are a huge company working for giant profits and that's what you'd expect. Their appliance level? Fantastic. If you have the $$$$$ and I mean at this point it is big $$$ then it's nice. But they've priced most people out. But here is the thing: I've tried multiple X670 and B650 boards on VMWare 8.0.. stability is non-existent. Crash. Crash. Tried different ram, different ram timings, going down to 4800, and tried turning off all onboard devices. Tried different cards. Nope. Nope. I could not get a stress test that was valid on VMWare 8 on this platform right now. I should note that in testing this I tried against a 2950X Threadripper I had on hand and a 5950X Ryzen I have on the bench, and VMWare was fine; though in one case it did not like the onboard NIC (not unexpected), still.. VMWare's hardware control is still big.
So, here's the performance rundown.
I tried a few boards: the MSI Tomahawk 670E; Asrock 670E Steel Legend; Gigabyte X670E Aorus; Asus X670E Prime-Pro X670E
I did not go for any of the super high-end motherboards, because as a part of a HyperV I am NOT needing a stack of onboard features I'm not going to use and I don't need to pay for a ton of overclocking. Expansion is the only item here that really matters as far as I'm concerned so I looked at M.2 options as well as SATA rust options. Brief rundown:
Asus Prime-Pro: Good news right off the bat was that the Realtek 2.5Gb onboard network adapter worked. I'm used to replacing with Intel 2.5Gb or 10Gbe NICs I have on hand for backup network purposes. Using as my base 128Gb of the Crucial RAM, 2 Seagate IronWolf Pro 14TB, I use an older Corsair Tower case for this test, 750W fully modular PSU (I'm not using a high end GPU, passing through a Quadro K2200, which doesn't even need a power connector). Tested the performance on the network adapter; good performance in the end, really happy with XCP-NGs run at this; performance is significantly (about 15Mb/s so you'll notice) better than MS Hyper-V, and again, VMWare unstable. So good work there. Every test VM ran solid, with stability very good, connectivity good, performance against local storage and run backup solid. I'm running a self-compile XO as I'm just using this in a test. I setup a few VMs as full "emergency recover" VMs and restore them from Acronis image universal restore: BOOM. No issues coming from a Xeon setup so transfer works smoothly. Performance comparison below.
MSI Tomahawk: Again, same Realtek 2.5Gbe, and had no problem right off of the bat. That's good news. I personally preferred the design of the MSI, and its performance is almost identical to the Asus. No real difference in the performance on any test I ran more than anything I couldn't associate with a fluke. Within 1% generally. This is using the newest version. So, pick your favorite.
Asrock Steel Legend: This is where things get interesting. Dual NICs, 1Gbe and 2.5Gbe, both were detected. Using the 7900X I ran into two things worth noting: boot time of both the OS as well as the VMs were considerably faster. I noticed, using the exact same setup, that I was getting a good uplift in both and it was noticeable. In the case of the VMs, the uplift was most significant when handling Windows Server 2019, where the boot time to get into an AD DC was reduced quite a bit. By quite a bit, I mean about 8-11% on each run. In case of getting data to and from TrueNAS, I also had a performance benefit, but this is largely due to the board itself; because of the dual inbuilt NICs, I reserved my management NIC to the 1Gbe and the Storage on the 2.5 and by dividing them out, I wasn't running management traffic over the storage network, able to put them on different ranges; something I generally recommend anyway. The benefits of this configuration is something anyone who has been through any network training coursework knows well.. hell, going back to when Novell was busy teaching me Tokenring, we talked about this, and by the time Windows NT4.0 was really rolling the benefits were obvious. We've known this one forever so it isn't surprising that a board with two supported NICs gives advantages if you want to virtualize. Also a benefit: it's cheaper than the Asus or the Tomahawk.
The Gigabyte option was the most expensive by far; as the X670E was only in the Master version I could find. That said, this is the one board that is not only outrageously more expensive than the others, but you are going to find it is the WORST option for XCP-NG, Proxmox, or anything else. The reason? Drivers. It's the only one that uses the Intel Lan, switching from a Realtek 2.5Gbe to Intel I225, and I had absolutely nothing but problems in making this function worth a damn. I normally recommend against Realtek and in favor of better options, but this adapter began non-functional, requiring me to install a card just to get an install taken care of, then I had to work like hell to try for functionality, and even after I could get some functionality resolved it was so miserably behind the others that I could never use it as a host for anything. Strong avoid; would probably be good for gaming.
Anyway, I had a weekend to waste, and it's always good to look. I will say for those that are thinking about it.. it isn't that hard right now to find companies who are replacing older Threadripper systems; and those machines are absolutely perfect hosts for XCP-NG. I managed to pickup a pair of X399 Motherboard/2950X combos, and I paid less than $150 each. In one case I even got grabbed a nice case with it. Grabbing a prior-gen Xeon or Threadripper for the most part is going to be a far, far better bang for the buck including support, than spending on high-end hardware for most virtualization tasks.
-
Far from being as detailed as everyone above but just to leave some contribution to the 'Non-server' topic, here are some experiences that I've had.
Everything is already outdated, but it might be useful as some reference anyway.
I have a client running a pair of XCP-ngs 8.1 on Ryzen 7 1800X w/ Gigabyte AB350M-GAMING 3 motherboards, Kingston consumer grade SSDs (almost all the write disk activity goes to a NAS), doing HA with halizard nosan version (DRBD to synchronize storage) for years without a hitch. We have even moved the servers to another location, syncing them through an EoIP vpn with virtually no downtime.
And I have just assembled a Ryzen 5700G with an Asus B550M TUF Plus motherboard and XCP-ng 8.2.1 for a homelab. The new AM5 is still a bit expensive in Brazil for a homelab, and I didn't needed too much cpu power anyway, mostly memory and storage. I had an issue with the RTL8125B NIC (just now I noticed the very first message states the same) but adding an Intel NIC made everything perfectly fine.
-
Today I installed a i350-T4 for testing and not depending on the realtek 2.5g card.
As I guess the computer didn't boot after installing the card.
DDR5 5600@C40 max supported on the QVL. Had to clear cmos and reconfigure again the bios so the computer can post again. Set up the L2APIC again, and SR-IOV, IOMMU.Got access to the server through SSH, but the server can't start any VM and have issues with the API not working locally. (XOA LITE) not working neither VMS.
With
xe vm-listsometimes list the vms, even withxe-toolstack-restart.The error:
Error : Connection Refused (calling connect)On the display in the vm list :
("'NoneType' object has no attribute 'xenapi' ", )Also in the xensource.log some erros on the string.
I will dump the logs and copy some over here.
-
I've got some logs:
xapi: [error||0 |starting up database engine D:93df1d58153e|backtrace] dbsync (update_env) R:ff17b7372476 failed with exception Stdlib.Scanf.Scan_failure("scanf: bad input at char number 6: looking for ':', found '.'")Also the xapi keep restarting, and:
Cannot add dependency job for unit lvm2-activation-early.service, ignoring: Unit is masked
Cannont add deppendency job for unit lvm2-activation.service, ignoring: Unit is maskedThis is a test on this machine. I'll clone the disk if you need further info of how to recover.
-
That sounds more like a damaged XAPI DB due to a corrupted disk/record than anything else
-
@olivierlambert that was my thought. I cloned live to a nother SSD, and that SSD reinstall the 8.3 on top of it and updated. But same result. I wanted also to reset the BM but keep the SR with the disks (thin) LVM.
This is a test, for a DR. The VM are not important. I can clone that SSD (nvme) and leave it for testing on the same HW.
If you want to test something, be my guest.
-
If you clone, you can clone the defect/corrupted files.
-
@olivierlambert I just cloned the drive, the issue is the same. I was thinking of cleaning the DB or trying to recover that instance.
I think I also saw on the screen the menu flashing with some colours.
-
All patches which were made available to XCP-ng 8.3 alpha through
yum updateare included directly in the XCP-ng 8.3 beta 1 installation images. -
@stormi
what about support for 8.2 and new CPUs like 7900/7950 ? -
@dave-opc We have internal nightly builds of 8.2 ISOs currently. I plan to publish them, as test ISOs. I don't have the hardware to test, but in theory they should support recent CPUs.
-
@stormi How soon can this be expected?
We planned to purchase a server on this hardware and include it in our 8.2 pool for work tasks.
But so far, the purchase has been suspended until the situation is clarified. -
If you have pro support, please open a ticket so we can see how to prioritize this
-
@dave-opc said in Non-server CPU compatibility - Ryzen and Intel:
@stormi
what about support for 8.2 and new CPUs like 7900/7950 ?I saw this post, but haven't had time to log back in to say anything. We were actually deploying a few virtualizations, and ran into one of the most incredible hacks I've ever seen on a client (I had never, ever heard of someone using AI to hack voice from an externally compromised Zoom call recording before.. until now)
But I've also had time to go through and check a few things out, as it is often easier for me to stress test units that will be for engineering with a hypervisor than it is for me to just throw windows on it and their licensed product and assume things will be good.
Here is what I can tell you: I've had zero issues with the 7900/7950X. Even better news: now that 48MB DDR5 DIMMS are now available, and BIOS update available, if you're using an X670 platform, I've had success at 192GB of RAM. That's a nice boost over 128GB for a platform so that you can get more RAM on board at a lesser cost than hopping to Threadripper, Epyc, or Xeon.
I have NOT had great luck with 7950X3D. Not frequent but not safe for production kernel panic.
-
@tmservo433
Hmm.. I don't see 48 GB modules on sale, only 2x24
Maybe you had the opportunity to compare the performance with the 5900X ? -
@alex821982 Plenty of 48Gb modules/96Gb Kits on the market right now. Here's an example: https://www.amazon.com/Crucial-2x48GB-5600MT-Desktop-CP2K48G56C46U5/dp/B0C79RMMCL/ref=sr_1_1_sspa?crid=9OY04CIGEEZH&keywords=96gb+ddr5&qid=1692893900&sprefix=96gb+ddr5%2Caps%2C143&sr=8-1-spons&sp_csd=d2lkZ2V0TmFtZT1zcF9hdGY&psc=1
Thanks to newer BIOS updates on the X670 boards, making memory work together (as long as same manufacturer/same speeds) is not as difficult. As to your question of between the 5900X.. the real question is what are you doing with your virtualization as a home lab? I generally say this because I don't think of Ryzens as what I use in production atmosphere; that is generally Epyc at this point on my end.
But if you virtualize for say, Plex/HomeAssistant/some Ubuntu/test environments/etc. then you're probably set with the 5900X.. IF, however, you are utilizing it for things that will intensively task passthrough IO and you want to need it (we have a few virtualized workstations running MS Project, and Adobe) then yes, being able to hand out more memory and cores does in fact help; then again, setting up on very fast NVMEs also helps.
Decide for yourself what the goal is. For most, the jump from a 5900x to a 7950 for virtualization is one where it is a "wait".. but the 8000 series is around the corner, and the one really nice thing about AMD is they aren't always switching sockets; so buyers of the X670 are going to be able to grab the next CPU, whereas it appears Intel's 14000 series is going to require new boards/sockets.
-
@tmservo433
Did I understand you correctly that on X670 and 7950 you have version 8.2 running from open access with the latest updates?
And no problems have been noticed in the work?
And according to the memory configuration, did you just say that there is an option with 96+96 or do you use it in your configurations too?
And can I ask what specific models of motherboards and memory you use for this configuration? -
@alex821982 A few notes: this was done using the Asrock Taichi board; at this point, in testing numerous boards, I no longer recommend ASUS boards for anything AM5. Nothing but trouble and their performance significantly lags everyone else. At first, I thought it was just me, but having talked to several others (like Gamer's Nexus/etc.) it seems as though this problem is universal and may be related to something they've done in the design process.
Out of the box, you likely will not be able to support 192Gb. You will need to flash the BIOS using a 16Gb module or something smaller to the newest BIOS. Do not boot directly to 192Gb. Boot to 96Gb, then 192Gb.
The "retrain" mode will take a bit. Your first time you boot is going to take you at least a minute, in my experience.. prepare for a blank screen for a while.
After BIOS has trained memory modules, boot modes are going to be fast and things are going to move exactly as you expect.
I have not had time to try on other AMD boards with this configuration on XCP-NG, as right now I've been spending too much of the later half of my summer in major Acronis and Sage/Mas update rollouts to clients to cover their SQL configurations, and that is what pays the bills..
Still, I'm looking at potentially upgrading my own home machines this fall; and I like the AM5 platform. For reference, I used Crucial 48Gb modules, as I linked above..I think those were the models. There are many others on the market now. In speaking to our AMD partner, I was told that 64Gb single modules should ship Q1 next year, and they expect AM5 to be fully compatible via BIOS update.
-
@tmservo433
Hmm.. interesting information, thanks...
and I was just choosing from ASUS...
Let's see what Asrock offers on AM5
If we consider the configuration with 192 Gb, then only 96Gb DDR5 6800MHz G.Skill 2 x 48 Gb KIT is available to me so far, that is, I can take two such kits, I hope there will be no problems... -
@alex821982 Go to YouTube and anywhere and find their Asus AM5 problems. If you go back in this thread, I started talking about it before they were talking about it there, but I could see some performance problems and we kept running into strange issues. MSI works, but I really just dislike their BIOS. I don't see many Gigabyte around here, and that leaves me Asrock. Now, Asrock's rack product (Asrack) is great stuff, and I will look into that soon enough for homelab purposes.
Right now, to be honest, outside of rolling out that software, we upgraded several PC units there, and took away 5-X99 Xeon PCs and 3-Threadripper 2950X PCs, and all of them are absolutely everything I would need (outside of power hunger) for a homelab test environment.
I can't vouch for that memory kit you are trying, but I'd assume you'd be OK. Do not expect DDR5-8000. Slow it down if you can for stability. You aren't trying to play for gamer-overclocker purposes.
