PCIe Pass-through lanes and lane performance
-
@JamesG Yeah, its scary that GPU passthrough still has a low priority for XCP-ng, during the boom of AI and local LLM Inferencing. I my self am starting to plan a move to Proxmox, though I do not really wish to move, cause I like XCP way better and have been with them for a long time... But that is just comfort...In the end, everything it comes down to is the capability and performance of the running VM guests.
-
@JamesG, Yea i ended up having to go with a bare metal ubuntu install + docker containers for everything and have been off to the races, I would still like a hypervisor but unfortunately had to move on.
The software stack on these intel's gpus is still maturing but i'm running some pretty impressive models on them and for the money it has been a good investment, hopefully these will gain some popularity and get better support all around.
GPU hosting definitely deserves some attention from xcpng. I would buy some licenses for some production use cases but things like this and nested virtualization (completely unrelated but also something i ran into this past week) needs attention.
-
-
@stormi I dont doubt that the team works hard. The GPU passthrough support has been of lower priority for a long time, it has been said many times before by the team and Oliver. 99 percent of the dev team is working security and basic hypervisor, and is not prioritized for edge cases. Which i guess that PCI passthrough is for GPUs Please dont ask me to dig up the sources. It has always been the case with Xen.... Proxmox, KVM and vmware has the edge there, and that should be well known. But I would in any case where PCI passthrough for edge cases like the GPU marked kind of is, is not a requirement, Always recommend XCP-ng over the others.
-
99 percent of the dev team is working security and basic hypervisor
The numbers don't match any reality though. Yes, maintenance is important, but we have a developer working on AMD iGPU passthrough fixes, and Teddy here is following the Intel Arc topic closely and obtained hardware (waiting for it to be available in the lab). We also work on AMD MxGPU for vGPU support on the supported devices. We also have work on virtio-gpu. Without producing something that can become an actual feature yet, as it's complex, but there's clearly investment on those fronts.
-
But I would in any case where PCI passthrough for edge cases like the GPU marked kind of is, is not a requirement, Always recommend XCP-ng over the others.
Thanks for that :).
There's no problem to state faithfully what works and what doesn't with a solution, and adapt the advice to the actual needs.
I just felt compelled to react because I didn't find it entirely right to state that GPU passthrough has low priority. It's supported in general, but there are devices that come with technical challenges not fully addressed yet in Xen or in XCP-ng, and we treat that as part of the general hardware support we aim for.
Same as we work on device drivers for devices mostly found in home labs, without dismissing them as low priority.
And it's a good point to state that the needs around GPUs are growing fast due to AI, indeed.
-
I get out of my silence to confirm that I'm indeed working on AMD MxGPU driver and AMD iGPU passthrough issues.
I confess that I have not communicated since a while on this work, but I'm making progress and I'm very close to publish a call to testing PoC packages that should improve the iGPU passthrough situation. -
@stormi My offer to give you guys VPN access to an Epyc server with Intel ARC B50 still stands.
IPSec, Wireguard, whatever...You can have IPMI access and at least some basic hardware to get you up and going.
JamesG
-
@JamesG Thanks. I'll let @teddyastie see. I think he would be more confortable with having the card in our lab, which should be imminent now, but I won't speak for him.
-
@stormi I am glad to hear there is progress, and I never doubted that there was interest and will from the Dev team, it was just stated, by Oliver himself, that the focus was mostly on the classic hypervisor features and that resources for edge cases were scarce. The numbers might not be spot on, but it was something similar.. Anyways I know you care and that the team always take interest in home-lab scenarios. And that, is why I keep recommending XCP-ng. You are good and motivated people, you care and interact with the community and you have a product that is simple in structure and free of most of the technical bureaucracy that the other ecosystems have.
And yes, this particular Intel Arc issue, is not only on XCP-ng, The intel toolbox and drivers for this platform is currently a mess for Linux.
And you are right to take home-labbers under your wing and prioritize it. As I'm sure you know, although it might not be of direct importance for revenue, they do have a huge impact on popularity. I dont think Proxmox would have had the traction it has, without them
Thank you for replying, and thank you to the team for their hard work!
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login