PCIe Pass-through lanes and lane performance
-
You can't PCI Passthrough a PCI bridge (and even if you could, it wouldn't be very useful), as it's a part of PCI internals and not really something the guest requires to use the device.
As I asked previously, is there any issue (aside display like PCIe 1.0 1x) with the device, is it behavior correctly or having the expected performance ?
-
@TeddyAstie From what I've tried...It won't work at all in Linux.
It kind of works in Windows, and while I do see some load on the GPU when running something, there's as much if not more load on the CPU. It's not what I expected at least.
I've never really gamed on PC, and definitely never Linux. My only uses for GPU's have been in media composition which was always on dedicated systems and Nvidia GPUs. More recently I've been doing some AI tasks, again with passed-through NVidia GPUs.
Side tangent...
I ran into another workload last night where the app wanted OpenGL...But OpenGL doesn't work with the "normal" VGA adapter in XCP-ng, and I don't have a GPU on that particular pool to try a pass-through. That lead me to VirGL, but it doesn't look like that's gone anywhere. -
@andriy.sultanov said:
I'll try to see why other devices are omitted and how this can be fixed.Any news on this?
I have been fighting with getting an Intel Arc Pro B50 to work on Linux for months and i suspect that, the fact that the audio device isn't present, is what makes Linux fail to load the driver correctly.Going the xen-cmdline way, seams to disable xe passthrough both in command line and in XO gui, causing all kinds of issues.
Windows seems to be able to initialise the GPU driver without the audio device, but it is in some kind of borked state. You can send workload to the GPU but the drivers are complaining about "no-pci compatibility".
XCP-ng 8.3 (latest patches)
Fedora server 44 - Kernel 7.1.5-201
on latest Arc firmware as well. -
@pandusen As Teddy said above, you can't passthrough a PCI bridge, so there's no PCI devices xapi shouldn't omit here.
But "going the xen-cmdline way" shouldn't break anything, that's what
xe pci-disable-dom0-accessdoes behind the scenes. What issues did you see? Which steps did you follow? -
@andriy.sultanov It's possible that this is just a fundamental issue with trying to use (maybe just Battlemage) Intel GPU's in XCP-ng.
The card itself appears to be a PCIe bridge in front of the actual GPU hardware on the card. If the driver on the guest requires "the whole thing" sent to it, and you can't pass-through the whole bridge and everything behind it, then there's no real way to use these GPU's in a guest on XCP-ng.
The lack of recent kernel or sufficient backports in the XCP-ng kernel, means the SRIOV features of Intel GPU's are lost as well.
All of this stuff works in Proxmox with seemingly little issues.
James
-
FYI I will get my hands on a Intel Arc Pro B50 to make various tests with it (in the next weeks). If something don't behave as expected, I will try to find a solution.
-
@TeddyAstie I think there's alot of interest in the Intel platform because of the lack of ridiculous licensing for virtualization features. The problem is that you need a pretty current 6.x kernel to use them and XCP-ng/XenServer is just stuck on a really aging kernel right now.
As exposed in this thread, the PCIe pass-through isn't really working properly under XCP-ng.
Even if pass-through worked, that's kinda lame because the real purpose would be to chop up the GPU into SR-IOV instances so multiple workloads could have a GPU.
The Proxmox community is already rockin' and rollin' with this. I watched a video a couple of days ago where a B70 was chopped up into Qty.4, 8GB GPU's via SR-IOV and passed to clients.
No ridiculous mega-$$$$ licensing. It just worked.
-
Thought...If you want access to a server with a B50, I think I could setup VPN access and put a host online with IPMI access for you to do whatever you want with.
You will have an AMD Epyc 7302 with 64GB of ram some flash and HDD storage, and an Intel B50. PM me and we can work out the details of what you need to experiment with, and I can setup some remote access for you to get into the system.
I will gladly work with you guys any which way I can to move this forward.
-
@JamesG Don't bother, I'm in PTO next week; so I may get access to hardware when I'm back.
-
P poddingue marked this topic as a question on
-
@pandusen As Teddy said above, you can't passthrough a PCI bridge, so there's no PCI devices xapi shouldn't omit here.
I am not trying to pass through the bridge only the end points. The Intel arc's have 2 end points: The GPU and the Sound device. "xe pci-list" only reveals the GPU, not the sound device. (this works for nvidia and AMD)
But "going the xen-cmdline way" shouldn't break anything, that's what
xe pci-disable-dom0-accessdoes behind the scenes. What issues did you see? Which steps did you follow?the sound device is available in the lspci list and can be passed through using CLI.
But doing so, (using CLI for passtrough) undoes everything done using xe or the passthrough gui in XO. and results in this:
https://xcp-ng.org/forum/topic/10609/xcp-ng-8.3-pci-passthrough-issue
so yes, its does break something.
-
@jamesg @teddyastie, any further findings on this?
I am building an AI Inference server with a handful of Arc Pro B70's and a Supermicro H12SSL-i MOBO + 128GB of ram. I was hoping to use xcpng as a hypervisor to setup different VMs / software setups, but have also run into roadblocks with pci passthrough.
I'm somewhat of a novice on low level issues like this but if there is any info i can provide to help with the support of these GPUs for passthrough, happy to do so.
Going to give proxmox a try, then bare metal but just wanted to add another somewhat different use case behind the Battlemage cards.
-
@dkidd255 I'm a LOOONG time XCP/XenServer/XCP-ng user in my homelab...And it pains me to say this...
But this just works in Proxmox.
If you want to help us dig in and troubleshoot/explore this topic...Fine.
If you want to actually use your hardware and make progress on your ideas, go to Proxmox.
I seem to recall some recent firmware shenanigans from Intel. Intel maybe thought a little too much of their hardware, or they've decided to just scale things back a bit, but newer versions of firmware have reduced VF's available. I don't think the B70 is impacted by this, but it has massive memory in comparison to the smaller cards.
-
@dkidd255 @jamesg
I didn't forgot about it, but I still don't have access to relevant hardware (for reasons outside of my control).In the meantime, if that happens to be related, can you try the patch that allows disabling hvm-pirq (this is going to be globally available soon) ?
-
@JamesG Yeah, its scary that GPU passthrough still has a low priority for XCP-ng, during the boom of AI and local LLM Inferencing. I my self am starting to plan a move to Proxmox, though I do not really wish to move, cause I like XCP way better and have been with them for a long time... But that is just comfort...In the end, everything it comes down to is the capability and performance of the running VM guests.
-
@JamesG, Yea i ended up having to go with a bare metal ubuntu install + docker containers for everything and have been off to the races, I would still like a hypervisor but unfortunately had to move on.
The software stack on these intel's gpus is still maturing but i'm running some pretty impressive models on them and for the money it has been a good investment, hopefully these will gain some popularity and get better support all around.
GPU hosting definitely deserves some attention from xcpng. I would buy some licenses for some production use cases but things like this and nested virtualization (completely unrelated but also something i ran into this past week) needs attention.
-
-
@stormi I dont doubt that the team works hard. The GPU passthrough support has been of lower priority for a long time, it has been said many times before by the team and Oliver. 99 percent of the dev team is working security and basic hypervisor, and is not prioritized for edge cases. Which i guess that PCI passthrough is for GPUs Please dont ask me to dig up the sources. It has always been the case with Xen.... Proxmox, KVM and vmware has the edge there, and that should be well known. But I would in any case where PCI passthrough for edge cases like the GPU marked kind of is, is not a requirement, Always recommend XCP-ng over the others.
-
99 percent of the dev team is working security and basic hypervisor
The numbers don't match any reality though. Yes, maintenance is important, but we have a developer working on AMD iGPU passthrough fixes, and Teddy here is following the Intel Arc topic closely and obtained hardware (waiting for it to be available in the lab). We also work on AMD MxGPU for vGPU support on the supported devices. We also have work on virtio-gpu. Without producing something that can become an actual feature yet, as it's complex, but there's clearly investment on those fronts.
-
But I would in any case where PCI passthrough for edge cases like the GPU marked kind of is, is not a requirement, Always recommend XCP-ng over the others.
Thanks for that :).
There's no problem to state faithfully what works and what doesn't with a solution, and adapt the advice to the actual needs.
I just felt compelled to react because I didn't find it entirely right to state that GPU passthrough has low priority. It's supported in general, but there are devices that come with technical challenges not fully addressed yet in Xen or in XCP-ng, and we treat that as part of the general hardware support we aim for.
Same as we work on device drivers for devices mostly found in home labs, without dismissing them as low priority.
And it's a good point to state that the needs around GPUs are growing fast due to AI, indeed.
-
I get out of my silence to confirm that I'm indeed working on AMD MxGPU driver and AMD iGPU passthrough issues.
I confess that I have not communicated since a while on this work, but I'm making progress and I'm very close to publish a call to testing PoC packages that should improve the iGPU passthrough situation.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login