PCIe Pass-through lanes and lane performance
-
@teddyastie I’ve added feedback in regards to the above on Vates Feedback portal, see link below.
-
@pandusen Your Windows test results are the perfect real-world example of this exact issue.
When you noted that the Intel driver panel was explicitly complaining about "PCI compatibility and reBar" and that performance was heavily borked, you hit the exact technical bottleneck. Without ReBAR support at the hypervisor layer, the guest OS is forced to interact with that massive 16 GB VRAM space through a rigid 256 MB window, choking it out with massive memory paging overhead.
Even @teddyastie's Linux testing reflects this—noting that host-device transfer bandwidth is "a bit slower than theoretical."
@olivierlambert From a strategic standpoint, as home labs and enterprise environments aggressively pivot toward localized AI, LLM inferencing, and heavy GPU compute, raw PCIe passthrough is shifting from an edge case to a core infrastructure requirement. Implementing a virtualized form of ReBAR support within Xen/XCP-ng would natively lift this 256 MB restriction. It ensures modern GPU silicon can run at 100% efficiency, keeping XCP-ng highly competitive against monolithic hypervisors that handle large BAR mapping.
Especially with VMware VCF 9.0 and up-to-date Hyper-V on Windows Server 2025 advancing their platforms, Vates needs to have virtualised ReBAR support to maintain its edge against this competition.
-
@john.c I would add that in addition to AI and VDI, media functions would be greatly assisted with GPU accelerated encode/decode. I need to do more with less. Having a GPU I could carve up and assign to VM workloads would be amazing.
-
@john.c I would add that in addition to AI and VDI, media functions would be greatly assisted with GPU accelerated encode/decode. I need to do more with less. Having a GPU I could carve up and assign to VM workloads would be amazing.
@JamesG Absolutely spot on. Media encoding/decoding and VDI density are huge components of this. Being able to slice up a GPU efficiently means VMs need to interact with that VRAM cleanly without hitting performance walls.
Whether it's splitting the card via vGPU profiles or passing it through fully for raw compute, lifting these platform memory limitations is what unlocks that "do more with less" capability for media stacks. It all ties back to how robustly the hypervisor fabric can handle modern GPU memory mapping.
If my feedback on Vates Feedback will help your use case please vote for it.
-
@Team-XAPI-Network @bleader Looping in the toolstack side as well. Even if the underlying Xen hypervisor layers align to handle large BAR configurations, XAPI will need the native capabilities to orchestrate and pass these extended memory mapping boundaries down to the VM configuration templates. Without XAPI awareness, we won't be able to easily toggle or manage virtual ReBAR allocations via the CLI or Xen Orchestra.
-
I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.
In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.
I never got to a point of playing with transcoding for media nor AI Inferencing engines, as the tools and drivers never got stable. The drivers were aware of the card and loaded, but as soon as the compute stack tried to talk to it, the driver reset / crashed.
-
I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.
In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.
Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.
-
Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.
Could very well be John. Was fighting for months with this, so my memory is a bit mushy on that part, but i think I got it to map the full 16gb... I later moved my card to a bare metal sidecar, and things just worked. Even though I hacked the bios to activate reBar on an old x99 board.
-
@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.
In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.
Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?
Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.
-
@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.
In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.
Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?
Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.
I’m not using the 1x1 as a definitive indicator, just listing it as one of the observed anomalies. Also, this is what the thread is about.
Yes, I ran workloads, but only in Windows. The drivers not working in Linux might or might not be from the audio device not being passed through.
Sure, it might not make sense, but these Arc architectures and drivers are a strange new breed. Some people have claimed, for instance, that the Windows drivers can fail to update the firmware if the audio device isn’t detected.
It’s all speculation. I’m not pointing fingers; I’m simply listing what I’ve observed. Nevertheless, all these points are abnormal and unusual compared to other platforms (AMD and NVIDIA).
-
@pandusen Are you using "Intel Xe" driver?
I am using that particular Linux in-tree driver (shipped with Ubuntu's linux 7.0 generic flavor) and can confirm that this particular driver works via PCIe passthrough and does not require any sort of audio device.//EDIT: also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case
-
This post is deleted! -
@john.c said:
also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this caseYes I did among a ton of other diagnoses, a billion times.. I am talking about findings i did over a long period(months). I am not going to sit down and write a book about something I gave up on almost a month ago. I know what I need to get it working, but I could not get to that state in XCP-ng. I could however get there using Proxmox. But that was on another system though, the x99 6900k. I am running as bare metal now... The former(my current xcp-ng box) was on a ROMED8-2T Epyc Milan. I tested that bare metal on a USB, and it also worked.
Yes Above 4G has to be enabled otherwise reBar wont report a bar-0 or bar-1 at 16gb in linux, I got it to report that at one point (Sorry, I was not aware of the importence at that point, I just thought I had missed something, and was at a perfectly normal point in the process), but the guest driver still had issues(So it was no success anyway). All this led me to other findings, both here and in other discussions about people not succeeding with intel pass-through in xcp-ng. Again, I only mentioned those points, because I could see that Teddy, basically hadn't tested it beyond what I had already succeed at. "speculation" I am sry, but that was really I could get to.. I have searched everywhere, I have had Claud and GPT chew on it and looked through Intel's documention as well as my own hardware. The only thing I couldnt see, was what was happening under the xen hood.
I will look through my flyby notes and if I find something of interest, or perhaps in regards to getting that reBar to report 16gb on the guest, I will post it here!
My current Ubuntu bare metal, works as expected, I get a 16x8 on PCI v3 wich is expected from the 8 lanes electrical of the Arc pro b50. reBar is working and reporting correctly, and the performance numbers match what others have measured in a healthy system. Yes I have run plenty of heavy inference and compute.. mostly inference.. I have not used it for transcoding media yet, as that was a job it was suppose to have on the xcp-ng server.. It is even running a rtx 2080 card in the same box as well, which is running isolated along side the intel card and they are juggling data back and forth every day. No issues.
But as I also said, I might suspect PCI grouping of certain motherboards, not playing well with xcp-ng, which might explain your "Its working on my computer" phenomenon.
I am not currently at a point where I feel like taking my AI server apart and put the Intel card back in the xcp-ng server.. Maybe when something has actually changed, I would be willing to give it a shot again and I promise at that point, I will be sure to have some notes with me.
-
This post is deleted! -
@john.c
Oh, that is most certainly an AI response.. No wonder I was frustrated with the questions.
Why do you feel obligated to infuse this slop into a thread on a support forum, with no regard of it being helpful or not? -
P poddingue has marked this topic as solved
-
@pandusen, I understand your frustration, but please focus on the content, not the person. Also, note that the request for
dmesgoutput came from @MajorP93.@john.c, if you use a tool, please check its commands before posting, and try to keep your posts shorter.
On the content: step 1 of @john.c's advice is
xen-cmdline --set-xen "pci=realloc", and that part is wrong. AFAIK, Xen'spci=option only acceptsserr,perrandquirks(Xen command line docs);reallocisn't a valid Xen option.pci=reallocis a Linux kernel option, right?It's okay to be wrong; I often am too. The important thing is not to be overly assertive when you're not sure you have the right answer.
Teddy's answer is sound, which is why I marked this thread as solved. However, the thread isn't locked, so it remains open for the Arc and ReBAR follow-up.
Thanks, folks, let's keep a friendly and helpful atmosphere.
-
OP here...
So by saying this is "solved" then there's nothing specific or special to do in XCP-ng to make these work in pass-through to guests and that everything related to any operational or performance issues is directly a guest OS issue?
-
@JamesG, I'm the one who marked it solved, so here's what I meant.
What Teddy showed in https://xcp-ng.org/forum/post/109066 is that there's one thing to do on the XCP-ng side for the B50: turn HVM PIRQ off for that VM (
xe vm-param-set uuid=<vm-uuid> platform:hvm-pirq=false, then reboot the VM), as described on the docs page he linked: https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7.With that, his benchmark came out normal for the card, and the Gen1 x1 you see inside the guest is something QEMU makes up, not the real link (his earlier post: https://xcp-ng.org/forum/post/106479). It doesn't mean every problem is on the guest side though: @pandusen tried the same setting (https://xcp-ng.org/forum/post/109084) and still had driver crashes on headless distros, and I don't know why. If you try it on your B50 and post what you see, that would help tell the two cases apart.
-
Did some investigations.
TL;DR: You have Resized BARs for the Intel Arc (and any device) in VMs, iff you have Resizable BAR feature enabled in the firmware.
64-bits BAR works inside guests. BARs too large to fit in the legacy area ("MMIO hole") are going to be placed over TOM (above guest RAM).
However, the logic to resize the BAR dynamically isn't available in the guest. If the device BAR starts at 256 MB, the guest won't be able to resize it.When Resizable BAR feature is enabled in the firmware, device BARs that can be resized are resized to their maximum early at boot, hence the device will have large BARs in XCP-ng.
Such BAR are passed through entirely, iow, you have resized BARs in the guest, this shows up in @jamesg guest lspci https://xcp-ng.org/forum/topic/12316/pcie-pass-through-lanes-and-lane-performance/12?_=1793630786768In cases firmware has no support for "Resizable BAR" but has support for 64-bits BARs, Resizable BAR can be triggered manually from Dom0 through
/sys/bus/pci/devices/.../resourceN_resize(yet, it's still going to be tricky to use in practice), but that's not a kernel feature available as of XCP-ng 8.3. -
I'm also investigating on a solution to make XCP-ng opportunistically resize device BAR; in case the firmware didn't do it (or has no support for it).
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login