PCIe Pass-through lanes and lane performance

john.c

@pandusen said:

@TeddyAstie
Great that you got around to test it.

I did test with the hvm-pirq=0 both pre and post official patch. As you suggested earlier in the thread,
It made no difference for me. I still had the same issues, and the 1x1 link detection.. I tested on Ubuntu, Fedora, Debian, Alma and Rocky. (all headless server)
The only other OS was windows, which was the only OS I could get it to do anything, although heavily borked and slower than expected. Intel driver panel was also complaining about PCI compatibility and reBar

Ubuntu is (as far as I know) the main distro that intel test and sync with.

You should test and see if you can get Level Zero and oneApi going. (Needed for any AI work)
I could never get these to start, driver crashed every time. If you can get that to work on Ubuntu, I would suspect it might be a grouping issue on certain motherboards.

Also, What about the missing audio device not showing in the list?

A part of the reason for it being slower without ReBAR (or Resizable BAR) is due to that technology lifting, the 256 MB limit (for per chunk) on CPU accessing GPU VRAM.

To improve performance when passing through modern GPUs (and graphics cards). With lots of VRAM the limit on accessing GPU VRAM by CPU will really crease the performance of the VM at the minimum. So support for ReBAR present in Xen and XCP-ng would help performance, when passing through the GPU. Basically a virtual form of ReBAR support for VMs, when passing through GPUs.

john.c

@teddyastie I’ve added feedback in regards to the above on Vates Feedback portal, see link below.

https://feedback.vates.tech/posts/123/feature-request-support-for-virtual-resizable-bar-rebar-large-bar-passthrough

john.c

@pandusen Your Windows test results are the perfect real-world example of this exact issue.

When you noted that the Intel driver panel was explicitly complaining about "PCI compatibility and reBar" and that performance was heavily borked, you hit the exact technical bottleneck. Without ReBAR support at the hypervisor layer, the guest OS is forced to interact with that massive 16 GB VRAM space through a rigid 256 MB window, choking it out with massive memory paging overhead.

Even @teddyastie's Linux testing reflects this—noting that host-device transfer bandwidth is "a bit slower than theoretical."

@olivierlambert From a strategic standpoint, as home labs and enterprise environments aggressively pivot toward localized AI, LLM inferencing, and heavy GPU compute, raw PCIe passthrough is shifting from an edge case to a core infrastructure requirement. Implementing a virtualized form of ReBAR support within Xen/XCP-ng would natively lift this 256 MB restriction. It ensures modern GPU silicon can run at 100% efficiency, keeping XCP-ng highly competitive against monolithic hypervisors that handle large BAR mapping.

Especially with VMware VCF 9.0 and up-to-date Hyper-V on Windows Server 2025 advancing their platforms, Vates needs to have virtualised ReBAR support to maintain its edge against this competition.

JamesG

@john.c I would add that in addition to AI and VDI, media functions would be greatly assisted with GPU accelerated encode/decode. I need to do more with less. Having a GPU I could carve up and assign to VM workloads would be amazing.

john.c

@JamesG said:

@john.c I would add that in addition to AI and VDI, media functions would be greatly assisted with GPU accelerated encode/decode. I need to do more with less. Having a GPU I could carve up and assign to VM workloads would be amazing.

@JamesG Absolutely spot on. Media encoding/decoding and VDI density are huge components of this. Being able to slice up a GPU efficiently means VMs need to interact with that VRAM cleanly without hitting performance walls.

Whether it's splitting the card via vGPU profiles or passing it through fully for raw compute, lifting these platform memory limitations is what unlocks that "do more with less" capability for media stacks. It all ties back to how robustly the hypervisor fabric can handle modern GPU memory mapping.

If my feedback on Vates Feedback will help your use case please vote for it.

john.c

@Team-XAPI-Network @bleader Looping in the toolstack side as well. Even if the underlying Xen hypervisor layers align to handle large BAR configurations, XAPI will need the native capabilities to orchestrate and pass these extended memory mapping boundaries down to the VM configuration templates. Without XAPI awareness, we won't be able to easily toggle or manage virtual ReBAR allocations via the CLI or Xen Orchestra.

pandusen

I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.

In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.

I never got to a point of playing with transcoding for media nor AI Inferencing engines, as the tools and drivers never got stable. The drivers were aware of the card and loaded, but as soon as the compute stack tried to talk to it, the driver reset / crashed.

john.c

@pandusen said:

I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.

In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.

Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.

pandusen

@john.c

Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.

Could very well be John. Was fighting for months with this, so my memory is a bit mushy on that part, but i think I got it to map the full 16gb... I later moved my card to a bare metal sidecar, and things just worked. Even though I hacked the bios to activate reBar on an old x99 board.

MajorP93

@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.

In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.

Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?

Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.

pandusen

@MajorP93 said:

@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.

In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.

Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?

Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.

I’m not using the 1x1 as a definitive indicator, just listing it as one of the observed anomalies. Also, this is what the thread is about.

Yes, I ran workloads, but only in Windows. The drivers not working in Linux might or might not be from the audio device not being passed through.

Sure, it might not make sense, but these Arc architectures and drivers are a strange new breed. Some people have claimed, for instance, that the Windows drivers can fail to update the firmware if the audio device isn’t detected.

It’s all speculation. I’m not pointing fingers; I’m simply listing what I’ve observed. Nevertheless, all these points are abnormal and unusual compared to other platforms (AMD and NVIDIA).

MajorP93

@pandusen Are you using "Intel Xe" driver?
I am using that particular Linux in-tree driver (shipped with Ubuntu's linux 7.0 generic flavor) and can confirm that this particular driver works via PCIe passthrough and does not require any sort of audio device.

//EDIT: also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

john.c

@MajorP93 said:

@pandusen Are you using "Intel Xe" driver?
I am using that particular Linux in-tree driver (shipped with Ubuntu's linux 7.0 generic flavor) and can confirm that this particular driver works via PCIe passthrough and does not require any sort of audio device.

//EDIT: also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

@MajorP93 You are completely spot on regarding the PCIe link status reporting anomaly—x1 info in the guest is often just a hypervisor topology quirk and not a real lane bottleneck.

However, a lack of virtualised ReBAR can absolutely cause functional visibility and usability issues for the card, even if the base kernel driver (like the Intel Xe driver you mentioned) loads perfectly fine.

Crucially, fixing this requires a coordinated chain of support—not just at the hypervisor and XAPI layer, but critically within the VM itself. The virtualised EFI BIOS inside the guest must actively support and enable both "Above 4G Decoding" and ReBAR support to successfully map the hardware.

Here is how the lack of virtualised ReBAR breaks things under the hood:

The 256 MB Bottleneck: Without native virtualised ReBAR support at the hypervisor layer and matching EFI BIOS flags inside the VM, the guest OS is forced to map and interact with a massive VRAM pool (e.g., 16 GB on an Arc card) through a rigid, un-resizable 256 MB MMIO window.
API/Tool-Stack Visibility Failures: While the low-level kernel driver recognizes the hardware ID and binds to it, the user-space compute stacks and management tools (like Intel's tool-stacks) will aggressively choke on this memory limitation. The massive memory paging overhead triggers an immediate driver reset or crash the moment a workload initializes.
Functional Invisibility: To the high-level application, AI framework, or orchestration tool, the GPU effectively becomes invisible or non-functional because the compute API communication layer breaks down entirely.

So while you're entirely correct that a headless Ubuntu server doesn't care about a missing audio endpoint, the lack of large BAR orchestration—from XAPI/Xen all the way down to the guest's virtualised EFI environment—remains a hard wall for these newer GPU architectures.

Checking dmesg on the affected guest is definitely the right move here—you will likely see the kernel or driver screaming about failed BAR allocation or page table manipulation failures rather than a simple missing device.