PCIe Pass-through lanes and lane performance

john.c

@pandusen said:

I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.

In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.

Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.

pandusen

@john.c

Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.

Could very well be John. Was fighting for months with this, so my memory is a bit mushy on that part, but i think I got it to map the full 16gb... I later moved my card to a bare metal sidecar, and things just worked. Even though I hacked the bios to activate reBar on an old x99 board.

MajorP93

@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.

In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.

Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?

Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.

pandusen

@MajorP93 said:

@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.

In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.

Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?

Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.

I’m not using the 1x1 as a definitive indicator, just listing it as one of the observed anomalies. Also, this is what the thread is about.

Yes, I ran workloads, but only in Windows. The drivers not working in Linux might or might not be from the audio device not being passed through.

Sure, it might not make sense, but these Arc architectures and drivers are a strange new breed. Some people have claimed, for instance, that the Windows drivers can fail to update the firmware if the audio device isn’t detected.

It’s all speculation. I’m not pointing fingers; I’m simply listing what I’ve observed. Nevertheless, all these points are abnormal and unusual compared to other platforms (AMD and NVIDIA).

MajorP93

@pandusen Are you using "Intel Xe" driver?
I am using that particular Linux in-tree driver (shipped with Ubuntu's linux 7.0 generic flavor) and can confirm that this particular driver works via PCIe passthrough and does not require any sort of audio device.

//EDIT: also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

john.c

@MajorP93 said:

@pandusen Are you using "Intel Xe" driver?
I am using that particular Linux in-tree driver (shipped with Ubuntu's linux 7.0 generic flavor) and can confirm that this particular driver works via PCIe passthrough and does not require any sort of audio device.

//EDIT: also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

@MajorP93 You are completely spot on regarding the PCIe link status reporting anomaly—x1 info in the guest is often just a hypervisor topology quirk and not a real lane bottleneck.

However, just because the base Intel Xe driver successfully loads on a headless system doesn't mean everything is functioning correctly under the hood. A lack of virtualised ReBAR can absolutely cause functional visibility and stability issues the moment you actual try to put a workload on the card.

Getting this working properly requires a complete end-to-end chain of support—not just at the hypervisor and XAPI layer, but crucially within the VM itself. The virtualised EFI BIOS inside the guest must actively support and enable both "Above 4G Decoding" and ReBAR support to handle the extended memory mapping.

Without that full stack alignment, here is what actually happens:

The 256 MB Bottleneck: Without native virtualised ReBAR support at the hypervisor layer and matching EFI BIOS flags inside the VM, the guest OS is forced to interact with a massive VRAM pool (e.g., 16 GB on an Arc card) through a rigid, un-resizable 256 MB MMIO window.
API/Tool-Stack Visibility Failures: The low-level kernel driver might successfully recognize the hardware ID and sit there quietly while idle. However, the moment user-space compute stacks or management tools attempt to initialize a real workload, they aggressively choke on this memory limitation, triggering an immediate driver reset or crash.
Functional Invisibility: To the high-level application, AI framework, or orchestration tool, the GPU effectively becomes invisible or non-functional because the compute API communication layer breaks down entirely.

Since you mentioned you are running this setup successfully on a headless Ubuntu guest, did you run actual heavy compute/inference workloads and look at the performance data or dmesg output?

If you've managed to bypass the 256 MB bar limitation without virtualised EFI/ReBAR support at the hypervisor layer, sharing your dmesg | grep -i BAR or baseline benchmark data would be incredibly helpful to see how your guest is actually mapping that VRAM. Otherwise, the driver simply loading at idle can be highly misleading.

pandusen

@john.c said:
also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

Yes I did among a ton of other diagnoses, a billion times.. I am talking about findings i did over a long period(months). I am not going to sit down and write a book about something I gave up on almost a month ago. I know what I need to get it working, but I could not get to that state in XCP-ng. I could however get there using Proxmox. But that was on another system though, the x99 6900k. I am running as bare metal now... The former(my current xcp-ng box) was on a ROMED8-2T Epyc Milan. I tested that bare metal on a USB, and it also worked.

Yes Above 4G has to be enabled otherwise reBar wont report a bar-0 or bar-1 at 16gb in linux, I got it to report that at one point (Sorry, I was not aware of the importence at that point, I just thought I had missed something, and was at a perfectly normal point in the process), but the guest driver still had issues(So it was no success anyway). All this led me to other findings, both here and in other discussions about people not succeeding with intel pass-through in xcp-ng. Again, I only mentioned those points, because I could see that Teddy, basically hadn't tested it beyond what I had already succeed at. "speculation" I am sry, but that was really I could get to.. I have searched everywhere, I have had Claud and GPT chew on it and looked through Intel's documention as well as my own hardware. The only thing I couldnt see, was what was happening under the xen hood.

I will look through my flyby notes and if I find something of interest, or perhaps in regards to getting that reBar to report 16gb on the guest, I will post it here!

My current Ubuntu bare metal, works as expected, I get a 16x8 on PCI v3 wich is expected from the 8 lanes electrical of the Arc pro b50. reBar is working and reporting correctly, and the performance numbers match what others have measured in a healthy system. Yes I have run plenty of heavy inference and compute.. mostly inference.. I have not used it for transcoding media yet, as that was a job it was suppose to have on the xcp-ng server.. It is even running a rtx 2080 card in the same box as well, which is running isolated along side the intel card and they are juggling data back and forth every day. No issues.

But as I also said, I might suspect PCI grouping of certain motherboards, not playing well with xcp-ng, which might explain your "Its working on my computer" phenomenon.

I am not currently at a point where I feel like taking my AI server apart and put the Intel card back in the xcp-ng server.. Maybe when something has actually changed, I would be willing to give it a shot again and I promise at that point, I will be sure to have some notes with me.

john.c

@pandusen said:

@john.c said:
also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

Yes I did among a ton of other diagnoses, a billion times.. I am talking about findings i did over a long period(months). I am not going to sit down and write a book about something I gave up on almost a month ago. I know what I need to get it working, but I could not get to that state in XCP-ng. I could however get there using Proxmox. But that was on another system though, the x99 6900k. I am running as bare metal now... The former(my current xcp-ng box) was on a ROMED8-2T Epyc Milan. I tested that bare metal on a USB, and it also worked.

Yes Above 4G has to be enabled otherwise reBar wont report a bar-0 or bar-1 at 16gb in linux, I got it to report that at one point (Sorry, I was not aware of the importence at that point, I just thought I had missed something, and was at a perfectly normal point in the process), but the guest driver still had issues(So it was no success anyway). All this led me to other findings, both here and in other discussions about people not succeeding with intel pass-through in xcp-ng. Again, I only mentioned those points, because I could see that Teddy, basically hadn't tested it beyond what I had already succeed at. "speculation" I am sry, but that was really I could get to.. I have searched everywhere, I have had Claud and GPT chew on it and looked through Intel's documention as well as my own hardware. The only thing I couldnt see, was what was happening under the xen hood.

I will look through my flyby notes and if I find something of interest, or perhaps in regards to getting that reBar to report 16gb on the guest, I will post it here!

My current Ubuntu bare metal, works as expected, I get a 16x8 on PCI v3 wich is expected from the 8 lanes electrical of the Arc pro b50. reBar is working and reporting correctly, and the performance numbers match what others have measured in a healthy system. Yes I have run plenty of heavy inference and compute.. mostly inference.. I have not used it for transcoding media yet, as that was a job it was suppose to have on the xcp-ng server.. It is even running a rtx 2080 card in the same box as well, which is running isolated along side the intel card and they are juggling data back and forth every day. No issues.

But as I also said, I might suspect PCI grouping of certain motherboards, not playing well with xcp-ng, which might explain your "Its working on my computer" phenomenon.

I am not currently at a point where I feel like taking my AI server apart and put the Intel card back in the xcp-ng server.. Maybe when something has actually changed, I would be willing to give it a shot again and I promise at that point, I will be sure to have some notes with me.

@pandusen Absolutely spot on regarding the bare-metal and Proxmox testing—that narrows the bottleneck down completely to how the Xen hypervisor layer and XAPI orchestrate the virtualized memory topology down to the guest VM.

Since the underlying XCP-ng OVMF (UEFI) builds do technically support ReBAR, the driver resets and oneAPI/compute-stack crashes under load usually mean the guest thinks it has a large BAR but is hitting a hard page fault or timing out the moment a workload tries to stream data past the old 256MB boundary.

As an aside, what CPU did you use in the ROMED8-2T Epyc Milan server? Ideally, what is the exact model (akin to how someone would specify an AMD Ryzen 7 5700X on the desktop side, such as an EPYC 73F3, 7543, etc.)? I'm curious if a specific core-die layout or NUMA topology is introducing firmware-level quirks with large BAR allocations under Xen.

If you ever feel like dropping the Intel card back into the ROMED8-2T for another round of testing, here are the exact diagnostic knobs to turn to see if we can get the Xen hypervisor to behave:

1. Force Xen to Re-allocate Host Memory BARs

Xen often locks down the host’s initial PCIe memory allocations statically, which can prevent a guest from cleanly widening its memory map. You can force the Xen microkernel to actively re-balance and open up contiguous MMIO holes by appending this to your Xen command line:

/opt/xensource/libexec/xen-cmdline --set-xen "pci=realloc"

2. Lock Down the EPYC Infinity Fabric (Prevent FLR Timeouts)

On the ASRock ROMED8-2T, Milan's dynamic power management frequently shifts Infinity Fabric clock speeds depending on load. While normal for enterprise NICs, these transient latency shifts can cause sensitive, first-generation Intel Arc compute drivers to panic and trigger an immediate Function Level Reset (FLR).

The Fix: In the motherboard BIOS, navigate to AMD CBS → Workload Profile and flip it to "NIC Throughput Intensive". This locks the Infinity Fabric P-States and prevents the root complex from modulating performance lanes mid-calculation.

3. Ensure Strict Multi-Function Passthrough

You mentioned speculation about the companion audio endpoint. Passing only the VGA function of a multi-function card frequently causes severe kernel driver resets if the guest driver tries to initiate a hardware handshake with its missing audio sibling. Ensure both endpoints are explicitly hidden from Dom0 and bound to the same template slot:

# Check lspci to grab your exact dual IDs (e.g., 0b:00.0 and 0b:00.1)
/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:0b:00.0)(0000:0b:00.1)"

4. Check the Guest's Internal Memory Mapping Map

If the compute stack chokes, running a quick log dump inside the headless Ubuntu guest right after a crash will tell us if the virtualized page tables are failing:

dmesg | grep -i -E "bar|xe|i915|memory"

Look closely to see if the guest logs a successful BAR 2 resized to 16384MiB or if it explicitly throws a failed to allocate/map BAR warning. If it claims a 16GB allocation but still faults, the hypervisor isn't dynamically updating the virtual page structures under pressure.

If you ever get around to digging out your flyby notes or throwing the card back into the server, let me know what the guest kernel throws back!

pandusen

@john.c
Oh, that is most certainly an AI response.. No wonder I was frustrated with the questions.
Why do you feel obligated to infuse this slop into a thread on a support forum, with no regard of it being helpful or not?

poddingue

@pandusen, I understand your frustration, but please focus on the content, not the person. Also, note that the request for dmesg output came from @MajorP93.

@john.c, if you use a tool, please check its commands before posting, and try to keep your posts shorter.

On the content: step 1 of @john.c's advice is xen-cmdline --set-xen "pci=realloc", and that part is wrong. AFAIK, Xen's pci= option only accepts serr, perr and quirks (Xen command line docs); realloc isn't a valid Xen option. pci=realloc is a Linux kernel option, right?

It's okay to be wrong; I often am too. The important thing is not to be overly assertive when you're not sure you have the right answer.

Teddy's answer is sound, which is why I marked this thread as solved. However, the thread isn't locked, so it remains open for the Arc and ReBAR follow-up.

Thanks, folks, let's keep a friendly and helpful atmosphere.