PCIe Pass-through lanes and lane performance

JamesG

@stormi My offer to give you guys VPN access to an Epyc server with Intel ARC B50 still stands.

IPSec, Wireguard, whatever...You can have IPMI access and at least some basic hardware to get you up and going.

JamesG

stormi

@JamesG Thanks. I'll let @teddyastie see. I think he would be more confortable with having the card in our lab, which should be imminent now, but I won't speak for him.

pandusen

@stormi I am glad to hear there is progress, and I never doubted that there was interest and will from the Dev team, it was just stated, by Oliver himself, that the focus was mostly on the classic hypervisor features and that resources for edge cases were scarce. The numbers might not be spot on, but it was something similar.. Anyways I know you care and that the team always take interest in home-lab scenarios. And that, is why I keep recommending XCP-ng. You are good and motivated people, you care and interact with the community and you have a product that is simple in structure and free of most of the technical bureaucracy that the other ecosystems have.

And yes, this particular Intel Arc issue, is not only on XCP-ng, The intel toolbox and drivers for this platform is currently a mess for Linux.

And you are right to take home-labbers under your wing and prioritize it. As I'm sure you know, although it might not be of direct importance for revenue, they do have a huge impact on popularity. I dont think Proxmox would have had the traction it has, without them

Thank you for replying, and thank you to the team for their hard work!

TeddyAstie

I finally got access to a machine with Intel Arc machine.

There are issues around hvm-pirq, hence hvm-pirq=0 is required (see https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7).

For some reason, Fedora Workstation has a issue with this GPU (a Fedora bug I guess), but CachyOS (desktop iso) works fine.

Ran a clpeak bench, and performance looks normal for this class of GPU.
Host-device transfer is a bit slower than "theorical", but definetely not PCIe 1x1 and more PCIe 3x8-ish (matching reality as it's plugged as PCIe 3.0).

in-VM inxi -CGM

Machine:
  Type: Xen System: Xen product: HVM domU v: 4.17 serial: 2c66cb09-57db-340c-ad8e-f505eb25ca3e
  Mobo: N/A model: N/A serial: N/A Firmware: UEFI vendor: Xen v: 4.17 date: 10/06/2026
CPU:
  Info: 6x 1-core model: Intel Xeon Silver 4112 bits: 64 type: SMP cache: L2: 6x 1024 KiB (6 MiB)
  Speed (MHz): avg: 2594 min/max: N/A cores: 1: 2594 2: 2594 3: 2594 4: 2594 5: 2594 6: 2594
Graphics:
  Device-1: driver: bochs-drm v: N/A
  Device-2: Intel Battlemage G21 [Arc Pro B50] driver: xe v: kernel
  Display: unspecified server: Xwayland v: 24.1.13 driver: gpu: bochs-drm tty: 212x48
  API: EGL v: 1.5 drivers: iris,swrast platforms: gbm,surfaceless,device
  API: OpenGL v: 4.6 vendor: mesa v: 26.1.6-arch3.1 note: console (EGL sourced) renderer: Mesa
    Intel Arc Pro B50 Graphics (BMG G21), llvmpipe (LLVM 22.1.8 256 bits)
  API: Vulkan v: 1.4.357 drivers: intel surfaces: N/A
  Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo de: kscreen-console,kscreen-doctor
    gpu: gputop, intel_gpu_top, lsgpu, nvidia-smi wl: wayland-info x11: xdpyinfo, xprop, xrandr

Raw numbers

Backend: Vulkan
  Device 0: Intel(R) Arc(tm) Pro B50 Graphics (BMG G21)
    Driver version   : 26.2.4
    API version      : 1.4.354
    VRAM             : 16048 MB

    Single-precision compute
      float    : 9.47 TFLOPS
      float2   : 9.48 TFLOPS
      float4   : 9.52 TFLOPS

    Half-precision compute fp16xfp16+fp16
      half     : 18.3 TFLOPS
      half2    : 18.8 TFLOPS
      half4    : 19.1 TFLOPS

    Double-precision compute
      double   : 650 GFLOPS
      double2  : 647 GFLOPS
      double4  : 640 GFLOPS

    Mixed-precision compute fp16xfp16+fp32
      mp       : 9.31 TFLOPS
      mp2      : 9.32 TFLOPS
      mp4      : 9.42 TFLOPS

    BF16 compute bf16xbf16+fp32
      bf16     : 9.17 TFLOPS
      bf16_2   : 9.30 TFLOPS
      bf16_4   : 9.28 TFLOPS

    Integer compute int32
      int      : 1.81 TOPS
      int2     : 1.88 TOPS
      int4     : 1.82 TOPS

    INT8 dot-product compute
      int8_dp  : 28.0 TOPS
      int8_dp2 : 26.3 TOPS
      int8_dp4 : 27.1 TOPS

    Cooperative matrix
      fp16     : 67.3 TFLOPS
      fp16 f16acc : 68.9 TFLOPS
      bf16        : 68.9 TFLOPS
      int8        : 141 TOPS

    Global memory bandwidth
      float    : 196 GB/s
      float2   : 206 GB/s
      float4   : 212 GB/s

    Local memory bandwidth
      float    : 2.55 TB/s
      float2   : 3.65 TB/s
      float4   : 4.23 TB/s

    Image memory bandwidth
      float4   : 213 GB/s

    Transfer bandwidth
      h2d      : 4.81 GB/s
      d2h      : 5.48 GB/s

    Kernel launch latency
      dispatch : 48.7 us
      roundtrip : 113 us

Real performance is much higher than the "displayed" Speed 2.5GT/s, Width x1.

pandusen

@TeddyAstie
Great that you got around to test it.

I did test with the hvm-pirq=0 both pre and post official patch. As you suggested earlier in the thread,
It made no difference for me. I still had the same issues, and the 1x1 link detection.. I tested on Ubuntu, Fedora, Debian, Alma and Rocky. (all headless server)
The only other OS was windows, which was the only OS I could get it to do anything, although heavily borked and slower than expected. Intel driver panel was also complaining about PCI compatibility and reBar

Ubuntu is (as far as I know) the main distro that intel test and sync with.

You should test and see if you can get Level Zero and oneApi going. (Needed for any AI work)
I could never get these to start, driver crashed every time. If you can get that to work on Ubuntu, I would suspect it might be a grouping issue on certain motherboards.

Also, What about the missing audio device not showing in the list?

john.c

@pandusen said:

@TeddyAstie
Great that you got around to test it.

I did test with the hvm-pirq=0 both pre and post official patch. As you suggested earlier in the thread,
It made no difference for me. I still had the same issues, and the 1x1 link detection.. I tested on Ubuntu, Fedora, Debian, Alma and Rocky. (all headless server)
The only other OS was windows, which was the only OS I could get it to do anything, although heavily borked and slower than expected. Intel driver panel was also complaining about PCI compatibility and reBar

Ubuntu is (as far as I know) the main distro that intel test and sync with.

You should test and see if you can get Level Zero and oneApi going. (Needed for any AI work)
I could never get these to start, driver crashed every time. If you can get that to work on Ubuntu, I would suspect it might be a grouping issue on certain motherboards.

Also, What about the missing audio device not showing in the list?

A part of the reason for it being slower without ReBAR (or Resizable BAR) is due to that technology lifting, the 256 MB limit (for per chunk) on CPU accessing GPU VRAM.

To improve performance when passing through modern GPUs (and graphics cards). With lots of VRAM the limit on accessing GPU VRAM by CPU will really crease the performance of the VM at the minimum. So support for ReBAR present in Xen and XCP-ng would help performance, when passing through the GPU. Basically a virtual form of ReBAR support for VMs, when passing through GPUs.

john.c

@teddyastie I’ve added feedback in regards to the above on Vates Feedback portal, see link below.

https://feedback.vates.tech/posts/123/feature-request-support-for-virtual-resizable-bar-rebar-large-bar-passthrough

john.c

@pandusen Your Windows test results are the perfect real-world example of this exact issue.

When you noted that the Intel driver panel was explicitly complaining about "PCI compatibility and reBar" and that performance was heavily borked, you hit the exact technical bottleneck. Without ReBAR support at the hypervisor layer, the guest OS is forced to interact with that massive 16 GB VRAM space through a rigid 256 MB window, choking it out with massive memory paging overhead.

Even @teddyastie's Linux testing reflects this—noting that host-device transfer bandwidth is "a bit slower than theoretical."

@olivierlambert From a strategic standpoint, as home labs and enterprise environments aggressively pivot toward localized AI, LLM inferencing, and heavy GPU compute, raw PCIe passthrough is shifting from an edge case to a core infrastructure requirement. Implementing a virtualized form of ReBAR support within Xen/XCP-ng would natively lift this 256 MB restriction. It ensures modern GPU silicon can run at 100% efficiency, keeping XCP-ng highly competitive against monolithic hypervisors that handle large BAR mapping.

Especially with VMware VCF 9.0 and up-to-date Hyper-V on Windows Server 2025 advancing their platforms, Vates needs to have virtualised ReBAR support to maintain its edge against this competition.

JamesG

@john.c I would add that in addition to AI and VDI, media functions would be greatly assisted with GPU accelerated encode/decode. I need to do more with less. Having a GPU I could carve up and assign to VM workloads would be amazing.

john.c

@JamesG said:

@john.c I would add that in addition to AI and VDI, media functions would be greatly assisted with GPU accelerated encode/decode. I need to do more with less. Having a GPU I could carve up and assign to VM workloads would be amazing.

@JamesG Absolutely spot on. Media encoding/decoding and VDI density are huge components of this. Being able to slice up a GPU efficiently means VMs need to interact with that VRAM cleanly without hitting performance walls.

Whether it's splitting the card via vGPU profiles or passing it through fully for raw compute, lifting these platform memory limitations is what unlocks that "do more with less" capability for media stacks. It all ties back to how robustly the hypervisor fabric can handle modern GPU memory mapping.

If my feedback on Vates Feedback will help your use case please vote for it.

john.c

@Team-XAPI-Network @bleader Looping in the toolstack side as well. Even if the underlying Xen hypervisor layers align to handle large BAR configurations, XAPI will need the native capabilities to orchestrate and pass these extended memory mapping boundaries down to the VM configuration templates. Without XAPI awareness, we won't be able to easily toggle or manage virtual ReBAR allocations via the CLI or Xen Orchestra.

pandusen

I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.

In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.

I never got to a point of playing with transcoding for media nor AI Inferencing engines, as the tools and drivers never got stable. The drivers were aware of the card and loaded, but as soon as the compute stack tried to talk to it, the driver reset / crashed.

john.c

@pandusen said:

I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.

In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.

Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.

pandusen

@john.c

Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.

Could very well be John. Was fighting for months with this, so my memory is a bit mushy on that part, but i think I got it to map the full 16gb... I later moved my card to a bare metal sidecar, and things just worked. Even though I hacked the bios to activate reBar on an old x99 board.

MajorP93

@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.

In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.

Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?

Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.

pandusen

@MajorP93 said:

@pandusen I think @teddyastie 's test was about proving that the displayed link does not reflect the actual performance.

In his case the link is also displayed as "x1" but the benchmark data shows that in reality more than 1 lane is being used.

Maybe you should stop using the 1x1 information in your systems as an indicator that passthrough is fully broken here. Did you run actual workloads and looked at performance data after applying the tweaks mentioned by @teddyastie ?

Also the statement that the drivers refuses to load due to an audio device does not really sound right. I am using Intel Xe driver on a headless Ubuntu system via GPU passthrough myself. No one needs audio on a headless server. Why would Intel built such a requirement (audio device being there) into their driver? Does not really make sense.

I’m not using the 1x1 as a definitive indicator, just listing it as one of the observed anomalies. Also, this is what the thread is about.

Yes, I ran workloads, but only in Windows. The drivers not working in Linux might or might not be from the audio device not being passed through.

Sure, it might not make sense, but these Arc architectures and drivers are a strange new breed. Some people have claimed, for instance, that the Windows drivers can fail to update the firmware if the audio device isn’t detected.

It’s all speculation. I’m not pointing fingers; I’m simply listing what I’ve observed. Nevertheless, all these points are abnormal and unusual compared to other platforms (AMD and NVIDIA).

MajorP93

@pandusen Are you using "Intel Xe" driver?
I am using that particular Linux in-tree driver (shipped with Ubuntu's linux 7.0 generic flavor) and can confirm that this particular driver works via PCIe passthrough and does not require any sort of audio device.

//EDIT: also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

john.c

@MajorP93 said:

@pandusen Are you using "Intel Xe" driver?
I am using that particular Linux in-tree driver (shipped with Ubuntu's linux 7.0 generic flavor) and can confirm that this particular driver works via PCIe passthrough and does not require any sort of audio device.

//EDIT: also did you check "dmesg" output on the affected guests? real error messages are more interesting than speculation in this case

@MajorP93 You are completely spot on regarding the PCIe link status reporting anomaly—x1 info in the guest is often just a hypervisor topology quirk and not a real lane bottleneck.

However, just because the base Intel Xe driver successfully loads on a headless system doesn't mean everything is functioning correctly under the hood. A lack of virtualised ReBAR can absolutely cause functional visibility and stability issues the moment you actual try to put a workload on the card.

Getting this working properly requires a complete end-to-end chain of support—not just at the hypervisor and XAPI layer, but crucially within the VM itself. The virtualised EFI BIOS inside the guest must actively support and enable both "Above 4G Decoding" and ReBAR support to handle the extended memory mapping.

Without that full stack alignment, here is what actually happens:

The 256 MB Bottleneck: Without native virtualised ReBAR support at the hypervisor layer and matching EFI BIOS flags inside the VM, the guest OS is forced to interact with a massive VRAM pool (e.g., 16 GB on an Arc card) through a rigid, un-resizable 256 MB MMIO window.
API/Tool-Stack Visibility Failures: The low-level kernel driver might successfully recognize the hardware ID and sit there quietly while idle. However, the moment user-space compute stacks or management tools attempt to initialize a real workload, they aggressively choke on this memory limitation, triggering an immediate driver reset or crash.
Functional Invisibility: To the high-level application, AI framework, or orchestration tool, the GPU effectively becomes invisible or non-functional because the compute API communication layer breaks down entirely.

Since you mentioned you are running this setup successfully on a headless Ubuntu guest, did you run actual heavy compute/inference workloads and look at the performance data or dmesg output?

If you've managed to bypass the 256 MB bar limitation without virtualised EFI/ReBAR support at the hypervisor layer, sharing your dmesg | grep -i BAR or baseline benchmark data would be incredibly helpful to see how your guest is actually mapping that VRAM. Otherwise, the driver simply loading at idle can be highly misleading.