PCIe Pass-through lanes and lane performance
-
99 percent of the dev team is working security and basic hypervisor
The numbers don't match any reality though. Yes, maintenance is important, but we have a developer working on AMD iGPU passthrough fixes, and Teddy here is following the Intel Arc topic closely and obtained hardware (waiting for it to be available in the lab). We also work on AMD MxGPU for vGPU support on the supported devices. We also have work on virtio-gpu. Without producing something that can become an actual feature yet, as it's complex, but there's clearly investment on those fronts.
-
But I would in any case where PCI passthrough for edge cases like the GPU marked kind of is, is not a requirement, Always recommend XCP-ng over the others.
Thanks for that :).
There's no problem to state faithfully what works and what doesn't with a solution, and adapt the advice to the actual needs.
I just felt compelled to react because I didn't find it entirely right to state that GPU passthrough has low priority. It's supported in general, but there are devices that come with technical challenges not fully addressed yet in Xen or in XCP-ng, and we treat that as part of the general hardware support we aim for.
Same as we work on device drivers for devices mostly found in home labs, without dismissing them as low priority.
And it's a good point to state that the needs around GPUs are growing fast due to AI, indeed.
-
I get out of my silence to confirm that I'm indeed working on AMD MxGPU driver and AMD iGPU passthrough issues.
I confess that I have not communicated since a while on this work, but I'm making progress and I'm very close to publish a call to testing PoC packages that should improve the iGPU passthrough situation. -
@stormi My offer to give you guys VPN access to an Epyc server with Intel ARC B50 still stands.
IPSec, Wireguard, whatever...You can have IPMI access and at least some basic hardware to get you up and going.
JamesG
-
@JamesG Thanks. I'll let @teddyastie see. I think he would be more confortable with having the card in our lab, which should be imminent now, but I won't speak for him.
-
@stormi I am glad to hear there is progress, and I never doubted that there was interest and will from the Dev team, it was just stated, by Oliver himself, that the focus was mostly on the classic hypervisor features and that resources for edge cases were scarce. The numbers might not be spot on, but it was something similar.. Anyways I know you care and that the team always take interest in home-lab scenarios. And that, is why I keep recommending XCP-ng. You are good and motivated people, you care and interact with the community and you have a product that is simple in structure and free of most of the technical bureaucracy that the other ecosystems have.
And yes, this particular Intel Arc issue, is not only on XCP-ng, The intel toolbox and drivers for this platform is currently a mess for Linux.
And you are right to take home-labbers under your wing and prioritize it. As I'm sure you know, although it might not be of direct importance for revenue, they do have a huge impact on popularity. I dont think Proxmox would have had the traction it has, without them
Thank you for replying, and thank you to the team for their hard work!
-
I finally got access to a machine with Intel Arc machine.
There are issues around hvm-pirq, hence hvm-pirq=0 is required (see https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7).
For some reason, Fedora Workstation has a issue with this GPU (a Fedora bug I guess), but CachyOS (desktop iso) works fine.
Ran a clpeak bench, and performance looks normal for this class of GPU.
Host-device transfer is a bit slower than "theorical", but definetely not PCIe 1x1 and more PCIe 3x8-ish (matching reality as it's plugged as PCIe 3.0).in-VM
inxi -CGMMachine: Type: Xen System: Xen product: HVM domU v: 4.17 serial: 2c66cb09-57db-340c-ad8e-f505eb25ca3e Mobo: N/A model: N/A serial: N/A Firmware: UEFI vendor: Xen v: 4.17 date: 10/06/2026 CPU: Info: 6x 1-core model: Intel Xeon Silver 4112 bits: 64 type: SMP cache: L2: 6x 1024 KiB (6 MiB) Speed (MHz): avg: 2594 min/max: N/A cores: 1: 2594 2: 2594 3: 2594 4: 2594 5: 2594 6: 2594 Graphics: Device-1: driver: bochs-drm v: N/A Device-2: Intel Battlemage G21 [Arc Pro B50] driver: xe v: kernel Display: unspecified server: Xwayland v: 24.1.13 driver: gpu: bochs-drm tty: 212x48 API: EGL v: 1.5 drivers: iris,swrast platforms: gbm,surfaceless,device API: OpenGL v: 4.6 vendor: mesa v: 26.1.6-arch3.1 note: console (EGL sourced) renderer: Mesa Intel Arc Pro B50 Graphics (BMG G21), llvmpipe (LLVM 22.1.8 256 bits) API: Vulkan v: 1.4.357 drivers: intel surfaces: N/A Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo de: kscreen-console,kscreen-doctor gpu: gputop, intel_gpu_top, lsgpu, nvidia-smi wl: wayland-info x11: xdpyinfo, xprop, xrandrRaw numbers
Backend: Vulkan Device 0: Intel(R) Arc(tm) Pro B50 Graphics (BMG G21) Driver version : 26.2.4 API version : 1.4.354 VRAM : 16048 MB Single-precision compute float : 9.47 TFLOPS float2 : 9.48 TFLOPS float4 : 9.52 TFLOPS Half-precision compute fp16xfp16+fp16 half : 18.3 TFLOPS half2 : 18.8 TFLOPS half4 : 19.1 TFLOPS Double-precision compute double : 650 GFLOPS double2 : 647 GFLOPS double4 : 640 GFLOPS Mixed-precision compute fp16xfp16+fp32 mp : 9.31 TFLOPS mp2 : 9.32 TFLOPS mp4 : 9.42 TFLOPS BF16 compute bf16xbf16+fp32 bf16 : 9.17 TFLOPS bf16_2 : 9.30 TFLOPS bf16_4 : 9.28 TFLOPS Integer compute int32 int : 1.81 TOPS int2 : 1.88 TOPS int4 : 1.82 TOPS INT8 dot-product compute int8_dp : 28.0 TOPS int8_dp2 : 26.3 TOPS int8_dp4 : 27.1 TOPS Cooperative matrix fp16 : 67.3 TFLOPS fp16 f16acc : 68.9 TFLOPS bf16 : 68.9 TFLOPS int8 : 141 TOPS Global memory bandwidth float : 196 GB/s float2 : 206 GB/s float4 : 212 GB/s Local memory bandwidth float : 2.55 TB/s float2 : 3.65 TB/s float4 : 4.23 TB/s Image memory bandwidth float4 : 213 GB/s Transfer bandwidth h2d : 4.81 GB/s d2h : 5.48 GB/s Kernel launch latency dispatch : 48.7 us roundtrip : 113 usReal performance is much higher than the "displayed"
Speed 2.5GT/s, Width x1. -
@TeddyAstie
Great that you got around to test it.I did test with the hvm-pirq=0 both pre and post official patch. As you suggested earlier in the thread,
It made no difference for me. I still had the same issues, and the 1x1 link detection.. I tested on Ubuntu, Fedora, Debian, Alma and Rocky. (all headless server)
The only other OS was windows, which was the only OS I could get it to do anything, although heavily borked and slower than expected. Intel driver panel was also complaining about PCI compatibility and reBarUbuntu is (as far as I know) the main distro that intel test and sync with.
You should test and see if you can get Level Zero and oneApi going. (Needed for any AI work)
I could never get these to start, driver crashed every time. If you can get that to work on Ubuntu, I would suspect it might be a grouping issue on certain motherboards.Also, What about the missing audio device not showing in the list?
-
@TeddyAstie
Great that you got around to test it.I did test with the hvm-pirq=0 both pre and post official patch. As you suggested earlier in the thread,
It made no difference for me. I still had the same issues, and the 1x1 link detection.. I tested on Ubuntu, Fedora, Debian, Alma and Rocky. (all headless server)
The only other OS was windows, which was the only OS I could get it to do anything, although heavily borked and slower than expected. Intel driver panel was also complaining about PCI compatibility and reBarUbuntu is (as far as I know) the main distro that intel test and sync with.
You should test and see if you can get Level Zero and oneApi going. (Needed for any AI work)
I could never get these to start, driver crashed every time. If you can get that to work on Ubuntu, I would suspect it might be a grouping issue on certain motherboards.Also, What about the missing audio device not showing in the list?
A part of the reason for it being slower without ReBAR (or Resizable BAR) is due to that technology lifting, the 256 MB limit (for per chunk) on CPU accessing GPU VRAM.
To improve performance when passing through modern GPUs (and graphics cards). With lots of VRAM the limit on accessing GPU VRAM by CPU will really crease the performance of the VM at the minimum. So support for ReBAR present in Xen and XCP-ng would help performance, when passing through the GPU. Basically a virtual form of ReBAR support for VMs, when passing through GPUs.
-
@teddyastie I’ve added feedback in regards to the above on Vates Feedback portal, see link below.
-
@pandusen Your Windows test results are the perfect real-world example of this exact issue.
When you noted that the Intel driver panel was explicitly complaining about "PCI compatibility and reBar" and that performance was heavily borked, you hit the exact technical bottleneck. Without ReBAR support at the hypervisor layer, the guest OS is forced to interact with that massive 16 GB VRAM space through a rigid 256 MB window, choking it out with massive memory paging overhead.
Even @teddyastie's Linux testing reflects this—noting that host-device transfer bandwidth is "a bit slower than theoretical."
@olivierlambert From a strategic standpoint, as home labs and enterprise environments aggressively pivot toward localized AI, LLM inferencing, and heavy GPU compute, raw PCIe passthrough is shifting from an edge case to a core infrastructure requirement. Implementing a virtualized form of ReBAR support within Xen/XCP-ng would natively lift this 256 MB restriction. It ensures modern GPU silicon can run at 100% efficiency, keeping XCP-ng highly competitive against monolithic hypervisors that handle large BAR mapping.
Especially with VMware VCF 9.0 and up-to-date Hyper-V on Windows Server 2025 advancing their platforms, Vates needs to have virtualised ReBAR support to maintain its edge against this competition.
-
@john.c I would add that in addition to AI and VDI, media functions would be greatly assisted with GPU accelerated encode/decode. I need to do more with less. Having a GPU I could carve up and assign to VM workloads would be amazing.
-
@john.c I would add that in addition to AI and VDI, media functions would be greatly assisted with GPU accelerated encode/decode. I need to do more with less. Having a GPU I could carve up and assign to VM workloads would be amazing.
@JamesG Absolutely spot on. Media encoding/decoding and VDI density are huge components of this. Being able to slice up a GPU efficiently means VMs need to interact with that VRAM cleanly without hitting performance walls.
Whether it's splitting the card via vGPU profiles or passing it through fully for raw compute, lifting these platform memory limitations is what unlocks that "do more with less" capability for media stacks. It all ties back to how robustly the hypervisor fabric can handle modern GPU memory mapping.
If my feedback on Vates Feedback will help your use case please vote for it.
-
@Team-XAPI-Network @bleader Looping in the toolstack side as well. Even if the underlying Xen hypervisor layers align to handle large BAR configurations, XAPI will need the native capabilities to orchestrate and pass these extended memory mapping boundaries down to the VM configuration templates. Without XAPI awareness, we won't be able to easily toggle or manage virtual ReBAR allocations via the CLI or Xen Orchestra.
-
I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.
In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1. I never got to a point of playing with transcoding for media nor AI Inferencing engines, as the tools and drivers never got stable.
-
I get all this.. I was merely trying to help Teddy run into issues with the Intel arc card.
In regards to reBar, It is partly solved if you run a Linux 7 kernel. It is mostly a challenge from the guest os, rather than the hypervisor. As far as i know, xcp-ng reBar is passed through for complete PCI passthrough (not yet for Dom-0 and vfio), but I didn't concern myself too much with that.. The bigger issue was with the device grouping, the missing endpoints like the audio device and the drivers crashing when calling any of the tool stacks from intel. As well as the PCI link only negotiating a 1x1.
Those tool-stacks from Intel crashing are evidence of the lack of virtualised ReBAR support, as they’re getting throttled to the point of choking on the 256MB per block limit.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login