PCIe Pass-through lanes and lane performance

JamesG

@TeddyAstie I think there's alot of interest in the Intel platform because of the lack of ridiculous licensing for virtualization features. The problem is that you need a pretty current 6.x kernel to use them and XCP-ng/XenServer is just stuck on a really aging kernel right now.

As exposed in this thread, the PCIe pass-through isn't really working properly under XCP-ng.

Even if pass-through worked, that's kinda lame because the real purpose would be to chop up the GPU into SR-IOV instances so multiple workloads could have a GPU.

The Proxmox community is already rockin' and rollin' with this. I watched a video a couple of days ago where a B70 was chopped up into Qty.4, 8GB GPU's via SR-IOV and passed to clients.

No ridiculous mega-$$$$ licensing. It just worked.

JamesG

@TeddyAstie

Thought...If you want access to a server with a B50, I think I could setup VPN access and put a host online with IPMI access for you to do whatever you want with.

You will have an AMD Epyc 7302 with 64GB of ram some flash and HDD storage, and an Intel B50. PM me and we can work out the details of what you need to experiment with, and I can setup some remote access for you to get into the system.

I will gladly work with you guys any which way I can to move this forward.

TeddyAstie

@JamesG Don't bother, I'm in PTO next week; so I may get access to hardware when I'm back.

pandusen

@andriy.sultanov

@andriy.sultanov said:

@pandusen As Teddy said above, you can't passthrough a PCI bridge, so there's no PCI devices xapi shouldn't omit here.

I am not trying to pass through the bridge only the end points. The Intel arc's have 2 end points: The GPU and the Sound device. "xe pci-list" only reveals the GPU, not the sound device. (this works for nvidia and AMD)

But "going the xen-cmdline way" shouldn't break anything, that's what xe pci-disable-dom0-access does behind the scenes. What issues did you see? Which steps did you follow?

the sound device is available in the lspci list and can be passed through using CLI.
But doing so, (using CLI for passtrough) undoes everything done using xe or the passthrough gui in XO. and results in this:
https://xcp-ng.org/forum/topic/10609/xcp-ng-8.3-pci-passthrough-issue
so yes, its does break something.

dkidd255

@jamesg @teddyastie, any further findings on this?

I am building an AI Inference server with a handful of Arc Pro B70's and a Supermicro H12SSL-i MOBO + 128GB of ram. I was hoping to use xcpng as a hypervisor to setup different VMs / software setups, but have also run into roadblocks with pci passthrough.

I'm somewhat of a novice on low level issues like this but if there is any info i can provide to help with the support of these GPUs for passthrough, happy to do so.

Going to give proxmox a try, then bare metal but just wanted to add another somewhat different use case behind the Battlemage cards.

JamesG

@dkidd255 I'm a LOOONG time XCP/XenServer/XCP-ng user in my homelab...And it pains me to say this...

But this just works in Proxmox.

If you want to help us dig in and troubleshoot/explore this topic...Fine.

If you want to actually use your hardware and make progress on your ideas, go to Proxmox.

I seem to recall some recent firmware shenanigans from Intel. Intel maybe thought a little too much of their hardware, or they've decided to just scale things back a bit, but newer versions of firmware have reduced VF's available. I don't think the B70 is impacted by this, but it has massive memory in comparison to the smaller cards.

TeddyAstie

@dkidd255 @jamesg
I didn't forgot about it, but I still don't have access to relevant hardware (for reasons outside of my control).

In the meantime, if that happens to be related, can you try the patch that allows disabling hvm-pirq (this is going to be globally available soon) ?

pandusen

@JamesG Yeah, its scary that GPU passthrough still has a low priority for XCP-ng, during the boom of AI and local LLM Inferencing. I my self am starting to plan a move to Proxmox, though I do not really wish to move, cause I like XCP way better and have been with them for a long time... But that is just comfort...In the end, everything it comes down to is the capability and performance of the running VM guests.

dkidd255

@JamesG, Yea i ended up having to go with a bare metal ubuntu install + docker containers for everything and have been off to the races, I would still like a hypervisor but unfortunately had to move on.

The software stack on these intel's gpus is still maturing but i'm running some pretty impressive models on them and for the money it has been a good investment, hopefully these will gain some popularity and get better support all around.

GPU hosting definitely deserves some attention from xcpng. I would buy some licenses for some production use cases but things like this and nested virtualization (completely unrelated but also something i ran into this past week) needs attention.

stormi

@pandusen said:
its scary that GPU passthrough still has a low priority for XCP-ng, during the boom of AI and local LLM Inferencing.

As XCP-ng product owner, and someone who sees how much work developers put into such topics, may I ask you to elaborate on this "has low priority" statement?

pandusen

@stormi I dont doubt that the team works hard. The GPU passthrough support has been of lower priority for a long time, it has been said many times before by the team and Oliver. 99 percent of the dev team is working security and basic hypervisor, and is not prioritized for edge cases. Which i guess that PCI passthrough is for GPUs Please dont ask me to dig up the sources. It has always been the case with Xen.... Proxmox, KVM and vmware has the edge there, and that should be well known. But I would in any case where PCI passthrough for edge cases like the GPU marked kind of is, is not a requirement, Always recommend XCP-ng over the others.

stormi

@pandusen said:

99 percent of the dev team is working security and basic hypervisor

The numbers don't match any reality though. Yes, maintenance is important, but we have a developer working on AMD iGPU passthrough fixes, and Teddy here is following the Intel Arc topic closely and obtained hardware (waiting for it to be available in the lab). We also work on AMD MxGPU for vGPU support on the supported devices. We also have work on virtio-gpu. Without producing something that can become an actual feature yet, as it's complex, but there's clearly investment on those fronts.

stormi

@pandusen said:

But I would in any case where PCI passthrough for edge cases like the GPU marked kind of is, is not a requirement, Always recommend XCP-ng over the others.

Thanks for that :).

There's no problem to state faithfully what works and what doesn't with a solution, and adapt the advice to the actual needs.

I just felt compelled to react because I didn't find it entirely right to state that GPU passthrough has low priority. It's supported in general, but there are devices that come with technical challenges not fully addressed yet in Xen or in XCP-ng, and we treat that as part of the general hardware support we aim for.

Same as we work on device drivers for devices mostly found in home labs, without dismissing them as low priority.

And it's a good point to state that the needs around GPUs are growing fast due to AI, indeed.

yannsionneau

I get out of my silence to confirm that I'm indeed working on AMD MxGPU driver and AMD iGPU passthrough issues.
I confess that I have not communicated since a while on this work, but I'm making progress and I'm very close to publish a call to testing PoC packages that should improve the iGPU passthrough situation.

JamesG

@stormi My offer to give you guys VPN access to an Epyc server with Intel ARC B50 still stands.

IPSec, Wireguard, whatever...You can have IPMI access and at least some basic hardware to get you up and going.

JamesG

stormi

@JamesG Thanks. I'll let @teddyastie see. I think he would be more confortable with having the card in our lab, which should be imminent now, but I won't speak for him.

pandusen

@stormi I am glad to hear there is progress, and I never doubted that there was interest and will from the Dev team, it was just stated, by Oliver himself, that the focus was mostly on the classic hypervisor features and that resources for edge cases were scarce. The numbers might not be spot on, but it was something similar.. Anyways I know you care and that the team always take interest in home-lab scenarios. And that, is why I keep recommending XCP-ng. You are good and motivated people, you care and interact with the community and you have a product that is simple in structure and free of most of the technical bureaucracy that the other ecosystems have.

And yes, this particular Intel Arc issue, is not only on XCP-ng, The intel toolbox and drivers for this platform is currently a mess for Linux.

And you are right to take home-labbers under your wing and prioritize it. As I'm sure you know, although it might not be of direct importance for revenue, they do have a huge impact on popularity. I dont think Proxmox would have had the traction it has, without them

Thank you for replying, and thank you to the team for their hard work!

TeddyAstie

I finally got access to a machine with Intel Arc machine.

There are issues around hvm-pirq, hence hvm-pirq=0 is required (see https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7).

For some reason, Fedora Workstation has a issue with this GPU (a Fedora bug I guess), but CachyOS (desktop iso) works fine.

Ran a clpeak bench, and performance looks normal for this class of GPU.
Host-device transfer is a bit slower than "theorical", but definetely not PCIe 1x1 and more PCIe 3x8-ish (matching reality as it's plugged as PCIe 3.0).

in-VM inxi -CGM

Machine:
  Type: Xen System: Xen product: HVM domU v: 4.17 serial: 2c66cb09-57db-340c-ad8e-f505eb25ca3e
  Mobo: N/A model: N/A serial: N/A Firmware: UEFI vendor: Xen v: 4.17 date: 10/06/2026
CPU:
  Info: 6x 1-core model: Intel Xeon Silver 4112 bits: 64 type: SMP cache: L2: 6x 1024 KiB (6 MiB)
  Speed (MHz): avg: 2594 min/max: N/A cores: 1: 2594 2: 2594 3: 2594 4: 2594 5: 2594 6: 2594
Graphics:
  Device-1: driver: bochs-drm v: N/A
  Device-2: Intel Battlemage G21 [Arc Pro B50] driver: xe v: kernel
  Display: unspecified server: Xwayland v: 24.1.13 driver: gpu: bochs-drm tty: 212x48
  API: EGL v: 1.5 drivers: iris,swrast platforms: gbm,surfaceless,device
  API: OpenGL v: 4.6 vendor: mesa v: 26.1.6-arch3.1 note: console (EGL sourced) renderer: Mesa
    Intel Arc Pro B50 Graphics (BMG G21), llvmpipe (LLVM 22.1.8 256 bits)
  API: Vulkan v: 1.4.357 drivers: intel surfaces: N/A
  Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo de: kscreen-console,kscreen-doctor
    gpu: gputop, intel_gpu_top, lsgpu, nvidia-smi wl: wayland-info x11: xdpyinfo, xprop, xrandr

Raw numbers

Backend: Vulkan
  Device 0: Intel(R) Arc(tm) Pro B50 Graphics (BMG G21)
    Driver version   : 26.2.4
    API version      : 1.4.354
    VRAM             : 16048 MB

    Single-precision compute
      float    : 9.47 TFLOPS
      float2   : 9.48 TFLOPS
      float4   : 9.52 TFLOPS

    Half-precision compute fp16xfp16+fp16
      half     : 18.3 TFLOPS
      half2    : 18.8 TFLOPS
      half4    : 19.1 TFLOPS

    Double-precision compute
      double   : 650 GFLOPS
      double2  : 647 GFLOPS
      double4  : 640 GFLOPS

    Mixed-precision compute fp16xfp16+fp32
      mp       : 9.31 TFLOPS
      mp2      : 9.32 TFLOPS
      mp4      : 9.42 TFLOPS

    BF16 compute bf16xbf16+fp32
      bf16     : 9.17 TFLOPS
      bf16_2   : 9.30 TFLOPS
      bf16_4   : 9.28 TFLOPS

    Integer compute int32
      int      : 1.81 TOPS
      int2     : 1.88 TOPS
      int4     : 1.82 TOPS

    INT8 dot-product compute
      int8_dp  : 28.0 TOPS
      int8_dp2 : 26.3 TOPS
      int8_dp4 : 27.1 TOPS

    Cooperative matrix
      fp16     : 67.3 TFLOPS
      fp16 f16acc : 68.9 TFLOPS
      bf16        : 68.9 TFLOPS
      int8        : 141 TOPS

    Global memory bandwidth
      float    : 196 GB/s
      float2   : 206 GB/s
      float4   : 212 GB/s

    Local memory bandwidth
      float    : 2.55 TB/s
      float2   : 3.65 TB/s
      float4   : 4.23 TB/s

    Image memory bandwidth
      float4   : 213 GB/s

    Transfer bandwidth
      h2d      : 4.81 GB/s
      d2h      : 5.48 GB/s

    Kernel launch latency
      dispatch : 48.7 us
      roundtrip : 113 us

Real performance is much higher than the "displayed" Speed 2.5GT/s, Width x1.

pandusen

@TeddyAstie
Great that you got around to test it.

I did test with the hvm-pirq=0 both pre and post official patch. As you suggested earlier in the thread,
It made no difference for me. I still had the same issues, and the 1x1 link detection.. I tested on Ubuntu, Fedora, Debian, Alma and Rocky. (all headless server)
The only other OS was windows, which was the only OS I could get it to do anything, although heavily borked and slower than expected. Intel driver panel was also complaining about PCI compatibility and reBar

Ubuntu is (as far as I know) the main distro that intel test and sync with.

You should test and see if you can get Level Zero and oneApi going. (Needed for any AI work)
I could never get these to start, driver crashed every time. If you can get that to work on Ubuntu, I would suspect it might be a grouping issue on certain motherboards.

Also, What about the missing audio device not showing in the list?

john.c

@pandusen said:

@TeddyAstie
Great that you got around to test it.

I did test with the hvm-pirq=0 both pre and post official patch. As you suggested earlier in the thread,
It made no difference for me. I still had the same issues, and the 1x1 link detection.. I tested on Ubuntu, Fedora, Debian, Alma and Rocky. (all headless server)
The only other OS was windows, which was the only OS I could get it to do anything, although heavily borked and slower than expected. Intel driver panel was also complaining about PCI compatibility and reBar

Ubuntu is (as far as I know) the main distro that intel test and sync with.

You should test and see if you can get Level Zero and oneApi going. (Needed for any AI work)
I could never get these to start, driver crashed every time. If you can get that to work on Ubuntu, I would suspect it might be a grouping issue on certain motherboards.

Also, What about the missing audio device not showing in the list?

A part of the reason for it being slower without ReBAR (or Resizable BAR) is due to that technology lifting, the 256 MB limit (for per chunk) on CPU accessing GPU VRAM.

To improve performance when passing through modern GPUs (and graphics cards). With lots of VRAM the limit on accessing GPU VRAM by CPU will really crease the performance of the VM at the minimum. So support for ReBAR present in Xen and XCP-ng would help performance, when passing through the GPU. Basically a virtual form of ReBAR support for VMs, when passing through GPUs.