I'm also investigating on a solution to make XCP-ng opportunistically resize device BAR; in case the firmware didn't do it (or has no support for it).

Did some investigations.

TL;DR: You have Resized BARs for the Intel Arc (and any device) in VMs, iff you have Resizable BAR feature enabled in the firmware.

64-bits BAR works inside guests. BARs too large to fit in the legacy area ("MMIO hole") are going to be placed over TOM (above guest RAM).
However, the logic to resize the BAR dynamically isn't available in the guest. If the device BAR starts at 256 MB, the guest won't be able to resize it.

When Resizable BAR feature is enabled in the firmware, device BARs that can be resized are resized to their maximum early at boot, hence the device will have large BARs in XCP-ng.
Such BAR are passed through entirely, iow, you have resized BARs in the guest, this shows up in @jamesg guest lspci https://xcp-ng.org/forum/topic/12316/pcie-pass-through-lanes-and-lane-performance/12?_=1793630786768

In cases firmware has no support for "Resizable BAR" but has support for 64-bits BARs, Resizable BAR can be triggered manually from Dom0 through /sys/bus/pci/devices/.../resourceN_resize (yet, it's still going to be tricky to use in practice), but that's not a kernel feature available as of XCP-ng 8.3.

Consider this feature as experimental

Many Intel CPUs have support for Intel APICv (APIC Virtualization) features that aims at reducing the number of APIC-related VM-Exits in guests, hence improving performance.
This has been introduced circa Intel Ivy Bridge-EP, and is also available on somewhat recent client CPUs as well. But keep in mind that not all SKUs have support for it.

Xen has support for it since roughly Xen 4.3, yet XenServer (and XCP-ng) originally disabled by default due to issues at this time around live migration between hardware that supports it and ones that don't.

Internal testing on the feature shows performance improvements in various benchmarks (5-10% in I/O benchmarks up to 30% on nginx with a Ice Lake Xeon). We didn't test on other CPUs, but older CPUs (in particular those affected by L1TF/Foreshadow) may observe larger improvements.

How to enable

This feature exists in the Xen version shipped in XCP-ng, but disabled by default, you can enable support on the host with

/opt/xensource/libexec/xen-cmdline --set-xen apicv=1
# then reboot

Once enabled, you can check hardware support with

xl dmesg | grep -A 25 "VMX: Supported advanced features"

And look for APIC Register Virtualization and Virtual Interrupt Delivery.

The feature can be disabled with

/opt/xensource/libexec/xen-cmdline --delete-xen apicv
# then reboot

What to test

Along with general feedback on the feature (does it work, supported hardware, ...), we're interested on :

• interactions with live migration, especially with hosts that have different configuration of APICv
• performance on various workloads (web server, databases, I/O, ...)

What about AMD ?

AMD counterpart is AVIC (Advanced Virtualized Interrupt Controller) introduced (in practice) with Zen 2 (but also existed on some Carrizo CPUs) then improved with x2AVIC with Zen 4.
Xen support for (x2)AVIC is currently work in progress.
Refer to Denis Mukhin's presentation on AVIC for Xen (Xen Summit 2026) : https://youtu.be/RhaJsOU_PIA

I finally got access to a machine with Intel Arc machine.

There are issues around hvm-pirq, hence hvm-pirq=0 is required (see https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7).

For some reason, Fedora Workstation has a issue with this GPU (a Fedora bug I guess), but CachyOS (desktop iso) works fine.

Ran a clpeak bench, and performance looks normal for this class of GPU.
Host-device transfer is a bit slower than "theorical", but definetely not PCIe 1x1 and more PCIe 3x8-ish (matching reality as it's plugged as PCIe 3.0).

in-VM inxi -CGM

Machine:
  Type: Xen System: Xen product: HVM domU v: 4.17 serial: 2c66cb09-57db-340c-ad8e-f505eb25ca3e
  Mobo: N/A model: N/A serial: N/A Firmware: UEFI vendor: Xen v: 4.17 date: 10/06/2026
CPU:
  Info: 6x 1-core model: Intel Xeon Silver 4112 bits: 64 type: SMP cache: L2: 6x 1024 KiB (6 MiB)
  Speed (MHz): avg: 2594 min/max: N/A cores: 1: 2594 2: 2594 3: 2594 4: 2594 5: 2594 6: 2594
Graphics:
  Device-1: driver: bochs-drm v: N/A
  Device-2: Intel Battlemage G21 [Arc Pro B50] driver: xe v: kernel
  Display: unspecified server: Xwayland v: 24.1.13 driver: gpu: bochs-drm tty: 212x48
  API: EGL v: 1.5 drivers: iris,swrast platforms: gbm,surfaceless,device
  API: OpenGL v: 4.6 vendor: mesa v: 26.1.6-arch3.1 note: console (EGL sourced) renderer: Mesa
    Intel Arc Pro B50 Graphics (BMG G21), llvmpipe (LLVM 22.1.8 256 bits)
  API: Vulkan v: 1.4.357 drivers: intel surfaces: N/A
  Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo de: kscreen-console,kscreen-doctor
    gpu: gputop, intel_gpu_top, lsgpu, nvidia-smi wl: wayland-info x11: xdpyinfo, xprop, xrandr

Raw numbers

Backend: Vulkan
  Device 0: Intel(R) Arc(tm) Pro B50 Graphics (BMG G21)
    Driver version   : 26.2.4
    API version      : 1.4.354
    VRAM             : 16048 MB

    Single-precision compute
      float    : 9.47 TFLOPS
      float2   : 9.48 TFLOPS
      float4   : 9.52 TFLOPS

    Half-precision compute fp16xfp16+fp16
      half     : 18.3 TFLOPS
      half2    : 18.8 TFLOPS
      half4    : 19.1 TFLOPS

    Double-precision compute
      double   : 650 GFLOPS
      double2  : 647 GFLOPS
      double4  : 640 GFLOPS

    Mixed-precision compute fp16xfp16+fp32
      mp       : 9.31 TFLOPS
      mp2      : 9.32 TFLOPS
      mp4      : 9.42 TFLOPS

    BF16 compute bf16xbf16+fp32
      bf16     : 9.17 TFLOPS
      bf16_2   : 9.30 TFLOPS
      bf16_4   : 9.28 TFLOPS

    Integer compute int32
      int      : 1.81 TOPS
      int2     : 1.88 TOPS
      int4     : 1.82 TOPS

    INT8 dot-product compute
      int8_dp  : 28.0 TOPS
      int8_dp2 : 26.3 TOPS
      int8_dp4 : 27.1 TOPS

    Cooperative matrix
      fp16     : 67.3 TFLOPS
      fp16 f16acc : 68.9 TFLOPS
      bf16        : 68.9 TFLOPS
      int8        : 141 TOPS

    Global memory bandwidth
      float    : 196 GB/s
      float2   : 206 GB/s
      float4   : 212 GB/s

    Local memory bandwidth
      float    : 2.55 TB/s
      float2   : 3.65 TB/s
      float4   : 4.23 TB/s

    Image memory bandwidth
      float4   : 213 GB/s

    Transfer bandwidth
      h2d      : 4.81 GB/s
      d2h      : 5.48 GB/s

    Kernel launch latency
      dispatch : 48.7 us
      roundtrip : 113 us

Real performance is much higher than the "displayed" Speed 2.5GT/s, Width x1.

@bvitnik
Various reasons :

• In some niche cases, it can cause compatibility issues with guests.
• Viridian exposes additionnal attack surface e.g VSA-2025-006 and VSA-2026-031
  and given that this is generally unused for non-Windows guests, there is no good reason to expose it to the guest

For Windows guests, use of the dedicated Windows template is strongly encouraged, as it brings additional viridian flags and other tunings for better compatibility and performance.

@leroyj
We already know that our k10temp module doesn't have support beyond Zen 2, and needs to be updated. That's not incredibly complex, but still needs to be done.

I was wondering if the amd cpu temp be probed the way you did for the intel cpu.

No, AMD temperature infos are not exposed through MSR but through PCIe/MMIO (through various subsystems, like "SMU" and other ones). Like what does https://github.com/torvalds/linux/blob/master/drivers/hwmon/k10temp.c or https://github.com/ocerman/zenpower.
(regarding the AI draft, there is no documented MSR 0xc0010292 in neither APM nor public Zen3 PPM)

That doesn't require any specific Xen support aside the right Linux drivers in Dom0.

While I think we need to do something about containers in VMs, I'm not convinced this is a good idea adding this to the regular guest agent for various reasons.

Docker is one way of running containers, but there are numerous other ones like podman or other runc/containerd based ones. So we can't really make a one size fits all solution. And that doesn't fully answer the maintenance and long-term aspect.

Well, actually what you are looking for is some form of control on guest docker runtime, but without using network, that's actually a solved problem, but plumbing is missing. And we don't need a custom docker-aware agent for this, we mostly need to use one of vsock/pv-channel/pvcalls to build a bridge between e.g guest docker socket and Xen Orchestra (which would talk to guest docker and control/get info from it).

@dkidd255 @jamesg
I didn't forgot about it, but I still don't have access to relevant hardware (for reasons outside of my control).

In the meantime, if that happens to be related, can you try the patch that allows disabling hvm-pirq (this is going to be globally available soon) ?

I'm not sure disabling Nagle is a good idea (even though it can improve things here). Fundamentally, we're doing bulk transfer of disk content, which Nagles tries to optimize by coalescing packets, so you're not flooding the network with small TCP packets.

The main problem here is that the progress is gated by NBD replies, which is going to be bad regardless of TCP configuration. TCP_NODELAY will workaround this problem, but with significant tradeoffs (and perhaps will perform worse in some other cases).
What should be done instead is that writes should be streamed (or pipelined) while reading replies in parralel, so that NBD reply delays doesn't bottleneck the whole transfer. But that actually requires a redesign of the whole NBD implementation which is not going to be a easy thing AFAICT.

I can't speak for the whole team, but will still give my view on it, at least on what I'm aware of.

@hydromike a dit:

...

VROC is the easier half

Graid licensed the Intel VROC IP outright — the deal closed last November, and they relaunched it as "VROC by Graid Technology" at Computex on 1 June with a 24-month roadmap and Xeon 6 support. Channel rollout started this quarter.

VROC on Linux is Intel VMD plus the kernel's own md driver with IMSM metadata. Their 2.0.1 release notes confirm it — graidctl ls vd -t VROC returns /dev/md123. In-tree code. No proprietary module, no kernel taint, no NVIDIA driver in dom0.

Roughly what that path needs:

• CONFIG_VMD in the dom0 kernel
• The VMD-under-Xen work @yann already did once for the demonstrator (Xen support for VMD also landed upstream in 6.12.30)
• mdadm with IMSM in dom0
• Licensing that works without a hardware key — their roadmap replaces the physical key with UEFI-based licensing, which would help

Worth reminding that installing custom software in Dom0, or replacing Dom0 packages is discouraged.

The part worth Vates' attention regardless of my use case: VROC RAID1 would give XCP-ng a supported dom0 boot mirror. That's been asked about here for years without an official answer. This is a plausible route to one.

SupremeRAID is harder

I don't want to undersell this. dom0 is a PV domain, not bare metal — memory-capped, limited vCPUs, doesn't own the IOMMU, and its physical addresses aren't machine addresses. If the SupremeRAID data path depends on peer-to-peer PCIe DMA between the GPU and the NVMe drives, that either works or it doesn't, and no amount of packaging effort fixes it. Their install guide also recommends disabling VT-d, which we obviously can't do. Those questions have gone to Graid and I'll post what comes back.

P2P DMA is more of a platform capability (e.g ACS stuff) than a hypervisor issue. At least, as long as there is no DMA to magic regions like MSI-X/PBA as that will potentially fail.

In the past, I had success doing P2P DMA between a PCI card and NVMe drives, so it's definetely doable.

...

Questions for @yann and @Team-Hypervisor-Kernel

These decide whether Graid's engineers should spend any time at all, so I'd rather ask now than after:

1. Is dom0 in 9.0 still PV, or has it moved to PVH? This changes the DMA and IOMMU picture substantially and it's the first thing their team will ask.

Dom0 will still be running in PV mode by default at least. There are no immediate plan to switch to PVH Dom0.

2. What's the target kernel for 9.0, and roughly when does the ABI freeze? I don't want to point them at 6.12 if the foundations rebuild lands elsewhere.
3. What's Vates' position on a proprietary, kernel-tainting out-of-tree module in dom0 — acceptable in principle, or a hard no? DRBD and now the backported Ceph client are both already in dom0, but both are GPL, so I read those as precedent for the mechanism rather than the licensing.
4. Is a supplemental pack a viable delivery vehicle, or would this need to go through Koji as a normal -module RPM?

Features like SecureBoot will add complexity with external out-of-tree modules, and while there can be solutions to that, but I don't expect any of that to land anytime soon.

5. Is CONFIG_VMD enabled in the 9.0 kernel builds, and did the VMD patch make it in?

If question 3 is a no, that's a useful answer — it closes the SupremeRAID track cleanly and we put the effort into VROC instead.

...

AIA, I don't expect anything practical with the current state of things unless there is a quite significant effort.
However, if these stack(s) can be moved in a dedicated VM with PCI Passthrough, that can make things more workable with the likes of storage domains.

Intel needs some special handling to support physical displays with PCI Passthrough; I don't know much of the details, but on "recent" machines, some bits are missing according to :
https://lore.kernel.org/all/20260802050824.10554-1-brchuckz@aol.com/

@JamesG Don't bother, I'm in PTO next week; so I may get access to hardware when I'm back.

FYI I will get my hands on a Intel Arc Pro B50 to make various tests with it (in the next weeks). If something don't behave as expected, I will try to find a solution.

@JamesG

You can't PCI Passthrough a PCI bridge (and even if you could, it wouldn't be very useful), as it's a part of PCI internals and not really something the guest requires to use the device.

As I asked previously, is there any issue (aside display like PCIe 1.0 1x) with the device, is it behavior correctly or having the expected performance ?

@ravenet said:
@teddyastie separate from the above, a standing question whenever you have a minute: is it expected that passthrough GPU MSIs are never delivered at all on 8.3? Both my working rig and PessimistTech's broken one show the amdgpu xen-pirq MSI vector at zero on all CPUs, forever — everything survives on driver-side polling. It works, but it means any driver path that hard-depends on an interrupt has no safety net, and I suspect it's why the amdgpu ring resets in his log keep failing. Curious whether that's a known limitation of the pirq MSI path for passthrough devices or something worth a ticket.

TL;DR This is likely a known issue, and we're currently discussing on a potential fix.

There is a known bug regarding MSI on AMDGPU when using "hvm_pirq" ("use event channels for device interrupts"), see
https://lore.kernel.org/xen-devel/20240110095311.61809-1-roger.pau@citrix.com/
https://github.com/QubesOS/qubes-issues/issues/7971

The easy fix is to disable "HVM PIRQ" feature (which has been made disabled by default in Xen 4.19), but on the XCP-ng side, this is configured by XAPI Project which needs to be updated as well to apply this change (which appears to be a bit tricky for various reasons).

@PessimistTech
That looks a bit odd indeed.

In addition to what proposed @yannsionneau, can you also give the output of :

• xl dmesg (in Dom0)
• dmesg (in Dom0)
• dmesg (in the guest)

So that we can try to pin-point what may be happening.

@JamesG said:

So I ordered some mini DP dummy plugs and took a look at this further today.

First thought...Just load Windows on the bare metal and see what the card does.

No problem. Loaded Win10, loaded the latest drivers, PCIe Gen4x8 lanes. Sweet. At least that works.

Put the XCP-ng drive back in, booted up, passed-through the GPU to a Win11 guest, updated the drivers, PCIe Gen1x1.

As I explained briefly in https://xcp-ng.org/forum/post/106479, this is mostly display as the real hardware may be running at a different speed.

Ultimately I need to install a Linux distro with a really current kernel on the bare metal and see how it sees the GPU. That might be a good project for tomorrow.

I think this just comes down to a combination of too new, not well supported hardware with some quirks and somewhat dated code and support in XCP-ng (ReBAR support in guests for example).

I did quick checks and I believe that ReBAR works; at least with UEFI guests (our OVMF build does support resizable bar), and it actually works as suggest https://xcp-ng.org/forum/post/106489 (VRAM BAR is 16 GB); but I didn't do extensive testing in that area.

@JamesG Can you try having a workload on the GPU (and also checking if the performance is lower, or if it's just display) ?

@JamesG said:

When I ask the system what it thinks the capabilities are, again I get Gen1x1:

cat /sys/bus/pci/devices/0000:00:08.0/current_link_speed
2.5 GT/s PCIe
cat /sys/bus/pci/devices/0000:00:08.0/current_link_width
1
cat /sys/bus/pci/devices/0000:00:08.0/max_link_speed
2.5 GT/s PCIe
cat /sys/bus/pci/devices/0000:00:08.0/max_link_width
1

This information is actually crafted by QEMU, it doesn't represent the actual real device PCI state; you should check this from Dom0 which should be closer to reality.

@JamesG

I know I'm a bit out in uncharted territory here. I don't think XCP-ng supports ReBAR via pass-through (I think that's on the road-map for XenServer9 and maybe XCP-ng 9 as well), current gen GPUs all want ReBAR support, Intel GPU's are in a mostly experimental/developmental mode and might be having a murky future anyway...

XCP-ng guest firmware have a very "legacy" initial BAR placement (i.e thus no explicit support for Resizable BAR, everything below 4G, ...), but the guest operating system can still replace the location of BAR itself if the hardware allows it (in particular Linux will try it, but I'm not sure Windows does).

In your case, the GPU VRAM BAR is 16 GB, so it's obviously resized. And according to xe, you should see in dmesg something like BAR2 resized to 16384MiB