Team - Hypervisor Kernel group | XCP-ng and XO forum

dinhngtu

@acebmxer Hello, could you look into the xen and kernel logs for errors?

TeddyAstie

Consider this feature as experimental

Many Intel CPUs have support for Intel APICv (APIC Virtualization) features that aims at reducing the number of APIC-related VM-Exits in guests, hence improving performance.
This has been introduced circa Intel Ivy Bridge-EP, and is also available on somewhat recent client CPUs as well. But keep in mind that not all SKUs have support for it.

Xen has support for it since roughly Xen 4.3, yet XenServer (and XCP-ng) originally disabled by default due to issues at this time around live migration between hardware that supports it and ones that don't.

Internal testing on the feature shows performance improvements in various benchmarks (5-10% in I/O benchmarks up to 30% on nginx with a Ice Lake Xeon). We didn't test on other CPUs, but older CPUs (in particular those affected by L1TF/Foreshadow) may observe larger improvements.

How to enable

This feature exists in the Xen version shipped in XCP-ng, but disabled by default, you can enable support on the host with

/opt/xensource/libexec/xen-cmdline --set-xen apicv=1
# then reboot

Once enabled, you can check hardware support with

xl dmesg | grep -A 25 "VMX: Supported advanced features"

And look for APIC Register Virtualization and Virtual Interrupt Delivery.

The feature can be disabled with

/opt/xensource/libexec/xen-cmdline --delete-xen apicv
# then reboot

What to test

Along with general feedback on the feature (does it work, supported hardware, ...), we're interested on :

interactions with live migration, especially with hosts that have different configuration of APICv
performance on various workloads (web server, databases, I/O, ...)

What about AMD ?

AMD counterpart is AVIC (Advanced Virtualized Interrupt Controller) introduced (in practice) with Zen 2 (but also existed on some Carrizo CPUs) then improved with x2AVIC with Zen 4.
Xen support for (x2)AVIC is currently work in progress.
Refer to Denis Mukhin's presentation on AVIC for Xen (Xen Summit 2026) : https://youtu.be/RhaJsOU_PIA

MrQ

@jr-m4 thank you so much for testing this updated driver.

Just to make sure, I understand now that you confirm the driver is working as expected and the issues you had were not related to it? Or are there any problems still with this new driver?

Many thanks again for spending the time to test and report.

dinhngtu

@cg @trelane I've updated the release notes and added a link to the 9.2.351 patch.

For the IP loss bug when toggling the "Manage Citrix PV..." setting, this bug should not happen on XCP-ng tools. In theory, it should not happen on the XenServer 9.6.0 drivers either, but I haven't tested.

If you need to reconfigure your VM's IP, check out the commands xe vif-configure-ipv4/ipv6. Currently it supports configuring IP/prefix/gateway; DNS configuration is coming soon.

TeddyAstie

I finally got access to a machine with Intel Arc machine.

There are issues around hvm-pirq, hence hvm-pirq=0 is required (see https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7).

For some reason, Fedora Workstation has a issue with this GPU (a Fedora bug I guess), but CachyOS (desktop iso) works fine.

Ran a clpeak bench, and performance looks normal for this class of GPU.
Host-device transfer is a bit slower than "theorical", but definetely not PCIe 1x1 and more PCIe 3x8-ish (matching reality as it's plugged as PCIe 3.0).

in-VM inxi -CGM

Machine:
  Type: Xen System: Xen product: HVM domU v: 4.17 serial: 2c66cb09-57db-340c-ad8e-f505eb25ca3e
  Mobo: N/A model: N/A serial: N/A Firmware: UEFI vendor: Xen v: 4.17 date: 10/06/2026
CPU:
  Info: 6x 1-core model: Intel Xeon Silver 4112 bits: 64 type: SMP cache: L2: 6x 1024 KiB (6 MiB)
  Speed (MHz): avg: 2594 min/max: N/A cores: 1: 2594 2: 2594 3: 2594 4: 2594 5: 2594 6: 2594
Graphics:
  Device-1: driver: bochs-drm v: N/A
  Device-2: Intel Battlemage G21 [Arc Pro B50] driver: xe v: kernel
  Display: unspecified server: Xwayland v: 24.1.13 driver: gpu: bochs-drm tty: 212x48
  API: EGL v: 1.5 drivers: iris,swrast platforms: gbm,surfaceless,device
  API: OpenGL v: 4.6 vendor: mesa v: 26.1.6-arch3.1 note: console (EGL sourced) renderer: Mesa
    Intel Arc Pro B50 Graphics (BMG G21), llvmpipe (LLVM 22.1.8 256 bits)
  API: Vulkan v: 1.4.357 drivers: intel surfaces: N/A
  Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo de: kscreen-console,kscreen-doctor
    gpu: gputop, intel_gpu_top, lsgpu, nvidia-smi wl: wayland-info x11: xdpyinfo, xprop, xrandr

Raw numbers

Backend: Vulkan
  Device 0: Intel(R) Arc(tm) Pro B50 Graphics (BMG G21)
    Driver version   : 26.2.4
    API version      : 1.4.354
    VRAM             : 16048 MB

    Single-precision compute
      float    : 9.47 TFLOPS
      float2   : 9.48 TFLOPS
      float4   : 9.52 TFLOPS

    Half-precision compute fp16xfp16+fp16
      half     : 18.3 TFLOPS
      half2    : 18.8 TFLOPS
      half4    : 19.1 TFLOPS

    Double-precision compute
      double   : 650 GFLOPS
      double2  : 647 GFLOPS
      double4  : 640 GFLOPS

    Mixed-precision compute fp16xfp16+fp32
      mp       : 9.31 TFLOPS
      mp2      : 9.32 TFLOPS
      mp4      : 9.42 TFLOPS

    BF16 compute bf16xbf16+fp32
      bf16     : 9.17 TFLOPS
      bf16_2   : 9.30 TFLOPS
      bf16_4   : 9.28 TFLOPS

    Integer compute int32
      int      : 1.81 TOPS
      int2     : 1.88 TOPS
      int4     : 1.82 TOPS

    INT8 dot-product compute
      int8_dp  : 28.0 TOPS
      int8_dp2 : 26.3 TOPS
      int8_dp4 : 27.1 TOPS

    Cooperative matrix
      fp16     : 67.3 TFLOPS
      fp16 f16acc : 68.9 TFLOPS
      bf16        : 68.9 TFLOPS
      int8        : 141 TOPS

    Global memory bandwidth
      float    : 196 GB/s
      float2   : 206 GB/s
      float4   : 212 GB/s

    Local memory bandwidth
      float    : 2.55 TB/s
      float2   : 3.65 TB/s
      float4   : 4.23 TB/s

    Image memory bandwidth
      float4   : 213 GB/s

    Transfer bandwidth
      h2d      : 4.81 GB/s
      d2h      : 5.48 GB/s

    Kernel launch latency
      dispatch : 48.7 us
      roundtrip : 113 us

Real performance is much higher than the "displayed" Speed 2.5GT/s, Width x1.

dinhngtu

@john.c Indeed I skipped over the part that the host is still on 7.6, that's a rather big upgrade blocker I think.