TeddyAstie

I finally got access to a machine with Intel Arc machine.

There are issues around hvm-pirq, hence hvm-pirq=0 is required (see https://docs.xcp-ng.org/troubleshooting/common-problems/#pci-passthrough-error-unsupported-msi-delivery-mode-7).

For some reason, Fedora Workstation has a issue with this GPU (a Fedora bug I guess), but CachyOS (desktop iso) works fine.

Ran a clpeak bench, and performance looks normal for this class of GPU.
Host-device transfer is a bit slower than "theorical", but definetely not PCIe 1x1 and more PCIe 3x8-ish (matching reality as it's plugged as PCIe 3.0).

in-VM inxi -CGM

Machine:
  Type: Xen System: Xen product: HVM domU v: 4.17 serial: 2c66cb09-57db-340c-ad8e-f505eb25ca3e
  Mobo: N/A model: N/A serial: N/A Firmware: UEFI vendor: Xen v: 4.17 date: 10/06/2026
CPU:
  Info: 6x 1-core model: Intel Xeon Silver 4112 bits: 64 type: SMP cache: L2: 6x 1024 KiB (6 MiB)
  Speed (MHz): avg: 2594 min/max: N/A cores: 1: 2594 2: 2594 3: 2594 4: 2594 5: 2594 6: 2594
Graphics:
  Device-1: driver: bochs-drm v: N/A
  Device-2: Intel Battlemage G21 [Arc Pro B50] driver: xe v: kernel
  Display: unspecified server: Xwayland v: 24.1.13 driver: gpu: bochs-drm tty: 212x48
  API: EGL v: 1.5 drivers: iris,swrast platforms: gbm,surfaceless,device
  API: OpenGL v: 4.6 vendor: mesa v: 26.1.6-arch3.1 note: console (EGL sourced) renderer: Mesa
    Intel Arc Pro B50 Graphics (BMG G21), llvmpipe (LLVM 22.1.8 256 bits)
  API: Vulkan v: 1.4.357 drivers: intel surfaces: N/A
  Info: Tools: api: clinfo, eglinfo, glxinfo, vulkaninfo de: kscreen-console,kscreen-doctor
    gpu: gputop, intel_gpu_top, lsgpu, nvidia-smi wl: wayland-info x11: xdpyinfo, xprop, xrandr

Raw numbers

Backend: Vulkan
  Device 0: Intel(R) Arc(tm) Pro B50 Graphics (BMG G21)
    Driver version   : 26.2.4
    API version      : 1.4.354
    VRAM             : 16048 MB

    Single-precision compute
      float    : 9.47 TFLOPS
      float2   : 9.48 TFLOPS
      float4   : 9.52 TFLOPS

    Half-precision compute fp16xfp16+fp16
      half     : 18.3 TFLOPS
      half2    : 18.8 TFLOPS
      half4    : 19.1 TFLOPS

    Double-precision compute
      double   : 650 GFLOPS
      double2  : 647 GFLOPS
      double4  : 640 GFLOPS

    Mixed-precision compute fp16xfp16+fp32
      mp       : 9.31 TFLOPS
      mp2      : 9.32 TFLOPS
      mp4      : 9.42 TFLOPS

    BF16 compute bf16xbf16+fp32
      bf16     : 9.17 TFLOPS
      bf16_2   : 9.30 TFLOPS
      bf16_4   : 9.28 TFLOPS

    Integer compute int32
      int      : 1.81 TOPS
      int2     : 1.88 TOPS
      int4     : 1.82 TOPS

    INT8 dot-product compute
      int8_dp  : 28.0 TOPS
      int8_dp2 : 26.3 TOPS
      int8_dp4 : 27.1 TOPS

    Cooperative matrix
      fp16     : 67.3 TFLOPS
      fp16 f16acc : 68.9 TFLOPS
      bf16        : 68.9 TFLOPS
      int8        : 141 TOPS

    Global memory bandwidth
      float    : 196 GB/s
      float2   : 206 GB/s
      float4   : 212 GB/s

    Local memory bandwidth
      float    : 2.55 TB/s
      float2   : 3.65 TB/s
      float4   : 4.23 TB/s

    Image memory bandwidth
      float4   : 213 GB/s

    Transfer bandwidth
      h2d      : 4.81 GB/s
      d2h      : 5.48 GB/s

    Kernel launch latency
      dispatch : 48.7 us
      roundtrip : 113 us

Real performance is much higher than the "displayed" Speed 2.5GT/s, Width x1.

Regarding upstream Linux, it should be addressed with https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=f24df84cbe05e4471c04ac4b921fc0340bbc7752

Although, I have no ETA on when it will land to distros.

@DustyArmstrong said:

@TeddyAstie yarp.

My bad, the VM has it as 00:08.0 but on the host it's actually 00:06.0, I just didn't think about the specifics of your request!

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at d000 [size=256]
	Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] MSI-X: Enable- Count=4 Masked-
		Vector table: BAR=5 offset=00042000
		PBA: BAR=5 offset=00043000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270 v1] #19
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000100, Page Request Allocation: 00000000
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [400 v1] #25
	Capabilities: [410 v1] #26
	Capabilities: [440 v1] #27
	Kernel driver in use: pciback

thanks.

So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly.

That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools).

I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.

@gb.123 said in XCP-ng 8.3 updates announcements and testing:

Here is the summary:

If USB Keyboard & Mouse is passed-through along-with GPU:
The GPU gets stuck in D3 state (on Shutdown/Restart of VM) (Classic GPU reset problem)

If no vUSB is passed but GPU is passed through:
The GPU works correctly and resets correctly (on Shutdown/Restart of VM)

I have no clue what vUSB may change regarding GPU passthrough.

When I run :

$> lspci
Extract of Output (Partial):

07:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8

However, this controller does not show up when I run :
xe pci-list

Is it a bug that lspci & xe pci-list have different number of devices ?

How can I pass this controller since xe pci-list does not show it so I can't get the UUID ?
Will kernel parameters (like XCP-ng 8.2) work in this case ?

Question for @Team-XAPI-Network regarding the filtering on PCI IDs.
I don't think XAPI allows using arbitrary BDF, but I may be wrong.

Is it safe to run on XCP-ng host ?

 echo 1 > /sys/bus/pci/rescan

(I'm trying to find a way where the PCI card is reset by the host without complete reboot, though I am aware that the above command will not reset it.)

Probably. But it's not going to change anything as the device doesn't completely leave the Dom0 when passed-through.
FYI a function-level-reset is systematically performed by Xen when doing PCI passthrough, thus your device should be reset before entering another guest (aside reset bugs like you may have).

Also is it advisable to use :

xl pci-assignable-add 07:00.0

in XCP-ng 8.3 ? or is this method deprecated ?

I don't think XAPI supports this PCI passthrough approach.
This is a command which allows dynamically to remove a device from Dom0 and put it into "quarantine domain", so that it will be ready to passthrough it.

Current XAPI uses the approach of having a set of "passthrough-able" devices at boot time by modifying the xen-pciback.hide kernel parameter, which does the same but at boot time.

Hello !

I am looking to get some feedback and evaluation on a performance-related patch for Xen (XCP-ng 8.3 only).
This patch changes the memcpy implementation of Xen to use the "ERMS variant" (aka REP MOVSB) instead of the current REP MOVSQ+B implementation.
This is expected to perform better on the vast majority of Intel CPUs and modern AMD ones (Zen3+), but may perform worse on some older AMD CPUs.

This change may impact the performance of PV drivers (especially network).

You can find more details regarding this proposed change in : https://github.com/xcp-ng-rpms/xen/pull/54
This change may be reworked in the future to take more in account the specificities of each CPUs (e.g check presence of ERMS flag).

Keep in mind that this patched version is experimental and not officially supported.

Installation :

# Download repo file for XCP-ng 8.3
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

# Installing the patched Xen packages (you should see `.erms` packages)
yum update --enablerepo=xcp-ng-tae1

You can revert the changes by downgrading the Xen package with the ones in the default repos.

yum downgrade --disablerepo=xcp-ng-tae1 "xen-*"

Consider this feature as experimental

Many Intel CPUs have support for Intel APICv (APIC Virtualization) features that aims at reducing the number of APIC-related VM-Exits in guests, hence improving performance.
This has been introduced circa Intel Ivy Bridge-EP, and is also available on somewhat recent client CPUs as well. But keep in mind that not all SKUs have support for it.

Xen has support for it since roughly Xen 4.3, yet XenServer (and XCP-ng) originally disabled by default due to issues at this time around live migration between hardware that supports it and ones that don't.

Internal testing on the feature shows performance improvements in various benchmarks (5-10% in I/O benchmarks up to 30% on nginx with a Ice Lake Xeon). We didn't test on other CPUs, but older CPUs (in particular those affected by L1TF/Foreshadow) may observe larger improvements.

How to enable

This feature exists in the Xen version shipped in XCP-ng, but disabled by default, you can enable support on the host with

/opt/xensource/libexec/xen-cmdline --set-xen apicv=1
# then reboot

Once enabled, you can check hardware support with

xl dmesg | grep -A 25 "VMX: Supported advanced features"

And look for APIC Register Virtualization and Virtual Interrupt Delivery.

The feature can be disabled with

/opt/xensource/libexec/xen-cmdline --delete-xen apicv
# then reboot

What to test

Along with general feedback on the feature (does it work, supported hardware, ...), we're interested on :

• interactions with live migration, especially with hosts that have different configuration of APICv
• performance on various workloads (web server, databases, I/O, ...)

What about AMD ?

AMD counterpart is AVIC (Advanced Virtualized Interrupt Controller) introduced (in practice) with Zen 2 (but also existed on some Carrizo CPUs) then improved with x2AVIC with Zen 4.
Xen support for (x2)AVIC is currently work in progress.
Refer to Denis Mukhin's presentation on AVIC for Xen (Xen Summit 2026) : https://youtu.be/RhaJsOU_PIA

Xen Project covered this as XSA-489.

@plaidypus I don't know a lot about NUMA on Xen, but we have a part in the docs regarding that
https://docs.xcp-ng.org/compute/#numa-affinity

And also other documentation on the subject
https://xapi-project.github.io/new-docs/toolstack/features/NUMA/index.html
there was a design session regarding NUMA in latest Xen Summit : https://youtu.be/KoNwEYMlhyU?list=PLQMQQsKgvLnvjRgDnb-5T51e1kGHgs1SO

The rule is oddly written, and may conflict with another similar one that already exist in the distro (hence may not be useful to begin with).

The modern generic rule for doing vCPU hotplug is, which would be preferable to the current z10-xen-vcpu-hotplug.rules.
ACTION=="add", SUBSYSTEM=="cpu", ATTR{online}=="0", ATTR{online}="1"

@dcskinner @olivierlambert

You can read key/values from the xenstore, and write some (from VM to outside), but you cannot write values "in live" from outside the VM to the inside.

It is, but XAPI doesn't provide a interface for it.

do the guest tools quiesce the filesystems before snapshotting?

Tools are aware of a snapshot so you don't have blocks in flight.

do the guest tools quiesce the filesystems before snapshotting?

Guests kernel are aware, as it is them that are performing a "suspend" on toolstack request (thus quiece filesystems); although "tools" can only observe that the system has been suspended after the fact by measuring side effects, and not orchestrate it.

It's because suspend/resume operation doesn't come from "guest tools" actually, but instead from the kernel drivers. So userland tools has no say on it.