What is the exact kernel patch that is required for the xen-platform-pci-bar-uc=false fix to work on a Linux guest? We're looking at potentially compiling our own kernel with the xen-netfront.c patch, and we would like to see about adding the other part of the Kernel code needed for the Grant table fix.

Patch is in Linux since 5.19-rc. You also find it in some stable branches like 5.15.

Otherwise, you can check this patch https://lore.kernel.org/xen-devel/ea4945df138527ed63e711cb77e3b333f7b3a4c9.1751633056.git.teddy.astie@vates.tech/

@Forza There's a new script here that will help you check the VM's status wrt. the Fix 1.

Thank you. It does indeed look like the EPYC fix is active in XOA.

[07:25 22] xoa:~$ python3 epyc-fix-check.py
'xen-platform-pci' PCI IO mem address is 0xFB000000
Grant table cacheability fix is ACTIVE.

Has Vates checked if a newer kernel would help the network performance with XOA?

Current kernel is: linux-image-amd64/oldstable,now 6.1.148-1 amd64 [installed]

When trying to install any of the newer kernels (6.12.43-*) it immediately fails dependency check:

[07:30 22] xoa:~$ apt install linux-image-6.12.43+deb12-
linux-image-6.12.43+deb12-amd64                 linux-image-6.12.43+deb12-cloud-amd64-unsigned
linux-image-6.12.43+deb12-amd64-dbg             linux-image-6.12.43+deb12-rt-amd64
linux-image-6.12.43+deb12-amd64-unsigned        linux-image-6.12.43+deb12-rt-amd64-dbg
linux-image-6.12.43+deb12-cloud-amd64           linux-image-6.12.43+deb12-rt-amd64-unsigned
linux-image-6.12.43+deb12-cloud-amd64-dbg

[07:30 22] xoa:~$ apt install linux-image-6.12.43+deb12-amd64
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 linux-image-6.12.43+deb12-amd64 : PreDepends: linux-base (>= 4.12~) but 4.9 is to be installed
E: Unable to correct problems, you have held broken packages.

Kernel version could have an impact and being unrelated to the fix we provided. Get an even more recent kernel on your XOA to test (eg one from testing)

I was not able to update the kernel due to some issue in XOA installation. I commented on this at https://xcp-ng.org/forum/post/97522

But to clarify — my goal is to reach the same performance in XOA as with our other VMs. I had assumed it lacked the kernel support, and that led to the confusion. Sorry for that.

If you want to compare perf between kernels, do that with the same number of vCPUs ideally (and also the same env, ie different kernel versions with the same iperf version)

Sure. but in this context it is not that relevant as all other VMs got a boost while XOA didnt as much.

Does the Debian 6.1 kernel that XOA uses have the backported fixes mentioned in https://xcp-ng.org/blog/2025/09/01/september-2025-maintenance-update-for-xcp-ng-8-3/

Even if it has, it is clear that more resent kernels are much faster. Why not release a XOA with more recent kernels?

@Forza By default XOA VM has 2 vcpus, how many vcpus do your ubuntu have? Althrough iperf isn't running multithreaded in your test, there is one queue on the kernel side of the VM per vcpu to process packets.

I have 8 CPUS on XOA, 4 CPUs in the Alpine VM and 6 in the Ubuntu VM

P4 uses 4 threads with iperf3.

@bleader Are you saying you do not see any performance differences between XOA and VMs with more recent kernels?

XOA's kernel should have the capability already, as it's a Debian 12 with stock kernel. Also, the bottleneck is ONLY between VMs on the same host.

OK, but then I do not understand the huge difference.

Tested some VM-VM traffic on the same host:

Ubuntu (kernel 6.8) -> Alpine (kernel 6.12): 13.7Gbit/s, P4 = 23.8 Gbit/s
XOA (kernel 6.1) -> Ubuntu (kernel 6.8) : 5.5 Gbit/s, P4 = 17.5 Gbit/s
XOA (kernel 6.1) -> Alpine (kernel 6.12): 11.9 Gbit/s, P4 = 13.6 Gbit/s

And for measure against the bare metal NFS server:
Alpine -> NFS SR (kernel 6.12): 13.7 Gbit/s, P4 = 23.4 Gbit/s

@Forza What patch are you referring to that would relate to XOA?

It seems only recent kernels can take advantage of the improvements. From the blog post mentioned above:

The change only affects Linux guests. To make it effective, their kernel must support the feature which enables this fix.

Iperf inside XOA is much slower than other VMs (like the Ubundu 24.04 above).

When I run on the same host (pool master) against our NFS SR:

Ubuntu 24.04 (kernel 6.8):

# iperf3 -c 10.12.9.4
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  16.0 GBytes  13.7 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  16.0 GBytes  13.7 Gbits/sec                  receiver

# iperf3 -c 10.12.9.4 P4
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.26 GBytes  5.37 Gbits/sec  8861             sender
[  5]   0.00-10.00  sec  6.25 GBytes  5.37 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  8.57 GBytes  7.36 Gbits/sec  8372             sender
[  7]   0.00-10.00  sec  8.57 GBytes  7.36 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  9.05 GBytes  7.77 Gbits/sec  10192             sender
[  9]   0.00-10.00  sec  9.05 GBytes  7.77 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  6.12 GBytes  5.25 Gbits/sec  7144             sender
[ 11]   0.00-10.00  sec  6.11 GBytes  5.25 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  30.0 GBytes  25.8 Gbits/sec  34569             sender
[SUM]   0.00-10.00  sec  30.0 GBytes  25.8 Gbits/sec                  receiver

XOA 2025.08 (kernel 6.1):

# iperf3 -c 10.12.9.4
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  6.26 GBytes  5.37 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  6.25 GBytes  5.37 Gbits/sec                  receiver

# iperf3 -c 10.12.9.4 -P4
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  4.71 GBytes  4.05 Gbits/sec  3987             sender
[  5]   0.00-10.00  sec  4.71 GBytes  4.05 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  4.61 GBytes  3.96 Gbits/sec  3086             sender
[  7]   0.00-10.00  sec  4.61 GBytes  3.96 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  6.77 GBytes  5.81 Gbits/sec  7745             sender
[  9]   0.00-10.00  sec  6.77 GBytes  5.81 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  5.42 GBytes  4.65 Gbits/sec  629             sender
[ 11]   0.00-10.00  sec  5.42 GBytes  4.65 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  21.5 GBytes  18.5 Gbits/sec  15447             sender
[SUM]   0.00-10.00  sec  21.5 GBytes  18.5 Gbits/sec                  receiver

Ubuntu 24.04 VM (6 cores) -> bare metal server (6 cores) over a 2x25Gbit LACP link.

Pre-patch

• iperf3 -P1 : 9.72Gbit/s
• iperf3 -P6 : 14.6GBis/s

Post Patch

• iperf3 -P1 : 11.3GBit/s
• iperf3 -P6 : 24.2GBit/s

Ubuntu 24.04 VM (6 cores) -> Ubuntu 24.04 VM (6 cores) on the same host

Pre Patch

Forgot to test this...

Post Patch

• iperf3 -P1 : 13.7GBit/s
• iperf3 -P6 : 30.8GBit/s
• iperf3 -P24 : 40.4GBit/s

Our servers have Last-Level Cache (LLC) as NUMA Node enabled as most our VMs do not have huge amount of vCPUs assigned. This means for the EPYC 7402P (24c/48t) we have 8 NUMA nodes. We however do not use xl cpupool-numa-split.