@gb.123 said in XCP-ng 8.3 updates announcements and testing:

Here is the summary:

If USB Keyboard & Mouse is passed-through along-with GPU:
The GPU gets stuck in D3 state (on Shutdown/Restart of VM) (Classic GPU reset problem)

If no vUSB is passed but GPU is passed through:
The GPU works correctly and resets correctly (on Shutdown/Restart of VM)

I have no clue what vUSB may change regarding GPU passthrough.

When I run :

$> lspci
Extract of Output (Partial):

07:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8

However, this controller does not show up when I run :
xe pci-list

Is it a bug that lspci & xe pci-list have different number of devices ?

How can I pass this controller since xe pci-list does not show it so I can't get the UUID ?
Will kernel parameters (like XCP-ng 8.2) work in this case ?

Question for @Team-XAPI-Network regarding the filtering on PCI IDs.
I don't think XAPI allows using arbitrary BDF, but I may be wrong.

Is it safe to run on XCP-ng host ?

 echo 1 > /sys/bus/pci/rescan

(I'm trying to find a way where the PCI card is reset by the host without complete reboot, though I am aware that the above command will not reset it.)

Probably. But it's not going to change anything as the device doesn't completely leave the Dom0 when passed-through.
FYI a function-level-reset is systematically performed by Xen when doing PCI passthrough, thus your device should be reset before entering another guest (aside reset bugs like you may have).

Also is it advisable to use :

xl pci-assignable-add 07:00.0

in XCP-ng 8.3 ? or is this method deprecated ?

I don't think XAPI supports this PCI passthrough approach.
This is a command which allows dynamically to remove a device from Dom0 and put it into "quarantine domain", so that it will be ready to passthrough it.

Current XAPI uses the approach of having a set of "passthrough-able" devices at boot time by modifying the xen-pciback.hide kernel parameter, which does the same but at boot time.

Hello !

I am looking to get some feedback and evaluation on a performance-related patch for Xen (XCP-ng 8.3 only).
This patch changes the memcpy implementation of Xen to use the "ERMS variant" (aka REP MOVSB) instead of the current REP MOVSQ+B implementation.
This is expected to perform better on the vast majority of Intel CPUs and modern AMD ones (Zen3+), but may perform worse on some older AMD CPUs.

This change may impact the performance of PV drivers (especially network).

You can find more details regarding this proposed change in : https://github.com/xcp-ng-rpms/xen/pull/54
This change may be reworked in the future to take more in account the specificities of each CPUs (e.g check presence of ERMS flag).

Keep in mind that this patched version is experimental and not officially supported.

Installation :

# Download repo file for XCP-ng 8.3
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

# Installing the patched Xen packages (you should see `.erms` packages)
yum update --enablerepo=xcp-ng-tae1

You can revert the changes by downgrading the Xen package with the ones in the default repos.

yum downgrade --disablerepo=xcp-ng-tae1 "xen-*"

TSnake41 opened this pull request in xcp-ng-rpms/xen

draft Use ERMS variant for memcpy #54

@plaidypus I don't know a lot about NUMA on Xen, but we have a part in the docs regarding that
https://docs.xcp-ng.org/compute/#numa-affinity

And also other documentation on the subject
https://xapi-project.github.io/new-docs/toolstack/features/NUMA/index.html
there was a design session regarding NUMA in latest Xen Summit : https://youtu.be/KoNwEYMlhyU?list=PLQMQQsKgvLnvjRgDnb-5T51e1kGHgs1SO

Hello,

Make sure Intel VMD is disabled (this is the hardware RAID feature of Intel, and it doesn't currently work on XCP-ng; you probably don't need it unless you are looking to make a RAID). We found some modern platforms enabling by default (which also causes issues with Windows).

@tuxen said (https://xcp-ng.org/forum/topic/3652/no-free-virtual-function-found-vgpu-s7150/4?_=1731502751059)

After some digging, could be the case of a GPU firmware being incompatible with UEFI. Do you have any spare server for testing XCP-ng boot in legacy/BIOS with this GPU?

Perhaps it is the issue ?

@ohajek

Nov 13 11:30:21 xen03 kernel: [10188.720655] AMD IOMMUv2 driver by Joerg Roedel jroedel@suse.de
Nov 13 11:30:21 xen03 kernel: [10188.720656] AMD IOMMUv2 functionality not available on this system

This is expected, Dom0 Kernel (Linux) is not supposed to access the IOMMU when it is already used by Xen. To check if AMD-Vi is working, you need to check xl dmesg instead.

I took a quick look at kern_gim_compiled.txt, and it look likes it timed-out somewhere

Oct 23 20:49:32 xen03 kernel: [   80.657394]        gim error:(wait_cmd_complete:2387)  wait_cmd_complete -- time out after 0.003004460 sec
Oct 23 20:49:32 xen03 kernel: [   80.657408]        gim error:(wait_cmd_complete:2390)   Cmd = 0x17, Status = 0x0, cmd_Complete=0

3ms looks like a short timeout for me, but aside that, it looks like a driver(gim) or hardware issue

@abudef Note that even with this update, nested virtualization is still not really supported in XCP-ng 8.3.
It's there, you can enable it at your own risk. It broke due to some change in XAPI (even though Xen hypervisor had "support" for it).
It never actually got removed from Xen hypervisor (it was marked experimental in Xen 4.13 used in XCP-ng 8.2, it is also the case for Xen 4.17), although nothing really changed, it still has the same issues and limitations as said.

The current state of nested virtualization in Xen is quite clumsy and there are future plans to remake it properly from ground without taking shortcuts and have proper tests to back it.

Aside that, after some experiments, it seems that mostly nested EPT is incomplete/buggy, so your L1 hypervisor should not rely on it. You should add hap=0 to nested XCP-ng Xen cmdline. Beware that it will imply a pretty large performance hit, but I had more consistent results with this.
I am quite suprised that Windows works while Linux don't, maybe it is somewhat related to PV drivers ?

@hoh said in Early testable PVH support:

Then I tried to get rid of the (fake) UEFI magic.

Well, it is actually a full standard UEFI implementation, but that works in PVH instead of HVM.

I thought it should work to just change the PV-bootloader to pygrub. Calling pygrub on the disk image works fine and is able to extract the images and args

# pygrub -l alpine.img
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
title: Alpine, with Linux virt
  root: None
  kernel: /boot/vmlinuz-virt
  args: root=UUID=4c6dcb06-20ff-4bcf-be4d-cb399244c4c6 ro  rootfstype=ext4 console=hvc0
  initrd: /boot/initramfs-virt

But starting the VM fails. It looks like it starts but then immediately something calls force shutdown, I'll dive deeper into the logs later.

But setting everything manually actually works. If extract the kernel and initrd to dom-0 and configure

PV-kernel=/var/lib/xcp/guest/kernel
PV-ramdisk=/var/lib/xcp/guest/ramdisk
PV-args="root=/dev/xvda1 ro rootfstype=ext4 console=hvc0"

it boots and I looks pretty much the same as with the pvh-ovmf magic. So perhaps the idea to use pygrub is wrong.

I don't know how good is supported pygrub nowadays, especially since PV support got deprecated in XCP-ng 8.2 then completely dropped in XCP-ng 8.3; with the pv-shim (pv-in-pvh) being the only remaining (but not endorsed) way of booting some PV guests today.

In my tests, pygrub was very clunky and rarely work as I expect. In practice (what upstream Xen Project mostly uses), it got replaced with pvgrub/pvhgrub and pvh-ovmf (OvmfXen) which are more reliable and less problematic security-wise (runs in the guest rather than in the dom0).
(for using pvhgrub, you need to set a pvhgrub binary (grub-i386-xen_pvh.bin which is packaged by some distros like Alpine Linux's grub-xenhost) as kernel like done with pvh-ovmf)

@SethNY said in Windows 11 WSL2 is not supported with your current machine configuration:

XCP-ng 8.3, XO from sources.
Created Win11 from ISO using the built-in Windows 11 template
Configured and turned into template for cloning.

Trouble installing WSL on the cloned Win11:
WSL2 is not supported with your current machine configuration

This worked a couple years ago Win10 on XCP-ng 8.2, Ubuntu 22.04.

From administrative powershell

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
(Accept the reboot, back to administrative powersehll)
wsl --install
(fails)
wsl --list --online
wsl --install -d Ubuntu-24.04
(fails again)

I tried enabling Nested Virtualization for the VM without success
Booted to (F2) BIOS and confirmed not virtualization options there to enable

Has anyone got Win11/WSL/8.3 working? I'm hoping it's not due to not installing a Windows license key.

You were probably using WSL 1 before, and now using WSL 2 (which requires Hyper-V thus nested virtualization). And I'm not aware of WSL 2 working on XCP-ng 8.2, I assume you were actually using WSL 1 previously.

You can use WSL 1 by using

wsl.exe --set-default-version 1

and

wsl.exe --set-version <Distro> 1

@Maelstrom96 said in Epyc VM to VM networking slow:

What is the exact kernel patch that is required for the xen-platform-pci-bar-uc=false fix to work on a Linux guest? We're looking at potentially compiling our own kernel with the xen-netfront.c patch, and we would like to see about adding the other part of the Kernel code needed for the Grant table fix.

Patch is in Linux since 5.19-rc. You also find it in some stable branches like 5.15.

Otherwise, you can check this patch https://lore.kernel.org/xen-devel/ea4945df138527ed63e711cb77e3b333f7b3a4c9.1751633056.git.teddy.astie@vates.tech/

@SethNY said in Windows 11 WSL2 is not supported with your current machine configuration:

XCP-ng 8.3, XO from sources.
Created Win11 from ISO using the built-in Windows 11 template
Configured and turned into template for cloning.

Trouble installing WSL on the cloned Win11:
WSL2 is not supported with your current machine configuration

This worked a couple years ago Win10 on XCP-ng 8.2, Ubuntu 22.04.

From administrative powershell

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
(Accept the reboot, back to administrative powersehll)
wsl --install
(fails)
wsl --list --online
wsl --install -d Ubuntu-24.04
(fails again)

I tried enabling Nested Virtualization for the VM without success
Booted to (F2) BIOS and confirmed not virtualization options there to enable

Has anyone got Win11/WSL/8.3 working? I'm hoping it's not due to not installing a Windows license key.

You were probably using WSL 1 before, and now using WSL 2 (which requires Hyper-V thus nested virtualization). And I'm not aware of WSL 2 working on XCP-ng 8.2, I assume you were actually using WSL 1 previously.

You can use WSL 1 by using

wsl.exe --set-default-version 1

and

wsl.exe --set-version <Distro> 1

@McHenry said in pfSense Guest Tools:

I have been using pfSense with xcp-ng for a while now without installing the guest tools.

Due to some networking complications I have decided to install the guest tools to eliminate this as the cause.

Q1) Are the guest tools required on pfSense and what do they do?

Actually no, their main functionnality is to provide some data (memory usage, IPs) to XCP-ng to report them upward, AFAICT, it doesn't impact behavior in a significant way. It's actually not "PV drivers".

Q2) Are these tools being maintained?

This version of the guest agent is very limited and ancient (I don't exactly what it does, but not much IIRC).

We're working on a new guest agent in Rust, which actually works pretty well on Linux, Windows (actually used in latest Windows drivers) and FreeBSD (NetBSD is also possible, but current code lacks some platform-specific bits). But we still need to sort-out some issues (some technical and some others non-technical) before making it broadly available / making it packaged.

@Maelstrom96 said in Epyc VM to VM networking slow:

What is the exact kernel patch that is required for the xen-platform-pci-bar-uc=false fix to work on a Linux guest? We're looking at potentially compiling our own kernel with the xen-netfront.c patch, and we would like to see about adding the other part of the Kernel code needed for the Grant table fix.

Patch is in Linux since 5.19-rc. You also find it in some stable branches like 5.15.

Otherwise, you can check this patch https://lore.kernel.org/xen-devel/ea4945df138527ed63e711cb77e3b333f7b3a4c9.1751633056.git.teddy.astie@vates.tech/

@joeymorin said in Building from source fails with commit cb96de6:

Greetings all,

Subject pretty much says it. Build output goes sideways starting with:

@xen-orchestra/qcow2:build: src/disk/QcowDisk.mts(94,52): error TS2365: Operator '&' cannot be applied to types 'bigint' and 'number'.

Github CI is failing on this commit, try reverting to a commit that pass CI.

@rk9268vc said in Having issues installing StartOS as a VM. Cant detect a disk for it to install to.:

@TeddyAstie so can i just not run this OS on xcp-ng? is there no workaround? Would this run on proxmox?

you can try adding to Linux command-line (in grub) xen_emul_unplug=never to disable PV drivers, thus making udev see ATA/NVMe drivers, maybe that's enough as long it's only the installer

@rk9268vc said in Having issues installing StartOS as a VM. Cant detect a disk for it to install to.:

@TeddyAstie so is this fixed in a newer version of xcp-ng? and if so, can i update live?

it's not fixable from XCP-ng side, it is a bug in Debian/others...

This happens because StartOS uses /dev/disk/by-path to get disk list and due to missing rules in 60-persistent-storage.rules (thus systemd/udev), Xen PV disks don't get their disks populated in /dev/disk/by-id.

In other words, it is a systemd bug which wants to be fixed, but getting that backported to older distros is gonna get tricky.

@Maelstrom96 said in Kioxia CM7 PCIe pass-through crash:

@TeddyAstie It's a newly deployed Rocky Linux 9.6 with all the latest updates applied to it.

Nested virtualization is disabled.

Can you do

xl debug-key G
xl debug-key +

and try restarting the guest to try having more informations.

Also, note that your installation is not up to date (even though I don't think it is going to change anything).

What guest you are trying to boot ?
Is nested virtualization enabled for this guest, if it is enabled, does that work if you disable it ?

@mickwilli said in Cannot Install Windows 10 in New VM:

@dinhngtu Thanks. I'll try and give that a go in the next few days and will report back.

Note that I have Windows 10 1803 installed without any issues. Trying to update to 22H2 fails with the same blue screen error as you'd expect. Sadly, no crash dump as it seems to fail very early in the boot process.

Do you get any additional BSOD details in xl dmesg ?