Regarding upstream Linux, it should be addressed with https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=f24df84cbe05e4471c04ac4b921fc0340bbc7752

Although, I have no ETA on when it will land to distros.

@DustyArmstrong said:

@TeddyAstie yarp.

My bad, the VM has it as 00:08.0 but on the host it's actually 00:06.0, I just didn't think about the specifics of your request!

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at d000 [size=256]
	Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] MSI-X: Enable- Count=4 Masked-
		Vector table: BAR=5 offset=00042000
		PBA: BAR=5 offset=00043000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270 v1] #19
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000100, Page Request Allocation: 00000000
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [400 v1] #25
	Capabilities: [410 v1] #26
	Capabilities: [440 v1] #27
	Kernel driver in use: pciback

thanks.

So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly.

That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools).

I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.

@gb.123 said in XCP-ng 8.3 updates announcements and testing:

Here is the summary:

If USB Keyboard & Mouse is passed-through along-with GPU:
The GPU gets stuck in D3 state (on Shutdown/Restart of VM) (Classic GPU reset problem)

If no vUSB is passed but GPU is passed through:
The GPU works correctly and resets correctly (on Shutdown/Restart of VM)

I have no clue what vUSB may change regarding GPU passthrough.

When I run :

$> lspci
Extract of Output (Partial):

07:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8

However, this controller does not show up when I run :
xe pci-list

Is it a bug that lspci & xe pci-list have different number of devices ?

How can I pass this controller since xe pci-list does not show it so I can't get the UUID ?
Will kernel parameters (like XCP-ng 8.2) work in this case ?

Question for @Team-XAPI-Network regarding the filtering on PCI IDs.
I don't think XAPI allows using arbitrary BDF, but I may be wrong.

Is it safe to run on XCP-ng host ?

 echo 1 > /sys/bus/pci/rescan

(I'm trying to find a way where the PCI card is reset by the host without complete reboot, though I am aware that the above command will not reset it.)

Probably. But it's not going to change anything as the device doesn't completely leave the Dom0 when passed-through.
FYI a function-level-reset is systematically performed by Xen when doing PCI passthrough, thus your device should be reset before entering another guest (aside reset bugs like you may have).

Also is it advisable to use :

xl pci-assignable-add 07:00.0

in XCP-ng 8.3 ? or is this method deprecated ?

I don't think XAPI supports this PCI passthrough approach.
This is a command which allows dynamically to remove a device from Dom0 and put it into "quarantine domain", so that it will be ready to passthrough it.

Current XAPI uses the approach of having a set of "passthrough-able" devices at boot time by modifying the xen-pciback.hide kernel parameter, which does the same but at boot time.

Hello !

I am looking to get some feedback and evaluation on a performance-related patch for Xen (XCP-ng 8.3 only).
This patch changes the memcpy implementation of Xen to use the "ERMS variant" (aka REP MOVSB) instead of the current REP MOVSQ+B implementation.
This is expected to perform better on the vast majority of Intel CPUs and modern AMD ones (Zen3+), but may perform worse on some older AMD CPUs.

This change may impact the performance of PV drivers (especially network).

You can find more details regarding this proposed change in : https://github.com/xcp-ng-rpms/xen/pull/54
This change may be reworked in the future to take more in account the specificities of each CPUs (e.g check presence of ERMS flag).

Keep in mind that this patched version is experimental and not officially supported.

Installation :

# Download repo file for XCP-ng 8.3
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

# Installing the patched Xen packages (you should see `.erms` packages)
yum update --enablerepo=xcp-ng-tae1

You can revert the changes by downgrading the Xen package with the ones in the default repos.

yum downgrade --disablerepo=xcp-ng-tae1 "xen-*"

TSnake41 opened this pull request in xcp-ng-rpms/xen

draft Use ERMS variant for memcpy #54

Xen Project covered this as XSA-489.

@plaidypus I don't know a lot about NUMA on Xen, but we have a part in the docs regarding that
https://docs.xcp-ng.org/compute/#numa-affinity

And also other documentation on the subject
https://xapi-project.github.io/new-docs/toolstack/features/NUMA/index.html
there was a design session regarding NUMA in latest Xen Summit : https://youtu.be/KoNwEYMlhyU?list=PLQMQQsKgvLnvjRgDnb-5T51e1kGHgs1SO

The rule is oddly written, and may conflict with another similar one that already exist in the distro (hence may not be useful to begin with).

The modern generic rule for doing vCPU hotplug is, which would be preferable to the current z10-xen-vcpu-hotplug.rules.
ACTION=="add", SUBSYSTEM=="cpu", ATTR{online}=="0", ATTR{online}="1"

@dcskinner @olivierlambert

You can read key/values from the xenstore, and write some (from VM to outside), but you cannot write values "in live" from outside the VM to the inside.

It is, but XAPI doesn't provide a interface for it.

do the guest tools quiesce the filesystems before snapshotting?

Tools are aware of a snapshot so you don't have blocks in flight.

do the guest tools quiesce the filesystems before snapshotting?

Guests kernel are aware, as it is them that are performing a "suspend" on toolstack request (thus quiece filesystems); although "tools" can only observe that the system has been suspended after the fact by measuring side effects, and not orchestrate it.

It's because suspend/resume operation doesn't come from "guest tools" actually, but instead from the kernel drivers. So userland tools has no say on it.

Hello,

Make sure Intel VMD is disabled (this is the hardware RAID feature of Intel, and it doesn't currently work on XCP-ng; you probably don't need it unless you are looking to make a RAID). We found some modern platforms enabling by default (which also causes issues with Windows).

@tuxen said (https://xcp-ng.org/forum/topic/3652/no-free-virtual-function-found-vgpu-s7150/4?_=1731502751059)

After some digging, could be the case of a GPU firmware being incompatible with UEFI. Do you have any spare server for testing XCP-ng boot in legacy/BIOS with this GPU?

Perhaps it is the issue ?

@ohajek

Nov 13 11:30:21 xen03 kernel: [10188.720655] AMD IOMMUv2 driver by Joerg Roedel jroedel@suse.de
Nov 13 11:30:21 xen03 kernel: [10188.720656] AMD IOMMUv2 functionality not available on this system

This is expected, Dom0 Kernel (Linux) is not supposed to access the IOMMU when it is already used by Xen. To check if AMD-Vi is working, you need to check xl dmesg instead.

I took a quick look at kern_gim_compiled.txt, and it look likes it timed-out somewhere

Oct 23 20:49:32 xen03 kernel: [   80.657394]        gim error:(wait_cmd_complete:2387)  wait_cmd_complete -- time out after 0.003004460 sec
Oct 23 20:49:32 xen03 kernel: [   80.657408]        gim error:(wait_cmd_complete:2390)   Cmd = 0x17, Status = 0x0, cmd_Complete=0

3ms looks like a short timeout for me, but aside that, it looks like a driver(gim) or hardware issue

@antest can you retry after doing /opt/xensource/libexec/xen-cmdline --set-xen iommu=debug then rebooting.
And also reporting full DomU and Dom0 dmesg (in addition to xl dmesg) ?

@abudef Note that even with this update, nested virtualization is still not really supported in XCP-ng 8.3.
It's there, you can enable it at your own risk. It broke due to some change in XAPI (even though Xen hypervisor had "support" for it).
It never actually got removed from Xen hypervisor (it was marked experimental in Xen 4.13 used in XCP-ng 8.2, it is also the case for Xen 4.17), although nothing really changed, it still has the same issues and limitations as said.

The current state of nested virtualization in Xen is quite clumsy and there are future plans to remake it properly from ground without taking shortcuts and have proper tests to back it.

Aside that, after some experiments, it seems that mostly nested EPT is incomplete/buggy, so your L1 hypervisor should not rely on it. You should add hap=0 to nested XCP-ng Xen cmdline. Beware that it will imply a pretty large performance hit, but I had more consistent results with this.
I am quite suprised that Windows works while Linux don't, maybe it is somewhat related to PV drivers ?

@Andrew said:
HP DL G8 Intel E5-2673 v2 shows 64 CPUs. The actual CPUs show correctly, the higher ones (that don't exist) show [CPUxx] Unable to fetch temperature (19 - No such device)

Machines with hotpluggable CPUs are a pretty tricky case, the logic tries up to "maximum possible CPU" and fails here because the CPU is not online (No such device error). That doesn't prevent the temperature from getting fetched for CPU that exists.
I can try to add a check to hide this specific error, so it's doesn't create noise for offline CPUs on such machines.

@acebmxer said:

@rzr

Just installed updates on host 1. Once host rebooted it took an extra min or two to reconnect to xo, but did finally connect. Applying updates on host 2 now.

Update - host2 no issues. Once reboot complete it connected to xo as expected without delay.

I see these updates include -

• xen: Add support for xenpm get-core-temp to query CPU temperature on Intel platforms.

• Use xenpm get-core-temp to get the temperature on Intel's CPU, to fallback unsupported coretemp. Doc update being reviewed .

My host are AMD so can verify these. I might be able to deploy a Intel host later tonight. Will this come to AMD later?

AMD rely on a different method to expose the temperature, that don't require this xenpm-based approach. In principle, it should already work with plain sensors (through k10temp), but our driver may not be up to date for recent AMD CPUs.

@hoh said in Early testable PVH support:

Then I tried to get rid of the (fake) UEFI magic.

Well, it is actually a full standard UEFI implementation, but that works in PVH instead of HVM.

I thought it should work to just change the PV-bootloader to pygrub. Calling pygrub on the disk image works fine and is able to extract the images and args

# pygrub -l alpine.img
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
title: Alpine, with Linux virt
  root: None
  kernel: /boot/vmlinuz-virt
  args: root=UUID=4c6dcb06-20ff-4bcf-be4d-cb399244c4c6 ro  rootfstype=ext4 console=hvc0
  initrd: /boot/initramfs-virt

But starting the VM fails. It looks like it starts but then immediately something calls force shutdown, I'll dive deeper into the logs later.

But setting everything manually actually works. If extract the kernel and initrd to dom-0 and configure

PV-kernel=/var/lib/xcp/guest/kernel
PV-ramdisk=/var/lib/xcp/guest/ramdisk
PV-args="root=/dev/xvda1 ro rootfstype=ext4 console=hvc0"

it boots and I looks pretty much the same as with the pvh-ovmf magic. So perhaps the idea to use pygrub is wrong.

I don't know how good is supported pygrub nowadays, especially since PV support got deprecated in XCP-ng 8.2 then completely dropped in XCP-ng 8.3; with the pv-shim (pv-in-pvh) being the only remaining (but not endorsed) way of booting some PV guests today.

In my tests, pygrub was very clunky and rarely work as I expect. In practice (what upstream Xen Project mostly uses), it got replaced with pvgrub/pvhgrub and pvh-ovmf (OvmfXen) which are more reliable and less problematic security-wise (runs in the guest rather than in the dom0).
(for using pvhgrub, you need to set a pvhgrub binary (grub-i386-xen_pvh.bin which is packaged by some distros like Alpine Linux's grub-xenhost) as kernel like done with pvh-ovmf)

@SethNY said in Windows 11 WSL2 is not supported with your current machine configuration:

XCP-ng 8.3, XO from sources.
Created Win11 from ISO using the built-in Windows 11 template
Configured and turned into template for cloning.

Trouble installing WSL on the cloned Win11:
WSL2 is not supported with your current machine configuration

This worked a couple years ago Win10 on XCP-ng 8.2, Ubuntu 22.04.

From administrative powershell

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Windows-Subsystem-Linux
(Accept the reboot, back to administrative powersehll)
wsl --install
(fails)
wsl --list --online
wsl --install -d Ubuntu-24.04
(fails again)

I tried enabling Nested Virtualization for the VM without success
Booted to (F2) BIOS and confirmed not virtualization options there to enable

Has anyone got Win11/WSL/8.3 working? I'm hoping it's not due to not installing a Windows license key.

You were probably using WSL 1 before, and now using WSL 2 (which requires Hyper-V thus nested virtualization). And I'm not aware of WSL 2 working on XCP-ng 8.2, I assume you were actually using WSL 1 previously.

You can use WSL 1 by using

wsl.exe --set-default-version 1

and

wsl.exe --set-version <Distro> 1

@Maelstrom96 said in Epyc VM to VM networking slow:

What is the exact kernel patch that is required for the xen-platform-pci-bar-uc=false fix to work on a Linux guest? We're looking at potentially compiling our own kernel with the xen-netfront.c patch, and we would like to see about adding the other part of the Kernel code needed for the Grant table fix.

Patch is in Linux since 5.19-rc. You also find it in some stable branches like 5.15.

Otherwise, you can check this patch https://lore.kernel.org/xen-devel/ea4945df138527ed63e711cb77e3b333f7b3a4c9.1751633056.git.teddy.astie@vates.tech/

@chicagomed said in Passthrough Contention Problems with Console and Linux VM:

@TeddyAstie great will take a look this weekend. Is there anything in particular you want us to test / check out?

What works/doesn't work and overall performance.

PCIe AER needs proper PCIe, which in practice needs Q35 chipset in the guest (or some other guest type/PCI passthrough way).

Q35 support is currently work in progress