Team - Hypervisor Kernel group | XCP-ng and XO forum

yannsionneau

Just FYI, we had some issues with some Debian VMs for instance, with old kernel
where the VM would suddenly take 100% CPU and would we completely frozen and unresponsive.
It was in fact Debian enabling the suspend, the guest would suspend and trigger a bug in xen PV driver in the guest kernel which would never be able to wake up anymore.
Newer kernel don't have this issue and are able to wake up.
I'm quoting Debian here but it might very well happen on other distro

TeddyAstie

@ravenet said:
@teddyastie separate from the above, a standing question whenever you have a minute: is it expected that passthrough GPU MSIs are never delivered at all on 8.3? Both my working rig and PessimistTech's broken one show the amdgpu xen-pirq MSI vector at zero on all CPUs, forever — everything survives on driver-side polling. It works, but it means any driver path that hard-depends on an interrupt has no safety net, and I suspect it's why the amdgpu ring resets in his log keep failing. Curious whether that's a known limitation of the pirq MSI path for passthrough devices or something worth a ticket.

TL;DR This is likely a known issue, and we're currently discussing on a potential fix.

There is a known bug regarding MSI on AMDGPU when using "hvm_pirq" ("use event channels for device interrupts"), see
https://lore.kernel.org/xen-devel/20240110095311.61809-1-roger.pau@citrix.com/
https://github.com/QubesOS/qubes-issues/issues/7971

The easy fix is to disable "HVM PIRQ" feature (which has been made disabled by default in Xen 4.19), but on the XCP-ng side, this is configured by XAPI Project which needs to be updated as well to apply this change (which appears to be a bit tricky for various reasons).

TeddyAstie

@PessimistTech
That looks a bit odd indeed.

In addition to what proposed @yannsionneau, can you also give the output of :

xl dmesg (in Dom0)
dmesg (in Dom0)
dmesg (in the guest)

So that we can try to pin-point what may be happening.

TeddyAstie

@JamesG said:

So I ordered some mini DP dummy plugs and took a look at this further today.

First thought...Just load Windows on the bare metal and see what the card does.

No problem. Loaded Win10, loaded the latest drivers, PCIe Gen4x8 lanes. Sweet. At least that works.

Put the XCP-ng drive back in, booted up, passed-through the GPU to a Win11 guest, updated the drivers, PCIe Gen1x1.

As I explained briefly in https://xcp-ng.org/forum/post/106479, this is mostly display as the real hardware may be running at a different speed.

Ultimately I need to install a Linux distro with a really current kernel on the bare metal and see how it sees the GPU. That might be a good project for tomorrow.

I think this just comes down to a combination of too new, not well supported hardware with some quirks and somewhat dated code and support in XCP-ng (ReBAR support in guests for example).

I did quick checks and I believe that ReBAR works; at least with UEFI guests (our OVMF build does support resizable bar), and it actually works as suggest https://xcp-ng.org/forum/post/106489 (VRAM BAR is 16 GB); but I didn't do extensive testing in that area.

yannsionneau

@PessimistTech Let's try to see if we have errors in the logs.

Run ls /boot/xen* and note the name of the xen file that ends with -d.gz

Then, can you reboot and , while in the grub menu, edit the XCP-ng menu entry (the first one) and make the following modifications:

modify the Xen ELF loaded by multiboot2 to load the debug Xen build by specifying the file that ends with -d.gz instead of /boot/xen.gz
on the same line, at the end, add iommu=debug

Then boot (by doing ctrl+X) and once booted and the VM with PCI passthrough started, please issue xl dmesg command in dom0 shell and report the content here.

Also please paste the result of dom0 dmesg.

then also an lspci -vvv -nn -s SBDF from both dom0 and the VM

replace SBDF with the segment bus device function address of the device (beware it won't be the same in dom0 and inside the VM).

Regards,

Yann

yannsionneau

Maybe let's ping the entire @Team-Hypervisor-Kernel here

yannsionneau

Hi @pessimisttech let me ping @Team-Hypervisor-Kernel here

yannsionneau

Hello @mgr42
I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng.
I am currently (slowly) working on it.

There are other references about this topic in the Forum and on GitHub FYI:

On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table.
I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build).

I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails).

So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR.

Regards,

Yann
Kernel & Hypervisor team

TeddyAstie

@JamesG Can you try having a workload on the GPU (and also checking if the performance is lower, or if it's just display) ?

yannsionneau

@yllar I'm not sure, maybe it'll slip to July.
But what I can say is that the ISO is currently being tested in our QA process.
Brace yourself, it's coming soon

Team - Hypervisor Kernel

Posts

Member List