Xen Guru group | XCP-ng and XO forum

TeddyAstie

@ravenet said:
@teddyastie separate from the above, a standing question whenever you have a minute: is it expected that passthrough GPU MSIs are never delivered at all on 8.3? Both my working rig and PessimistTech's broken one show the amdgpu xen-pirq MSI vector at zero on all CPUs, forever — everything survives on driver-side polling. It works, but it means any driver path that hard-depends on an interrupt has no safety net, and I suspect it's why the amdgpu ring resets in his log keep failing. Curious whether that's a known limitation of the pirq MSI path for passthrough devices or something worth a ticket.

TL;DR This is likely a known issue, and we're currently discussing on a potential fix.

There is a known bug regarding MSI on AMDGPU when using "hvm_pirq" ("use event channels for device interrupts"), see
https://lore.kernel.org/xen-devel/20240110095311.61809-1-roger.pau@citrix.com/
https://github.com/QubesOS/qubes-issues/issues/7971

The easy fix is to disable "HVM PIRQ" feature (which has been made disabled by default in Xen 4.19), but on the XCP-ng side, this is configured by XAPI Project which needs to be updated as well to apply this change (which appears to be a bit tricky for various reasons).

neowutran created this issue in QubesOS/qubes-issues

closed MSI problems with `amdgpu` driver on recent kernels #7971

TeddyAstie

@PessimistTech
That looks a bit odd indeed.

In addition to what proposed @yannsionneau, can you also give the output of :

xl dmesg (in Dom0)
dmesg (in Dom0)
dmesg (in the guest)

So that we can try to pin-point what may be happening.

TeddyAstie

@JamesG said:

So I ordered some mini DP dummy plugs and took a look at this further today.

First thought...Just load Windows on the bare metal and see what the card does.

No problem. Loaded Win10, loaded the latest drivers, PCIe Gen4x8 lanes. Sweet. At least that works.

Put the XCP-ng drive back in, booted up, passed-through the GPU to a Win11 guest, updated the drivers, PCIe Gen1x1.

As I explained briefly in https://xcp-ng.org/forum/post/106479, this is mostly display as the real hardware may be running at a different speed.

Ultimately I need to install a Linux distro with a really current kernel on the bare metal and see how it sees the GPU. That might be a good project for tomorrow.

I think this just comes down to a combination of too new, not well supported hardware with some quirks and somewhat dated code and support in XCP-ng (ReBAR support in guests for example).

I did quick checks and I believe that ReBAR works; at least with UEFI guests (our OVMF build does support resizable bar), and it actually works as suggest https://xcp-ng.org/forum/post/106489 (VRAM BAR is 16 GB); but I didn't do extensive testing in that area.

TeddyAstie

@JamesG Can you try having a workload on the GPU (and also checking if the performance is lower, or if it's just display) ?

TeddyAstie

@JamesG said:

When I ask the system what it thinks the capabilities are, again I get Gen1x1:

cat /sys/bus/pci/devices/0000:00:08.0/current_link_speed
2.5 GT/s PCIe
cat /sys/bus/pci/devices/0000:00:08.0/current_link_width
1
cat /sys/bus/pci/devices/0000:00:08.0/max_link_speed
2.5 GT/s PCIe
cat /sys/bus/pci/devices/0000:00:08.0/max_link_width
1

This information is actually crafted by QEMU, it doesn't represent the actual real device PCI state; you should check this from Dom0 which should be closer to reality.

TeddyAstie

@JamesG

I know I'm a bit out in uncharted territory here. I don't think XCP-ng supports ReBAR via pass-through (I think that's on the road-map for XenServer9 and maybe XCP-ng 9 as well), current gen GPUs all want ReBAR support, Intel GPU's are in a mostly experimental/developmental mode and might be having a murky future anyway...

XCP-ng guest firmware have a very "legacy" initial BAR placement (i.e thus no explicit support for Resizable BAR, everything below 4G, ...), but the guest operating system can still replace the location of BAR itself if the hardware allows it (in particular Linux will try it, but I'm not sure Windows does).

In your case, the GPU VRAM BAR is 16 GB, so it's obviously resized. And according to xe, you should see in dmesg something like BAR2 resized to 16384MiB

TeddyAstie

@Andrew said:
HP DL G8 Intel E5-2673 v2 shows 64 CPUs. The actual CPUs show correctly, the higher ones (that don't exist) show [CPUxx] Unable to fetch temperature (19 - No such device)

Machines with hotpluggable CPUs are a pretty tricky case, the logic tries up to "maximum possible CPU" and fails here because the CPU is not online (No such device error). That doesn't prevent the temperature from getting fetched for CPU that exists.
I can try to add a check to hide this specific error, so it's doesn't create noise for offline CPUs on such machines.

TeddyAstie

@acebmxer said:

@rzr

Just installed updates on host 1. Once host rebooted it took an extra min or two to reconnect to xo, but did finally connect. Applying updates on host 2 now.

Update - host2 no issues. Once reboot complete it connected to xo as expected without delay.

I see these updates include -

xen: Add support for xenpm get-core-temp to query CPU temperature on Intel platforms.

Use xenpm get-core-temp to get the temperature on Intel's CPU, to fallback unsupported coretemp. Doc update being reviewed .

My host are AMD so can verify these. I might be able to deploy a Intel host later tonight. Will this come to AMD later?

AMD rely on a different method to expose the temperature, that don't require this xenpm-based approach. In principle, it should already work with plain sensors (through k10temp), but our driver may not be up to date for recent AMD CPUs.

TeddyAstie

Regarding upstream Linux, it should be addressed with https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=f24df84cbe05e4471c04ac4b921fc0340bbc7752

Although, I have no ETA on when it will land to distros.

Xen Guru

Posts

closed MSI problems with `amdgpu` driver on recent kernels #7971

Member List