@JamesG Can you try having a workload on the GPU (and also checking if the performance is lower, or if it's just display) ?

@JamesG said:

When I ask the system what it thinks the capabilities are, again I get Gen1x1:

cat /sys/bus/pci/devices/0000:00:08.0/current_link_speed
2.5 GT/s PCIe
cat /sys/bus/pci/devices/0000:00:08.0/current_link_width
1
cat /sys/bus/pci/devices/0000:00:08.0/max_link_speed
2.5 GT/s PCIe
cat /sys/bus/pci/devices/0000:00:08.0/max_link_width
1

This information is actually crafted by QEMU, it doesn't represent the actual real device PCI state; you should check this from Dom0 which should be closer to reality.

@JamesG

I know I'm a bit out in uncharted territory here. I don't think XCP-ng supports ReBAR via pass-through (I think that's on the road-map for XenServer9 and maybe XCP-ng 9 as well), current gen GPUs all want ReBAR support, Intel GPU's are in a mostly experimental/developmental mode and might be having a murky future anyway...

XCP-ng guest firmware have a very "legacy" initial BAR placement (i.e thus no explicit support for Resizable BAR, everything below 4G, ...), but the guest operating system can still replace the location of BAR itself if the hardware allows it (in particular Linux will try it, but I'm not sure Windows does).

In your case, the GPU VRAM BAR is 16 GB, so it's obviously resized. And according to xe, you should see in dmesg something like BAR2 resized to 16384MiB

@Andrew said:
HP DL G8 Intel E5-2673 v2 shows 64 CPUs. The actual CPUs show correctly, the higher ones (that don't exist) show [CPUxx] Unable to fetch temperature (19 - No such device)

Machines with hotpluggable CPUs are a pretty tricky case, the logic tries up to "maximum possible CPU" and fails here because the CPU is not online (No such device error). That doesn't prevent the temperature from getting fetched for CPU that exists.
I can try to add a check to hide this specific error, so it's doesn't create noise for offline CPUs on such machines.

@acebmxer said:

@rzr

Just installed updates on host 1. Once host rebooted it took an extra min or two to reconnect to xo, but did finally connect. Applying updates on host 2 now.

Update - host2 no issues. Once reboot complete it connected to xo as expected without delay.

I see these updates include -

• xen: Add support for xenpm get-core-temp to query CPU temperature on Intel platforms.

• Use xenpm get-core-temp to get the temperature on Intel's CPU, to fallback unsupported coretemp. Doc update being reviewed .

My host are AMD so can verify these. I might be able to deploy a Intel host later tonight. Will this come to AMD later?

AMD rely on a different method to expose the temperature, that don't require this xenpm-based approach. In principle, it should already work with plain sensors (through k10temp), but our driver may not be up to date for recent AMD CPUs.

Regarding upstream Linux, it should be addressed with https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=f24df84cbe05e4471c04ac4b921fc0340bbc7752

Although, I have no ETA on when it will land to distros.

@TrapoSAMA https://updates.xcp-ng.org/isos/8.2/
And https://updates.xcp-ng.org/8/8.2/ appears fine.

The rule is oddly written, and may conflict with another similar one that already exist in the distro (hence may not be useful to begin with).

The modern generic rule for doing vCPU hotplug is, which would be preferable to the current z10-xen-vcpu-hotplug.rules.
ACTION=="add", SUBSYSTEM=="cpu", ATTR{online}=="0", ATTR{online}="1"

@MajorP93 said:
Do you think it is possible to fix this on hypervisor level while still having live migration etc. enabled or do we have to wait for an upstream fix within Linux kernel tree?

Yes it's possible to fix it on the hypervisor level (Invariant TSC in guest), but it's quite a bit of work that still needs to be done. A Linux upstream fix for the underlying bug should come at some point hopefully.

@majorp93 @henri9813 @acebmxer
Do you observe the same behavior after setting this for the VM ?

xe vm-param-add uuid=$UUID param-name=platform tsc_mode=2
xe vm-param-add uuid=$UUID param-name=platform nomigrate=true

(beware you lose live migration support doing this, you can cancel these changes with matching vm-param-remove like xe vm-param-remove uuid=$UUID param-name=platform param-key=nomigrate)

@MajorP93 can you give the kernel version of all the affected vs non-affected guests ?

@acebmxer I don't observe the same issue on Debian 13 Cloud-Init (both 6.12.38+deb13-amd64 and updated 6.12.90+deb13.1-amd64).

Though it still takes some time to boot (especially at loading the ramdisk) but it's not related to this pv spinlock issue and mostly a "BIOS guest" related issue.
But I'm testing on a Intel machine.

@acebmxer which kernel version you have in your Debian guest (uname -a) ?

Can reproduce on Fedora 44 and Alpine Linux (6.18.22-0-virt).
But doesn't occur on Debian 13 (6.12).

i2c error seems related to (I guess) a RGB controller that is presumably controlled by something else (or maybe not usable/non plugged). But it's harmless unless you are looking to configure RGB (but I don't think you want to do that anyway).

EFI_MEMMAP warning is probably because Dom0 doesn't see UEFI mappings, as it relies on a different method to do UEFI calls. This is expected and this warning doesn't indicate a problem.

The latest error is likely related to the first one.

what I will miss /won't be able to do vs a user that has not these messages? (otherwise, why they would be raised)

Nothing meaningful.

if these messages /failures have an impact on the time during which the machine is booting? (it seems the load process is hanging for about a minute)

No.

if these messages /failures are officially documented somewhere?

For the EFI_MEMMAP, XenServer has a article on it stating the same as me : https://support.citrix.com/external/article/CTX331542/citrix-hypervisor-82-efi-efimemmap-is-n.html

how can I resolve these failures, because failures are failures... even if "one should not worry about them"?

You may be able to hide it, like by blacklisting i2c and such, but TBF it's not worth the time.

Do you have multiple guests agents in the VM (e.g Citrix and XCP-ng ones) that may step on each other for this IP ?

Xen Project covered this as XSA-489.

@sluflyer06 said:

@olivierlambert I'm willing to go procure another to try to troubleshoot, 2 of us tried 3 different cards on different servers and spent a lot of time and never got anywhere with the cards. This is some messages I still have have from back when I was trying. More or less, XCPNG could see the cards, they could be passed to a VM either in XO or CLI, we tried a couple different linux VM's on 6.17 kernel and would get errors like below, I also tried a windows VM and it couldn't identify the card at all, tried latest intel drivers, etc.

[    3.109115] xe 0000:00:08.0: [drm] Found battlemage (device ID e212) discrete display version 14.01 stepping B0
[    3.110277] xe 0000:00:08.0: Direct firmware load for xe/bmg_guc_70.bin failed with error -2
[    3.110281] xe 0000:00:08.0: [drm] GuC firmware xe/bmg_guc_70.bin: fetch failed with error -2
[    3.110283] xe 0000:00:08.0: [drm] GuC firmware(s) can be downloaded from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
[    3.110284] xe 0000:00:08.0: [drm] *ERROR* GT0: GuC init failed with -ENOENT
[    3.110299] xe 0000:00:08.0: [drm] *ERROR* GT0: Failed to initialize uC (-ENOENT)
[    3.110526] xe 0000:00:08.0: probe with driver xe failed with error -2

The driver is telling you that you are missing the Intel Xe firmwares (that are probably in linux-firmware package).

@tsukraw Can you give the output of xen-cpuid -p (in Dom0) ?

I think you're looking for a software like com0com (https://com0com.sourceforge.net/, see ReactOS for modern Windows versions https://reactos.org/wiki/Com0com), which has a com2tcp.