@MajorP93 said:
Do you think it is possible to fix this on hypervisor level while still having live migration etc. enabled or do we have to wait for an upstream fix within Linux kernel tree?

Yes it's possible to fix it on the hypervisor level (Invariant TSC in guest), but it's quite a bit of work that still needs to be done. A Linux upstream fix for the underlying bug should come at some point hopefully.

@majorp93 @henri9813 @acebmxer
Do you observe the same behavior after setting this for the VM ?

xe vm-param-add uuid=$UUID param-name=platform tsc_mode=2
xe vm-param-add uuid=$UUID param-name=platform nomigrate=true

(beware you lose live migration support doing this, you can cancel these changes with matching vm-param-remove like xe vm-param-remove uuid=$UUID param-name=platform param-key=nomigrate)

@MajorP93 can you give the kernel version of all the affected vs non-affected guests ?

@acebmxer I don't observe the same issue on Debian 13 Cloud-Init (both 6.12.38+deb13-amd64 and updated 6.12.90+deb13.1-amd64).

Though it still takes some time to boot (especially at loading the ramdisk) but it's not related to this pv spinlock issue and mostly a "BIOS guest" related issue.
But I'm testing on a Intel machine.

@acebmxer which kernel version you have in your Debian guest (uname -a) ?

Can reproduce on Fedora 44 and Alpine Linux (6.18.22-0-virt).
But doesn't occur on Debian 13 (6.12).

i2c error seems related to (I guess) a RGB controller that is presumably controlled by something else (or maybe not usable/non plugged). But it's harmless unless you are looking to configure RGB (but I don't think you want to do that anyway).

EFI_MEMMAP warning is probably because Dom0 doesn't see UEFI mappings, as it relies on a different method to do UEFI calls. This is expected and this warning doesn't indicate a problem.

The latest error is likely related to the first one.

what I will miss /won't be able to do vs a user that has not these messages? (otherwise, why they would be raised)

Nothing meaningful.

if these messages /failures have an impact on the time during which the machine is booting? (it seems the load process is hanging for about a minute)

No.

if these messages /failures are officially documented somewhere?

For the EFI_MEMMAP, XenServer has a article on it stating the same as me : https://support.citrix.com/external/article/CTX331542/citrix-hypervisor-82-efi-efimemmap-is-n.html

how can I resolve these failures, because failures are failures... even if "one should not worry about them"?

You may be able to hide it, like by blacklisting i2c and such, but TBF it's not worth the time.

Do you have multiple guests agents in the VM (e.g Citrix and XCP-ng ones) that may step on each other for this IP ?

Xen Project covered this as XSA-489.

@sluflyer06 said:

@olivierlambert I'm willing to go procure another to try to troubleshoot, 2 of us tried 3 different cards on different servers and spent a lot of time and never got anywhere with the cards. This is some messages I still have have from back when I was trying. More or less, XCPNG could see the cards, they could be passed to a VM either in XO or CLI, we tried a couple different linux VM's on 6.17 kernel and would get errors like below, I also tried a windows VM and it couldn't identify the card at all, tried latest intel drivers, etc.

[    3.109115] xe 0000:00:08.0: [drm] Found battlemage (device ID e212) discrete display version 14.01 stepping B0
[    3.110277] xe 0000:00:08.0: Direct firmware load for xe/bmg_guc_70.bin failed with error -2
[    3.110281] xe 0000:00:08.0: [drm] GuC firmware xe/bmg_guc_70.bin: fetch failed with error -2
[    3.110283] xe 0000:00:08.0: [drm] GuC firmware(s) can be downloaded from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
[    3.110284] xe 0000:00:08.0: [drm] *ERROR* GT0: GuC init failed with -ENOENT
[    3.110299] xe 0000:00:08.0: [drm] *ERROR* GT0: Failed to initialize uC (-ENOENT)
[    3.110526] xe 0000:00:08.0: probe with driver xe failed with error -2

The driver is telling you that you are missing the Intel Xe firmwares (that are probably in linux-firmware package).

@tsukraw Can you give the output of xen-cpuid -p (in Dom0) ?

I think you're looking for a software like com0com (https://com0com.sourceforge.net/, see ReactOS for modern Windows versions https://reactos.org/wiki/Com0com), which has a com2tcp.

@DustyArmstrong said:

@TeddyAstie yarp.

My bad, the VM has it as 00:08.0 but on the host it's actually 00:06.0, I just didn't think about the specifics of your request!

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at d000 [size=256]
	Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] MSI-X: Enable- Count=4 Masked-
		Vector table: BAR=5 offset=00042000
		PBA: BAR=5 offset=00043000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270 v1] #19
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000100, Page Request Allocation: 00000000
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [400 v1] #25
	Capabilities: [410 v1] #26
	Capabilities: [440 v1] #27
	Kernel driver in use: pciback

thanks.

So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly.

That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools).

I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.

@DustyArmstrong said:

lspci -vvv -s 00:08.0

00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Ah that's not the one I'm looking for.

Can you do lspci -vvv (without the -s ...) and take the part related to the GPU ?

@DustyArmstrong
Can you give the result of :

lspci -vvv -s 00:08.0

(inside Dom0)

Another question, what guest were you trying ?
Can you try with a recent Linux kernel (some changes were made recently regarding video bios requirement) ? Latest Fedora should have a recent enough kernel for testing, that could maybe help workaround the issue in the meantime (and knowing if there are more issues), with no guarantee.

@benapetr said in New project - XenAdminQt - a cross-platform GNU/Linux, macOS, Windows native thick client:

@Pilow I know them a little bit, I will have a look, but I am now working on another new cool thing! It's called xen_exporter: https://github.com/benapetr/xen_exporter

It's a prometheus exporter that hooks directly to xen kernel via xenctrl library from dom0 and extract all low-level metrics from the host, allowing very detailed graphs with very low granularity with stuff I always missed in both XenOrchestra and XenAdmin:

...

We have a similar project : https://github.com/xcp-ng/xcp-metrics, but unfortunately it's not used as of today (though it could get revived as Rust for Xen matures, i.e easier to build).
There is also Xen Orchestra OpenMetrics support but it's not on XCP-ng itself.

@DustyArmstrong

EDIT: It looks like I may just have a fake BIOS? The settings to enable all the relevant components (IOMMU, DMAr support etc) don't actually seem to do anything, they might just be for show - dmesg | grep -i iommu returns nothing, dmesg | grep -i -e dmar -e vfio -e pciback only shows pciback info, and cat /proc/cmdline contains nothing about IOMMU. Oddly, XO is still reporting that IOMMU is enabled:

dmesg in the Dom0 will not report the information you're looking for.
To know if PCI Passthrough is supported (e.g IOMMU enabled), you should check xl info | grep virt_caps and look for hvm_directio. You can also look for IOMMU-related stuff in xl dmesg.
As you managed to passthrough the device (even if not working in the guest), I don't see a issue there.

[ 4.655776] amdgpu 0000:00:08.0: amdgpu: Unable to locate a BIOS ROM
[ 4.655797] amdgpu 0000:00:08.0: amdgpu: Fatal error during GPU init
[ 4.655812] amdgpu 0000:00:08.0: amdgpu: amdgpu: finishing device.
[ 4.656681] amdgpu 0000:00:08.0: probe with driver amdgpu failed with error -22

Is there a trick to this, has anyone had success with this kind of AMD GPU? On my old hosts, enabling pass through was enough for it to just kind of work (Intel HD 530). The host machine outputs to a display normally when the card is in-use by the host. I am of the understanding the ROM is just part of the motherboard/GPU, there is some suggestion it can be dumped from the host-side, but I'm unsure on this.

Looks like the GPU ROMBAR is missing in the guest, while it's ok for many devices, many others will fail to work without it (like this GPU).
To me, there's something missing on the PCI Passthrough logic, I just brought the topic internally to see what we can do.

@jbamford
Hello, can you give the output of xl info in Dom0 ? If you see hvm_directio in front of virt_caps, then PCI Passthrough is supported (i.e you have VT-d).

You can also check xl dmesg for vt-d specific messages (you can also give the output of it).

@EddieA Can you try differents combinations of passedthrough hardware in this VM ?

e.g try with each device one by one at a time; at least in the VM

@benapetr
Consider making a PR to https://github.com/xcp-ng/xcp-ng-org (https://github.com/xcp-ng/xcp-ng-org/blob/master/docs/management/management.md) if you want to make your project more visible.