Do you have multiple guests agents in the VM (e.g Citrix and XCP-ng ones) that may step on each other for this IP ?

Xen Project covered this as XSA-489.

@sluflyer06 said:

@olivierlambert I'm willing to go procure another to try to troubleshoot, 2 of us tried 3 different cards on different servers and spent a lot of time and never got anywhere with the cards. This is some messages I still have have from back when I was trying. More or less, XCPNG could see the cards, they could be passed to a VM either in XO or CLI, we tried a couple different linux VM's on 6.17 kernel and would get errors like below, I also tried a windows VM and it couldn't identify the card at all, tried latest intel drivers, etc.

[    3.109115] xe 0000:00:08.0: [drm] Found battlemage (device ID e212) discrete display version 14.01 stepping B0
[    3.110277] xe 0000:00:08.0: Direct firmware load for xe/bmg_guc_70.bin failed with error -2
[    3.110281] xe 0000:00:08.0: [drm] GuC firmware xe/bmg_guc_70.bin: fetch failed with error -2
[    3.110283] xe 0000:00:08.0: [drm] GuC firmware(s) can be downloaded from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
[    3.110284] xe 0000:00:08.0: [drm] *ERROR* GT0: GuC init failed with -ENOENT
[    3.110299] xe 0000:00:08.0: [drm] *ERROR* GT0: Failed to initialize uC (-ENOENT)
[    3.110526] xe 0000:00:08.0: probe with driver xe failed with error -2

The driver is telling you that you are missing the Intel Xe firmwares (that are probably in linux-firmware package).

@tsukraw Can you give the output of xen-cpuid -p (in Dom0) ?

I think you're looking for a software like com0com (https://com0com.sourceforge.net/, see ReactOS for modern Windows versions https://reactos.org/wiki/Com0com), which has a com2tcp.

@DustyArmstrong said:

@TeddyAstie yarp.

My bad, the VM has it as 00:08.0 but on the host it's actually 00:06.0, I just didn't think about the specifics of your request!

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at d000 [size=256]
	Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] MSI-X: Enable- Count=4 Masked-
		Vector table: BAR=5 offset=00042000
		PBA: BAR=5 offset=00043000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270 v1] #19
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000100, Page Request Allocation: 00000000
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [400 v1] #25
	Capabilities: [410 v1] #26
	Capabilities: [440 v1] #27
	Kernel driver in use: pciback

thanks.

So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly.

That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools).

I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.

@DustyArmstrong said:

lspci -vvv -s 00:08.0

00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Ah that's not the one I'm looking for.

Can you do lspci -vvv (without the -s ...) and take the part related to the GPU ?

@DustyArmstrong
Can you give the result of :

lspci -vvv -s 00:08.0

(inside Dom0)

Another question, what guest were you trying ?
Can you try with a recent Linux kernel (some changes were made recently regarding video bios requirement) ? Latest Fedora should have a recent enough kernel for testing, that could maybe help workaround the issue in the meantime (and knowing if there are more issues), with no guarantee.

@benapetr said in New project - XenAdminQt - a cross-platform GNU/Linux, macOS, Windows native thick client:

@Pilow I know them a little bit, I will have a look, but I am now working on another new cool thing! It's called xen_exporter: https://github.com/benapetr/xen_exporter

It's a prometheus exporter that hooks directly to xen kernel via xenctrl library from dom0 and extract all low-level metrics from the host, allowing very detailed graphs with very low granularity with stuff I always missed in both XenOrchestra and XenAdmin:

...

We have a similar project : https://github.com/xcp-ng/xcp-metrics, but unfortunately it's not used as of today (though it could get revived as Rust for Xen matures, i.e easier to build).
There is also Xen Orchestra OpenMetrics support but it's not on XCP-ng itself.

@DustyArmstrong

EDIT: It looks like I may just have a fake BIOS? The settings to enable all the relevant components (IOMMU, DMAr support etc) don't actually seem to do anything, they might just be for show - dmesg | grep -i iommu returns nothing, dmesg | grep -i -e dmar -e vfio -e pciback only shows pciback info, and cat /proc/cmdline contains nothing about IOMMU. Oddly, XO is still reporting that IOMMU is enabled:

dmesg in the Dom0 will not report the information you're looking for.
To know if PCI Passthrough is supported (e.g IOMMU enabled), you should check xl info | grep virt_caps and look for hvm_directio. You can also look for IOMMU-related stuff in xl dmesg.
As you managed to passthrough the device (even if not working in the guest), I don't see a issue there.

[ 4.655776] amdgpu 0000:00:08.0: amdgpu: Unable to locate a BIOS ROM
[ 4.655797] amdgpu 0000:00:08.0: amdgpu: Fatal error during GPU init
[ 4.655812] amdgpu 0000:00:08.0: amdgpu: amdgpu: finishing device.
[ 4.656681] amdgpu 0000:00:08.0: probe with driver amdgpu failed with error -22

Is there a trick to this, has anyone had success with this kind of AMD GPU? On my old hosts, enabling pass through was enough for it to just kind of work (Intel HD 530). The host machine outputs to a display normally when the card is in-use by the host. I am of the understanding the ROM is just part of the motherboard/GPU, there is some suggestion it can be dumped from the host-side, but I'm unsure on this.

Looks like the GPU ROMBAR is missing in the guest, while it's ok for many devices, many others will fail to work without it (like this GPU).
To me, there's something missing on the PCI Passthrough logic, I just brought the topic internally to see what we can do.

@jbamford
Hello, can you give the output of xl info in Dom0 ? If you see hvm_directio in front of virt_caps, then PCI Passthrough is supported (i.e you have VT-d).

You can also check xl dmesg for vt-d specific messages (you can also give the output of it).

@EddieA Can you try differents combinations of passedthrough hardware in this VM ?

e.g try with each device one by one at a time; at least in the VM

@benapetr
Consider making a PR to https://github.com/xcp-ng/xcp-ng-org (https://github.com/xcp-ng/xcp-ng-org/blob/master/docs/management/management.md) if you want to make your project more visible.

@nvs can you try with a serial console and something listening on it; so when it crashes, we get the crash reason ?

@dcskinner @olivierlambert

You can read key/values from the xenstore, and write some (from VM to outside), but you cannot write values "in live" from outside the VM to the inside.

It is, but XAPI doesn't provide a interface for it.

do the guest tools quiesce the filesystems before snapshotting?

Tools are aware of a snapshot so you don't have blocks in flight.

do the guest tools quiesce the filesystems before snapshotting?

Guests kernel are aware, as it is them that are performing a "suspend" on toolstack request (thus quiece filesystems); although "tools" can only observe that the system has been suspended after the fact by measuring side effects, and not orchestrate it.

It's because suspend/resume operation doesn't come from "guest tools" actually, but instead from the kernel drivers. So userland tools has no say on it.

@antest can you try running the VM with only 1 vCPU ?

@antest can you retry after doing /opt/xensource/libexec/xen-cmdline --set-xen iommu=debug then rebooting.
And also reporting full DomU and Dom0 dmesg (in addition to xl dmesg) ?

Do you get anything special written to xl dmesg in Dom0 ?

@hitechhillbilly no it doesn't, it just ensures the N-th vCPU of Dom0 only runs on N-th pCPU of the machine.
Not sure about the practical impact of it, in the past it has been used for getting meaningful CPU temperatures from coretemp (with physical core matching virtual one), but that doesn't work anymore since Xen filters MSR accesses (including Dom0).

@dlucas said in XCP-NG 8.3 Crash on Shutdown - HP DL360 g9:

Have checked the IML log and the only thing that is being logged is this:

"13","Critical","PCI Bus","12/04/2025 00:03","12/04/2025 00:03","1","PCI Bus Error (Slot 0, Bus 0, Device 2, Function 2)",
"12","Critical","PCI Bus","12/03/2025 23:01","12/03/2025 23:01","1","PCI Bus Error (Slot 0, Bus 0, Device 2, Function 2)",
"11","Critical","PCI Bus","12/03/2025 22:53","12/03/2025 22:53","1","PCI Bus Error (Slot 0, Bus 0, Device 2, Function 2)",
"10","Critical","PCI Bus","12/01/2025 23:44","12/01/2025 23:44","1","PCI Bus Error (Slot 0, Bus 0, Device 2, Function 2)",
"9","Critical","PCI Bus","12/01/2025 01:31","12/01/2025 01:31","1","PCI Bus Error (Slot 0, Bus 0, Device 2, Function 2)",

I thought that a shutdown command from the command prompt solved the issue but it seems to always reboot when ever you tell it to shutdown.

What is device 00:02.2 (through lspci) ?