Xen Guru group | XCP-ng and XO forum

dinhngtu

@MajorP93 AFAIK @teddyastie is working on a 1.0 release for Linux guests with this bug fix.

dinhngtu

@MajorP93 Hi, are you using the Rust guest agent? We suspect that it's failing to rearm the ballooning feature after resuming from suspend.

dinhngtu

@MajorP93 Thanks for the info, I'll need to discuss with the team first

dinhngtu

@MajorP93 Hello, what does xe vm-param-get uuid=... param-name=memory-target say on your VM after migrating?

dinhngtu

@acebmxer process 1775 (unattended-upgr<ade>) is your hint. Your VM was presumably running an unattended upgrade while you were trying to install the tools.

TeddyAstie

@DustyArmstrong said:

@TeddyAstie yarp.

My bad, the VM has it as 00:08.0 but on the host it's actually 00:06.0, I just didn't think about the specifics of your request!

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at d000 [size=256]
	Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] MSI-X: Enable- Count=4 Masked-
		Vector table: BAR=5 offset=00042000
		PBA: BAR=5 offset=00043000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270 v1] #19
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000100, Page Request Allocation: 00000000
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [400 v1] #25
	Capabilities: [410 v1] #26
	Capabilities: [440 v1] #27
	Kernel driver in use: pciback

thanks.

So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly.

That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools).

I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.

TeddyAstie

@DustyArmstrong said:

lspci -vvv -s 00:08.0

00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Ah that's not the one I'm looking for.

Can you do lspci -vvv (without the -s ...) and take the part related to the GPU ?

TeddyAstie

@DustyArmstrong
Can you give the result of :

lspci -vvv -s 00:08.0

(inside Dom0)

Another question, what guest were you trying ?
Can you try with a recent Linux kernel (some changes were made recently regarding video bios requirement) ? Latest Fedora should have a recent enough kernel for testing, that could maybe help workaround the issue in the meantime (and knowing if there are more issues), with no guarantee.

TeddyAstie

@benapetr said in New project - XenAdminQt - a cross-platform GNU/Linux, macOS, Windows native thick client:

@Pilow I know them a little bit, I will have a look, but I am now working on another new cool thing! It's called xen_exporter: https://github.com/benapetr/xen_exporter

It's a prometheus exporter that hooks directly to xen kernel via xenctrl library from dom0 and extract all low-level metrics from the host, allowing very detailed graphs with very low granularity with stuff I always missed in both XenOrchestra and XenAdmin:

...

We have a similar project : https://github.com/xcp-ng/xcp-metrics, but unfortunately it's not used as of today (though it could get revived as Rust for Xen matures, i.e easier to build).
There is also Xen Orchestra OpenMetrics support but it's not on XCP-ng itself.

TeddyAstie

@DustyArmstrong

EDIT: It looks like I may just have a fake BIOS? The settings to enable all the relevant components (IOMMU, DMAr support etc) don't actually seem to do anything, they might just be for show - dmesg | grep -i iommu returns nothing, dmesg | grep -i -e dmar -e vfio -e pciback only shows pciback info, and cat /proc/cmdline contains nothing about IOMMU. Oddly, XO is still reporting that IOMMU is enabled:

dmesg in the Dom0 will not report the information you're looking for.
To know if PCI Passthrough is supported (e.g IOMMU enabled), you should check xl info | grep virt_caps and look for hvm_directio. You can also look for IOMMU-related stuff in xl dmesg.
As you managed to passthrough the device (even if not working in the guest), I don't see a issue there.

[ 4.655776] amdgpu 0000:00:08.0: amdgpu: Unable to locate a BIOS ROM
[ 4.655797] amdgpu 0000:00:08.0: amdgpu: Fatal error during GPU init
[ 4.655812] amdgpu 0000:00:08.0: amdgpu: amdgpu: finishing device.
[ 4.656681] amdgpu 0000:00:08.0: probe with driver amdgpu failed with error -22

Is there a trick to this, has anyone had success with this kind of AMD GPU? On my old hosts, enabling pass through was enough for it to just kind of work (Intel HD 530). The host machine outputs to a display normally when the card is in-use by the host. I am of the understanding the ROM is just part of the motherboard/GPU, there is some suggestion it can be dumped from the host-side, but I'm unsure on this.

Looks like the GPU ROMBAR is missing in the guest, while it's ok for many devices, many others will fail to work without it (like this GPU).
To me, there's something missing on the PCI Passthrough logic, I just brought the topic internally to see what we can do.

Xen Guru

Posts

Member List