AMD 'Barcelo' passthrough issues - any success stories?

TeddyAstie

@DustyArmstrong said:

lspci -vvv -s 00:08.0

00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

Ah that's not the one I'm looking for.

Can you do lspci -vvv (without the -s ...) and take the part related to the GPU ?

DustyArmstrong

@TeddyAstie yarp.

My bad, the VM has it as 00:08.0 but on the host it's actually 00:06.0, I just didn't think about the specifics of your request!

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at d000 [size=256]
	Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] MSI-X: Enable- Count=4 Masked-
		Vector table: BAR=5 offset=00042000
		PBA: BAR=5 offset=00043000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270 v1] #19
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000100, Page Request Allocation: 00000000
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [400 v1] #25
	Capabilities: [410 v1] #26
	Capabilities: [440 v1] #27
	Kernel driver in use: pciback

TeddyAstie

@DustyArmstrong said:

@TeddyAstie yarp.

My bad, the VM has it as 00:08.0 but on the host it's actually 00:06.0, I just didn't think about the specifics of your request!

06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller])
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 38
	Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M]
	Region 4: I/O ports at d000 [size=256]
	Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [c0] MSI-X: Enable- Count=4 Masked-
		Vector table: BAR=5 offset=00042000
		PBA: BAR=5 offset=00043000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270 v1] #19
	Capabilities: [2a0 v1] Access Control Services
		ACSCap:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
		ACSCtl:	SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
	Capabilities: [2b0 v1] Address Translation Service (ATS)
		ATSCap:	Invalidate Queue Depth: 00
		ATSCtl:	Enable-, Smallest Translation Unit: 00
	Capabilities: [2c0 v1] Page Request Interface (PRI)
		PRICtl: Enable- Reset-
		PRISta: RF- UPRGI- Stopped+
		Page Request Capacity: 00000100, Page Request Allocation: 00000000
	Capabilities: [2d0 v1] Process Address Space ID (PASID)
		PASIDCap: Exec+ Priv+, Max PASID Width: 10
		PASIDCtl: Enable- Exec- Priv-
	Capabilities: [400 v1] #25
	Capabilities: [410 v1] #26
	Capabilities: [440 v1] #27
	Kernel driver in use: pciback

thanks.

So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly.

That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools).

I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.

DustyArmstrong

@TeddyAstie Thanks for the update.

I do actually have a VBIOS for that GPU, but I wasn't entirely sure what to do with it - is there a process to inject it? I've found resources for Proxmox and others, but I really like XCP and equally I don't want to migrate my entire setup just for that.

If it's really tricky then I'm not too worried about it, as I say the VM actually runs the cameras perfectly fine on CPU alone, it's negligible.

Edit: Looks like this post might answer my question: https://github.com/xcp-ng/xcp/issues/786

"Even when specifying romfile and rombar properties on the xen-pci-passthrough device in QEMU, the ROM region is not mapped into guest memory."

timemaster5 created this issue in xcp-ng/xcp

open PCI ROM BAR not exposed to guest when using xen-pci-passthrough #786

timemaster5

@DustyArmstrong @teddyastie

Guys, try this https://xcp-ng.org/forum/post/103216

A deep dive is on the linked medium article in that post.

You're right in your findings about the ROMBAR.

It is not a huge problem that is not passed through. What annoys me is that the support for "fake" ROMBAR (config options rombar and romfile) is present in the xen_pci_passthrough QEMU device, but it doesn't work

I found a workaround by using the “loader” QEMU device. This device simply places your vBIOS in the right memory region where the amdgpu driver can find it. It is a quite nice, clean solution, and I hope my pull request for required changes in qemu-wrapper will be integrated soon. In my opinion, it doesn't hurt anything, but I am still waiting.

DustyArmstrong

@timemaster5 Ah very cool, thank you for looking into this and the comprehensive write up, great work. I went on a small journey to try various workarounds but didn't get as far as yourself and others, particularly as my use-case is only a "nice-to-have". Most everything I found was in relation to Proxmox and others. Little did I know when I thought I was helping you over on Github, you would actually end up helping me!

As I understand from your notes, the full procedure would be:

Enable passthrough for the GPU on the host
Apply the patched wrapper to enable custom model args and respect platform vga
Disable emulated VGA via platform:vga=none
Apply the config to the VM via -device loader options
Reboot the host
Mount the GPU to the VM and boot it up

timemaster5

@DustyArmstrong

Thanks for responding to the GitHub issue. It’s great that more people want this working; it’s difficult to gain traction otherwise.

Regarding your list, it’s correct. A reboot should be on the second place. You need to reboot only to detach your PCI device (video card) from its driver and assign it to the pciback driver instead on the next boot. This effectively creates a reservation for the device and allows you to dynamically assign it to VMs.

Once your card is free from other kernel drivers, the rest doesn’t require a reboot.

yannsionneau

Can you retry with an up-to-date xcp-ng 8.3 please?

FYI on recent XCP-ng 8.3 versions the pci-passthrough will enable the ROM expansion bar. The guest VM will have access to it, so no need to pass it via qemu anymore.
See my comment on GitHub: https://github.com/xcp-ng/xcp/issues/786#issuecomment-4281846490

Regards,

Yann

timemaster5 created this issue in xcp-ng/xcp

open PCI ROM BAR not exposed to guest when using xen-pci-passthrough #786

timemaster5

@yannsionneau said:

Can you retry with an up-to-date xcp-ng 8.3 please?

FYI on recent XCP-ng 8.3 versions the pci-passthrough will enable the ROM expansion bar. The guest VM will have access to it, so no need to pass it via qemu anymore.
See my comment on GitHub: https://github.com/xcp-ng/xcp/issues/786#issuecomment-4281846490

Regards,

Yann

I tried, yes. I confirmed as well on github that the patch does what it suppose to do - it exposes ROM BAR, but that alone is not sufficient to get our cards working. More in my other topic..

I believe it will still fix a lot of issues, and it could potentially fix amd gpu passthrough for some card, but not Phoenix, Raphael and this generation of Ryzen iGPUs (not sure specifically about Barcelo)..

But it is a good progress anyway as I wouldn't be able to fix this ever on my own, so I am really glad it is done. Now I will probably get back to the topic and try to patch whatever else is needit and give it to people in a form of rpm package for the time being.. we will see, it is all about time

yannsionneau

@timemaster5 To give you more feedback, I am currently testing with an AMD Ryzen Embedded V2748 with Radeon Graphics host with a Renoir [Radeon Vega Series / Radeon Vega Mobile Series] iGPU.
I'm also talking to XAPI team to coordinate the effort and try to see what we can/want to do about this.
My first effort is to confirm what works/does not work and give all the information they need to the XAPI team so that they can make the decision on what to do.
I'll update you with my findings.

timemaster5

@yannsionneau Thank you very much. Please let me know if there’s anything I can help with.