AMD 'Barcelo' passthrough issues - any success stories?
-
@DustyArmstrong
Can you give the result of :lspci -vvv -s 00:08.0(inside Dom0)
Another question, what guest were you trying ?
Can you try with a recent Linux kernel (some changes were made recently regarding video bios requirement) ? Latest Fedora should have a recent enough kernel for testing, that could maybe help workaround the issue in the meantime (and knowing if there are more issues), with no guarantee. -
@TeddyAstie Sure no problem.
lspci -vvv -s 00:08.0 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-Host is Debian, slightly older kernel so that may be why? Happy to try a different VM distro if you think it might help.
Linux cctv 6.12.69+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.69-1 (2026-02-08) x86_64 GNU/Linux -
lspci -vvv -s 00:08.0
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-Ah that's not the one I'm looking for.
Can you do
lspci -vvv(without the-s ...) and take the part related to the GPU ? -
@TeddyAstie yarp.
My bad, the VM has it as
00:08.0but on the host it's actually00:06.0, I just didn't think about the specifics of your request!06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 38 Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M] Region 4: I/O ports at d000 [size=256] Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable- Count=4 Masked- Vector table: BAR=5 offset=00042000 PBA: BAR=5 offset=00043000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [270 v1] #19 Capabilities: [2a0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] Page Request Interface (PRI) PRICtl: Enable- Reset- PRISta: RF- UPRGI- Stopped+ Page Request Capacity: 00000100, Page Request Allocation: 00000000 Capabilities: [2d0 v1] Process Address Space ID (PASID) PASIDCap: Exec+ Priv+, Max PASID Width: 10 PASIDCtl: Enable- Exec- Priv- Capabilities: [400 v1] #25 Capabilities: [410 v1] #26 Capabilities: [440 v1] #27 Kernel driver in use: pciback -
@TeddyAstie yarp.
My bad, the VM has it as
00:08.0but on the host it's actually00:06.0, I just didn't think about the specifics of your request!06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barcelo (rev c1) (prog-if 00 [VGA controller]) Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 1636 Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Interrupt: pin A routed to IRQ 38 Region 0: Memory at d0000000 (64-bit, prefetchable) [size=256M] Region 2: Memory at e0000000 (64-bit, prefetchable) [size=2M] Region 4: I/O ports at d000 [size=256] Region 5: Memory at fca00000 (32-bit, non-prefetchable) [size=512K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- UncorrErr+ FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR-, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+ EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/4 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [c0] MSI-X: Enable- Count=4 Masked- Vector table: BAR=5 offset=00042000 PBA: BAR=5 offset=00043000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [270 v1] #19 Capabilities: [2a0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [2b0 v1] Address Translation Service (ATS) ATSCap: Invalidate Queue Depth: 00 ATSCtl: Enable-, Smallest Translation Unit: 00 Capabilities: [2c0 v1] Page Request Interface (PRI) PRICtl: Enable- Reset- PRISta: RF- UPRGI- Stopped+ Page Request Capacity: 00000100, Page Request Allocation: 00000000 Capabilities: [2d0 v1] Process Address Space ID (PASID) PASIDCap: Exec+ Priv+, Max PASID Width: 10 PASIDCtl: Enable- Exec- Priv- Capabilities: [400 v1] #25 Capabilities: [410 v1] #26 Capabilities: [440 v1] #27 Kernel driver in use: pcibackthanks.
So basically, there is a more annoying issue, as the device doesn't even have a ROMBAR, in this case, the VBIOS is likely in the VFCT ACPI table of host (which the guest can't see); which needs to be injected as a "fake" rombar for the guest to behave properly.
That doable on its own, but it's quite tricky to integrate (and you would e.g need to extract VBIOS from VFCT using external tools).
I just discussed with Xen/AMD people, and there are known issues regarding PCI Passthrough of integrated AMD GPUs (not specific to Xen AFAIU). There are some projects regarding alternative approaches to bring AMD GPUs to VMs (virtio-gpu native context) which is the current focus.
-
@TeddyAstie Thanks for the update.
I do actually have a VBIOS for that GPU, but I wasn't entirely sure what to do with it - is there a process to inject it? I've found resources for Proxmox and others, but I really like XCP and equally I don't want to migrate my entire setup just for that.
If it's really tricky then I'm not too worried about it, as I say the VM actually runs the cameras perfectly fine on CPU alone, it's negligible.
Edit: Looks like this post might answer my question: https://github.com/xcp-ng/xcp/issues/786
"Even when specifying romfile and rombar properties on the xen-pci-passthrough device in QEMU, the ROM region is not mapped into guest memory."
-
Guys, try this https://xcp-ng.org/forum/post/103216
A deep dive is on the linked medium article in that post.
You're right in your findings about the ROMBAR.
It is not a huge problem that is not passed through. What annoys me is that the support for "fake" ROMBAR (config options rombar and romfile) is present in the xen_pci_passthrough QEMU device, but it doesn't work
I found a workaround by using the “loader” QEMU device. This device simply places your vBIOS in the right memory region where the amdgpu driver can find it. It is a quite nice, clean solution, and I hope my pull request for required changes in qemu-wrapper will be integrated soon. In my opinion, it doesn't hurt anything, but I am still waiting.
-
@timemaster5 Ah very cool, thank you for looking into this and the comprehensive write up, great work. I went on a small journey to try various workarounds but didn't get as far as yourself and others, particularly as my use-case is only a "nice-to-have". Most everything I found was in relation to Proxmox and others. Little did I know when I thought I was helping you over on Github, you would actually end up helping me!
As I understand from your notes, the full procedure would be:
- Enable passthrough for the GPU on the host
- Apply the patched wrapper to enable custom model args and respect platform vga
- Disable emulated VGA via
platform:vga=none - Apply the config to the VM via
-device loaderoptions - Reboot the host
- Mount the GPU to the VM and boot it up
-
Thanks for responding to the GitHub issue. It’s great that more people want this working; it’s difficult to gain traction otherwise.
Regarding your list, it’s correct. A reboot should be on the second place. You need to reboot only to detach your PCI device (video card) from its driver and assign it to the pciback driver instead on the next boot. This effectively creates a reservation for the device and allows you to dynamically assign it to VMs.
Once your card is free from other kernel drivers, the rest doesn’t require a reboot.
-
Can you retry with an up-to-date xcp-ng 8.3 please?
FYI on recent XCP-ng 8.3 versions the pci-passthrough will enable the ROM expansion bar. The guest VM will have access to it, so no need to pass it via qemu anymore.
See my comment on GitHub: https://github.com/xcp-ng/xcp/issues/786#issuecomment-4281846490Regards,
Yann
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login