XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    RDNA 4 GPU Passthrough

    Scheduled Pinned Locked Moved Compute
    6 Posts 3 Posters 189 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Online
      PessimistTech
      last edited by

      Hi all,
      I am trying to pass through 2 AMD Radeon Ai Pro 9700 GPUs to a ubuntu VM. I have gottent to the point where I have the GPUs passed through, I can see them fine in the VM (Ie rocm-info/rocm-smi/nvtop all show the GPUs and current status without isssue), and I can even start workloads against them and see load start on the GPUs, but the workloads always time out or fail as they don't seem to get a response back or something along those lines.

      In my digging so far I have seen that no interrupts are being registered, and as far as I can tell this seems to be an issue with the XCP-NG -> VM layer in the repsponse (as I cannot fine any issues, errors or misconfigs on the VM other than the timeouts).

      Has anyone tried/had success with newer RDNA 4 GPUs? Any suggestions on things to try next? I'm at a bit of a loss here...

      Y TeddyAstieT 3 Replies Last reply Reply Quote 0
      • Y Offline
        yannsionneau Vates 🪐 XCP-ng Team @PessimistTech
        last edited by

        Hi @pessimisttech let me ping @Team-Hypervisor-Kernel here 🙂

        P 1 Reply Last reply Reply Quote 0
        • Y Offline
          yannsionneau Vates 🪐 XCP-ng Team @PessimistTech
          last edited by

          @PessimistTech Let's try to see if we have errors in the logs.

          Run ls /boot/xen* and note the name of the xen file that ends with -d.gz

          Then, can you reboot and , while in the grub menu, edit the XCP-ng menu entry (the first one) and make the following modifications:

          • modify the Xen ELF loaded by multiboot2 to load the debug Xen build by specifying the file that ends with -d.gz instead of /boot/xen.gz
          • on the same line, at the end, add iommu=debug

          Then boot (by doing ctrl+X) and once booted and the VM with PCI passthrough started, please issue xl dmesg command in dom0 shell and report the content here.

          Also please paste the result of dom0 dmesg.

          then also an lspci -vvv -nn -s SBDF from both dom0 and the VM

          replace SBDF with the segment bus device function address of the device (beware it won't be the same in dom0 and inside the VM).

          Regards,

          Yann

          P 1 Reply Last reply Reply Quote 0
          • TeddyAstieT Offline
            TeddyAstie Vates 🪐 XCP-ng Team Xen Guru @PessimistTech
            last edited by TeddyAstie

            @PessimistTech
            That looks a bit odd indeed.

            In addition to what proposed @yannsionneau, can you also give the output of :

            • xl dmesg (in Dom0)
            • dmesg (in Dom0)
            • dmesg (in the guest)

            So that we can try to pin-point what may be happening.

            1 Reply Last reply Reply Quote 0
            • P Online
              PessimistTech @yannsionneau
              last edited by

              @yannsionneau Thank you!

              1 Reply Last reply Reply Quote 0
              • P Online
                PessimistTech @yannsionneau
                last edited by PessimistTech

                @yannsionneau said:

                segment bus device function address of the device

                Hi all, thanks for your reponses!

                the below is done with the assumption that dom0 is just the host cli. If that is incorrect please let me know and I can update this info:

                dom0: lspci -vvv -nn -s 03:00.0:
                03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
                Subsystem: ASRock Incorporation Device [1849:5413]
                Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
                Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
                Latency: 0, Cache Line Size: 64 bytes
                Interrupt: pin A routed to IRQ 120
                Region 0: Memory at 7f000000000 (64-bit, prefetchable) [size=32G]
                Region 2: Memory at 7f800000000 (64-bit, prefetchable) [size=256M]
                Region 4: I/O ports at 3000 [size=256]
                Region 5: Memory at f2200000 (32-bit, non-prefetchable) [size=512K]
                Expansion ROM at f2280000 [disabled] [size=128K]
                Capabilities: [48] Vendor Specific Information: Len=08 <?>
                Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
                Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
                DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
                RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed unknown, Width x16, ASPM L1, Exit Latency L0s unlimited, L1 <1us
                ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed unknown, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-
                Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
                Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000 Data: 0000
                Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
                Capabilities: [150 v2] Advanced Error Reporting
                UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
                CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
                CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
                Capabilities: [200 v1] #15
                Capabilities: [240 v1] Power Budgeting <?>
                Capabilities: [270 v1] #19
                Capabilities: [2a0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                Capabilities: [2d0 v1] Process Address Space ID (PASID)
                PASIDCap: Exec+ Priv+, Max PASID Width: 10
                PASIDCtl: Enable- Exec- Priv-
                Capabilities: [320 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
                Capabilities: [410 v1] #26
                Capabilities: [450 v1] #27
                Capabilities: [500 v1] #2a
                Kernel driver in use: pciback

                VM lspci -vvv -nn -s 03:00.0:
                00:08.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
                Subsystem: ASRock Incorporation Device [1849:5413]
                Physical Slot: 8
                Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
                Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
                Latency: 0
                Interrupt: pin A routed to IRQ 17
                Region 0: Memory at 800000000 (64-bit, prefetchable) [size=32G]
                Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M]
                Region 4: I/O ports at c300 [size=256]
                Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K]
                Expansion ROM at 000c0000 [disabled] [size=128K]
                Capabilities: [48] Vendor Specific Information: Len=08 <?>
                Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
                Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
                ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
                ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 32GT/s, Width x16
                TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
                EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                FRS-
                AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                AtomicOpsCtl: ReqEn+
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                Retimer- 2Retimers- CrosslinkRes: unsupported
                Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000 Data: 0000
                Kernel driver in use: amdgpu
                Kernel modules: amdgpu

                dom0 xl dmesg output: xldmesg.txt
                dom0 dmesg output: dom0dmesg.txt
                vm dmesg output: vmdmesg.txt

                I believe this is all the output requested (I only provided the lspci output for a single GPU as they are identical GPUs , but happy to provide all the outputs if desired (will note I am not entirely sure I selected the exact GPU in both contexts due to the mismatched IDs in vm vs dom0, not sure that is an issue but may be worth calling out)

                edit: all output was taken with the VM running (but at idle). I did apply the grub changes to use the -d.gz file and iommu=debug as well so ideally that output should be included)

                1 Reply Last reply Reply Quote 0

                Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                With your input, this post could be even better 💗

                Register Login
                • First post
                  Last post