XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    RDNA 4 GPU Passthrough

    Scheduled Pinned Locked Moved Compute
    13 Posts 4 Posters 339 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Y Offline
      yannsionneau Vates 🪐 XCP-ng Team @PessimistTech
      last edited by

      @PessimistTech Let's try to see if we have errors in the logs.

      Run ls /boot/xen* and note the name of the xen file that ends with -d.gz

      Then, can you reboot and , while in the grub menu, edit the XCP-ng menu entry (the first one) and make the following modifications:

      • modify the Xen ELF loaded by multiboot2 to load the debug Xen build by specifying the file that ends with -d.gz instead of /boot/xen.gz
      • on the same line, at the end, add iommu=debug

      Then boot (by doing ctrl+X) and once booted and the VM with PCI passthrough started, please issue xl dmesg command in dom0 shell and report the content here.

      Also please paste the result of dom0 dmesg.

      then also an lspci -vvv -nn -s SBDF from both dom0 and the VM

      replace SBDF with the segment bus device function address of the device (beware it won't be the same in dom0 and inside the VM).

      Regards,

      Yann

      P 1 Reply Last reply Reply Quote 0
      • TeddyAstieT Offline
        TeddyAstie Vates 🪐 XCP-ng Team Xen Guru @PessimistTech
        last edited by TeddyAstie

        @PessimistTech
        That looks a bit odd indeed.

        In addition to what proposed @yannsionneau, can you also give the output of :

        • xl dmesg (in Dom0)
        • dmesg (in Dom0)
        • dmesg (in the guest)

        So that we can try to pin-point what may be happening.

        1 Reply Last reply Reply Quote 0
        • P Online
          PessimistTech @yannsionneau
          last edited by

          @yannsionneau Thank you!

          1 Reply Last reply Reply Quote 0
          • P Online
            PessimistTech @yannsionneau
            last edited by PessimistTech

            @yannsionneau said:

            segment bus device function address of the device

            Hi all, thanks for your reponses!

            the below is done with the assumption that dom0 is just the host cli. If that is incorrect please let me know and I can update this info:

            dom0: lspci -vvv -nn -s 03:00.0:
            03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
            Subsystem: ASRock Incorporation Device [1849:5413]
            Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
            Latency: 0, Cache Line Size: 64 bytes
            Interrupt: pin A routed to IRQ 120
            Region 0: Memory at 7f000000000 (64-bit, prefetchable) [size=32G]
            Region 2: Memory at 7f800000000 (64-bit, prefetchable) [size=256M]
            Region 4: I/O ports at 3000 [size=256]
            Region 5: Memory at f2200000 (32-bit, non-prefetchable) [size=512K]
            Expansion ROM at f2280000 [disabled] [size=128K]
            Capabilities: [48] Vendor Specific Information: Len=08 <?>
            Capabilities: [50] Power Management version 3
            Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+)
            Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
            Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
            DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
            DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
            DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
            LnkCap: Port #0, Speed unknown, Width x16, ASPM L1, Exit Latency L0s unlimited, L1 <1us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
            LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
            LnkSta: Speed unknown, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
            DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported
            DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
            LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-
            Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
            Compliance De-emphasis: -6dB
            LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
            EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
            Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
            Address: 0000000000000000 Data: 0000
            Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
            Capabilities: [150 v2] Advanced Error Reporting
            UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
            UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
            UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol-
            CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
            CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
            AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
            Capabilities: [200 v1] #15
            Capabilities: [240 v1] Power Budgeting <?>
            Capabilities: [270 v1] #19
            Capabilities: [2a0 v1] Access Control Services
            ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
            ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
            Capabilities: [2d0 v1] Process Address Space ID (PASID)
            PASIDCap: Exec+ Priv+, Max PASID Width: 10
            PASIDCtl: Enable- Exec- Priv-
            Capabilities: [320 v1] Latency Tolerance Reporting
            Max snoop latency: 0ns
            Max no snoop latency: 0ns
            Capabilities: [410 v1] #26
            Capabilities: [450 v1] #27
            Capabilities: [500 v1] #2a
            Kernel driver in use: pciback

            VM lspci -vvv -nn -s 03:00.0:
            00:08.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller])
            Subsystem: ASRock Incorporation Device [1849:5413]
            Physical Slot: 8
            Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+
            Latency: 0
            Interrupt: pin A routed to IRQ 17
            Region 0: Memory at 800000000 (64-bit, prefetchable) [size=32G]
            Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M]
            Region 4: I/O ports at c300 [size=256]
            Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K]
            Expansion ROM at 000c0000 [disabled] [size=128K]
            Capabilities: [48] Vendor Specific Information: Len=08 <?>
            Capabilities: [50] Power Management version 3
            Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
            Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
            Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00
            DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
            DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 128 bytes, MaxReadReq 512 bytes
            DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
            LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us
            ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
            LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
            LnkSta: Speed 32GT/s, Width x16
            TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
            DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
            10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1
            EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
            FRS-
            AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
            DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
            AtomicOpsCtl: ReqEn+
            LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
            EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
            Retimer- 2Retimers- CrosslinkRes: unsupported
            Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
            Address: 0000000000000000 Data: 0000
            Kernel driver in use: amdgpu
            Kernel modules: amdgpu

            dom0 xl dmesg output: xldmesg.txt
            dom0 dmesg output: dom0dmesg.txt
            vm dmesg output: vmdmesg.txt

            I believe this is all the output requested (I only provided the lspci output for a single GPU as they are identical GPUs , but happy to provide all the outputs if desired (will note I am not entirely sure I selected the exact GPU in both contexts due to the mismatched IDs in vm vs dom0, not sure that is an issue but may be worth calling out)

            edit: all output was taken with the VM running (but at idle). I did apply the grub changes to use the -d.gz file and iommu=debug as well so ideally that output should be included)

            1 Reply Last reply Reply Quote 0
            • R Online
              ravenet @PessimistTech
              last edited by

              @PessimistTech

              I am running a radeon ai Pro 9700 on xcp-ng 8.3 passed through to an opensuse Slowroll, with rocm and lemonade docker. Gemma 4 31B-it-MTP gets 38tps

              did you pass through as pcie passthrough with both the video and audio, or just the 'gpu passthrough' option in xoa? I did the 2 devices pcie passthrough
              Make sure you have Resizable BAR enabled in bios or it just will not work

              R

              P 1 Reply Last reply Reply Quote 0
              • P Online
                PessimistTech @ravenet
                last edited by

                @ravenet glad to hear you were able to get it working!
                Yes I passed through both the audio device and video device via PCI passthrough. I have also checked and ensured Resizeable BAR is enabled in bios.

                Did you do anything to set it up directly on the host? Or did you just use the PCI device passthrough options in XO?

                1 Reply Last reply Reply Quote 0
                • P Online
                  PessimistTech
                  last edited by PessimistTech

                  Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue.

                  EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...

                  R 1 Reply Last reply Reply Quote 0
                  • R Online
                    ravenet @PessimistTech
                    last edited by ravenet

                    @PessimistTech said:

                    Well thanks everyone for the responses! After some more trial and error I got it working. I dug through the bios on my board and enabled ARI Support, PCIe AER, and ACS. After reboot the VM was able to run workloads without issue.

                    EDIT: at least partially working. Some workloads seem fine, other workloads seem to behave the same as before...

                    Do you have examples of workloads that are struggling still?
                    and no, didn't do anything special on the xcp host that I'm aware of. Was already using it to passthrough a radeon pro 7500 to a windows vm for revit clarity automation server. It's been a year so I'll triple check if anything special was done back then.

                    P 1 Reply Last reply Reply Quote 0
                    • P Online
                      PessimistTech @ravenet
                      last edited by PessimistTech

                      @ravenet sure, the workloads I have been testing with include ollama, vLLM, and lemonade server (all via docker for easier setup, but I do have all the rocm/AMD drivers and what not setup on the VM directly). Ollama works just fine now, but lemonade server still hangs and vLLM fails with the same errors on startup.

                      Cool, thanks for info/checking!

                      R 1 Reply Last reply Reply Quote 0
                      • R Online
                        ravenet @PessimistTech
                        last edited by

                        @PessimistTech make sure lemonade is set to use rocm as backend. It was slow and unusable from vulkan

                        docker compose exec lemonade lemonade config set llamacpp.backend=rocm

                        does it finish loading model then just respond horribly slowly, or does it not complete the model load?
                        did you setup as straight docker, or as a docker compose? I did mine as a compose, but shouldn't matter outside of how the config is laid out

                        is your guest seeing the whole mem on gpu and the PCIE link speed?

                        sudo lspci -vv -d 1002: | grep -E "Region [0-9]|LnkSta:"
        Region 0: Memory at 1000000000 (64-bit, prefetchable) [size=32G]
        Region 2: Memory at 1800000000 (64-bit, prefetchable) [size=256M]
        Region 4: I/O ports at c100 [size=256]
        Region 5: Memory at f0800000 (32-bit, non-prefetchable) [size=512K]
                LnkSta: Speed 32GT/s, Width x16
        Region 0: Memory at f08c0000 (32-bit, non-prefetchable) [size=16K]
                LnkSta: Speed 32GT/s, Width x16]

                        You'll also want to make sure AtomicOP is working, below is example command and result

                        sudo lspci -vv -d 1002: | grep -iE "AtomicOp"

                         AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
                         AtomicOpsCtl: ReqEn-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                         AtomicOpsCtl: ReqEn-
                        P 1 Reply Last reply Reply Quote 0
                        • P Online
                          PessimistTech @ravenet
                          last edited by

                          @ravenet I have not tried with the rocm backend yet, but I will give that a whirl. When using the GPUs it never completes the model load.

                          For testing I was just using the docker run command from the lemonade server docs.

                          docker run -d \
	--name lemonade-server \
	-p 13305:13305 \
	-v lemonade-cache:/root/.cache/huggingface \
	-v lemonade-llama:/opt/lemonade/llama \
	-v lemonade-recipe:/root/.cache/lemonade \
	--device=/dev/kfd \
	--device=/dev/dri \
	ghcr.io/lemonade-sdk/lemonade-server:latest

                          Command output:

                          lspci -vv -s 00:08.0 | grep -E "Region [0-9]|LnkSta:"
	Region 0: Memory at 1000000000 (64-bit, prefetchable) [size=32G]
	Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 4: I/O ports at c300 [size=256]
	Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K]
		LnkSta:	Speed 32GT/s, Width x16

                          lspci -vv -s 00:08.0 | grep -iE AtomicOP
			 AtomicOpsCap: 32bit+ 64bit+ 128bitCAS-
			 AtomicOpsCtl: ReqEn+
                          1 Reply Last reply Reply Quote 0

                          Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                          Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                          With your input, this post could be even better 💗

                          Register Login
                          • First post
                            Last post