XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work

    Scheduled Pinned Locked Moved Hardware
    99 Posts 7 Posters 17.1k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S Offline
      steff22 @steff22
      last edited by steff22

      @steff22 I really think there is a more serious problem with the new amd dGpu. since then the whole xcp-ng hangs and restarts itself.

      But when windows installs drivers itself it expands to the screen connected to the gpu but the image becomes a rainbow image. But no error messages.

      But when I install drivers from amd the whole win wm hangs and restarts right after login. And if I press shut down vm from xen orchestra the whole xcp-ng hangs and restarts.

      20:19 xcp-ng-AsRoCk ~]# tail -f /var/log/kern.log
Nov 26 20:17:10 xcp-ng-AsRoCk kernel: [  905.801329] device tap3.0 entered promiscuous mode
Nov 26 20:17:44 xcp-ng-AsRoCk kernel: [  939.567508] device tap3.0 left promiscuous mode
Nov 26 20:17:44 xcp-ng-AsRoCk kernel: [  940.403154] device vif3.0 left promiscuous mode
Nov 26 20:17:48 xcp-ng-AsRoCk kernel: [  944.072124] block tda: sector-size: 512/512 capacity: 524288000
Nov 26 20:17:48 xcp-ng-AsRoCk kernel: [  944.368061] block tdb: sector-size: 512/512 capacity: 9558400
Nov 26 20:17:49 xcp-ng-AsRoCk kernel: [  944.699150] device vif4.0 entered promiscuous mode
Nov 26 20:17:49 xcp-ng-AsRoCk kernel: [  944.866671] device tap4.0 entered promiscuous mode
Nov 26 20:18:00 xcp-ng-AsRoCk kernel: [  955.782415] device tap4.0 left promiscuous mode
Nov 26 20:18:03 xcp-ng-AsRoCk kernel: [  959.318049] vif vif-4-0 vif4.0: Guest Rx ready
Nov 26 20:18:47 xcp-ng-AsRoCk kernel: [ 1003.398295] device vif4.0 left promiscuous mode
      1 Reply Last reply Reply Quote 0
      • T Offline
        tuxen Top contributor @steff22
        last edited by tuxen

        @steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:

        03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 32 [Radeon RX 7700 XT / 7800 XT] (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: pciback

        03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: pciback

        Something went wrong with the card detection. Since it's a big file, instead of posting, could you upload (last button) /var/log/kern.log ?

        Searching around some old forums with this error, it's said that if the system has the AMD iGPU + AMD dGPU combo, the iGPU takes precedence and power down the dGPU. Verify in the BIOS (under Advanced > AMD PBS > Graphics or something like that) if there's any configuration regarding this hybrid mode.

        Those last messages you posted are normal (guest net and disk infos).

        Tux

        S 1 Reply Last reply Reply Quote 0
        • S Offline
          steff22 @tuxen
          last edited by

          @tuxen
          I found some settings in the bios that I could disable igpu and make dGpu primary.

          After this change I get more info about the card but same error.

          But I can't run dom0 /var/log/kern.log. I only get message about bash: /var/log/kern.log: Permission denied

          [06:03 xcp-ng-AsRoCk ~]# lspci -k -v
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14d8
	Subsystem: Advanced Micro Devices, Inc. [AMD] Device 14d8
	Flags: fast devsel

00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Device 14d9
	Subsystem: Advanced Micro Devices, Inc. [AMD] Device 14d9
	Flags: bus master, fast devsel, latency 0, IRQ 255
	Capabilities: [40] Secure device <?>
	Capabilities: [64] MSI: Enable- Count=1/4 Maskable- 64bit+
	Capabilities: [74] HyperTransport: MSI Mapping Enable+ Fixed+

00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
	Flags: fast devsel

00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=01, subordinate=03, sec-latency=0
	I/O behind bridge: 0000f000-0000ffff
	Memory behind bridge: fbc00000-fbefffff
	Prefetchable memory behind bridge: 000000f800000000-000000fc0fffffff
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Port (Slot+), MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
	Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [370] L1 PM Substates
	Capabilities: [3c4] #23
	Capabilities: [400] #25
	Capabilities: [410] #26
	Capabilities: [440] #27
	Capabilities: [500] #2a
	Kernel driver in use: pcieport

00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	Memory behind bridge: fc100000-fc1fffff
	Prefetchable memory behind bridge: 000000fc11000000-000000fc130fffff
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Port (Slot+), MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
	Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [370] L1 PM Substates
	Capabilities: [3c4] #23
	Capabilities: [400] #25
	Capabilities: [410] #26
	Capabilities: [440] #27
	Capabilities: [500] #2a
	Kernel driver in use: pcieport

00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
	Flags: fast devsel

00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=05, subordinate=0f, sec-latency=0
	I/O behind bridge: 0000c000-0000efff
	Memory behind bridge: fa000000-fb6fffff
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Port (Slot+), MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
	Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [370] L1 PM Substates
	Capabilities: [3c4] #23
	Capabilities: [400] #25
	Capabilities: [410] #26
	Capabilities: [440] #27
	Capabilities: [500] #2a
	Kernel driver in use: pcieport

00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14db (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=10, subordinate=10, sec-latency=0
	Memory behind bridge: fc000000-fc0fffff
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Port (Slot+), MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1453
	Capabilities: [c8] HyperTransport: MSI Mapping Enable+ Fixed+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [370] L1 PM Substates
	Capabilities: [3c4] #23
	Capabilities: [400] #25
	Capabilities: [410] #26
	Capabilities: [440] #27
	Capabilities: [500] #2a
	Kernel driver in use: pcieport

00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
	Flags: fast devsel

00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
	Flags: fast devsel

00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14da
	Flags: fast devsel

00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14dd (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 26
	Bus: primary=00, secondary=11, subordinate=11, sec-latency=0
	Memory behind bridge: fb800000-fbbfffff
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Port (Slot-), MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 14dd
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [400] #25
	Capabilities: [410] #26
	Capabilities: [450] #27
	Kernel driver in use: pcieport

00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 14dd (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 26
	Bus: primary=00, secondary=12, subordinate=12, sec-latency=0
	Memory behind bridge: fbf00000-fbffffff
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Root Port (Slot-), MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD] Device 14dd
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [400] #25
	Capabilities: [410] #26
	Capabilities: [450] #27
	Kernel driver in use: pcieport

00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 71)
	Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
	Flags: 66MHz, medium devsel
	Kernel driver in use: piix4_smbus
	Kernel modules: i2c_piix4

00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
	Subsystem: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
	Flags: bus master, 66MHz, medium devsel, latency 0

00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e0
	Flags: fast devsel

00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e1
	Flags: fast devsel

00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e2
	Flags: fast devsel

00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e3
	Flags: fast devsel

00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e4
	Flags: fast devsel

00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e5
	Flags: fast devsel

00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e6
	Flags: fast devsel

00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 14e7
	Flags: fast devsel

01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev 11) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 24
	Memory at fbe00000 (32-bit, non-prefetchable) [size=16K]
	Bus: primary=01, secondary=02, subordinate=03, sec-latency=0
	I/O behind bridge: 0000f000-0000ffff
	Memory behind bridge: fbc00000-fbdfffff
	Prefetchable memory behind bridge: 000000f800000000-000000fc0fffffff
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Upstream Port, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [270] #19
	Capabilities: [320] Latency Tolerance Reporting
	Capabilities: [400] #25
	Capabilities: [410] #26
	Capabilities: [440] #27
	Kernel driver in use: pcieport

02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (rev 11) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 24
	Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 0000f000-0000ffff
	Memory behind bridge: fbc00000-fbdfffff
	Prefetchable memory behind bridge: 000000f800000000-000000fc0fffffff
	Capabilities: [50] Power Management version 3
	Capabilities: [58] Express Downstream Port (Slot-), MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [c0] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [400] #25
	Capabilities: [410] #26
	Capabilities: [450] #27
	Kernel driver in use: pcieport

03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 32 [Radeon RX 7700 XT / 7800 XT] (rev c8) (prog-if 00 [VGA controller])
	Subsystem: Gigabyte Technology Co., Ltd Device 2413
	Flags: bus master, fast devsel, latency 0, IRQ 24
	Memory at f800000000 (64-bit, prefetchable) [size=16G]
	Memory at fc00000000 (64-bit, prefetchable) [size=256M]
	I/O ports at f000 [size=256]
	Memory at fbc00000 (32-bit, non-prefetchable) [size=1M]
	Expansion ROM at fbd00000 [disabled] [size=128K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [200] #15
	Capabilities: [240] Power Budgeting <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [2d0] Process Address Space ID (PASID)
	Capabilities: [320] Latency Tolerance Reporting
	Capabilities: [410] #26
	Capabilities: [450] #27
	Kernel driver in use: pciback

03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 31 HDMI/DP Audio
	Flags: bus master, fast devsel, latency 0, IRQ 25
	Memory at fbd20000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150] Advanced Error Reporting
	Capabilities: [2a0] Access Control Services
	Kernel driver in use: pciback

04:00.0 Ethernet controller: Broadcom Inc. and subsidiaries BCM57502 NetXtreme-E 10Gb/25Gb/40Gb/50Gb Ethernet (rev 12)
	Subsystem: ASRock Incorporation Device 1752
	Flags: bus master, fast devsel, latency 0, IRQ 28
	Memory at fc13010000 (64-bit, prefetchable) [size=64K]
	Memory at fc12000000 (64-bit, prefetchable) [size=16M]
	Memory at fc13028000 (64-bit, prefetchable) [size=32K]
	Expansion ROM at fc180000 [disabled] [size=512K]
	Capabilities: [48] Power Management version 3
	Capabilities: [50] Vital Product Data
	Capabilities: [a0] MSI-X: Enable+ Count=74 Masked-
	Capabilities: [ac] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [13c] Device Serial Number 9c-6b-00-ff-fe-65-38-6b
	Capabilities: [150] Power Budgeting <?>
	Capabilities: [160] Virtual Channel
	Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?>
	Capabilities: [1b0] Latency Tolerance Reporting
	Capabilities: [1b8] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [230] Transaction Processing Hints
	Capabilities: [300] #19
	Capabilities: [200] Precision Time Measurement
	Capabilities: [358] #26
	Capabilities: [388] #27
	Kernel driver in use: bnxt_en
	Kernel modules: bnxt_en

04:00.1 Ethernet controller: Broadcom Inc. and subsidiaries BCM57502 NetXtreme-E 10Gb/25Gb/40Gb/50Gb Ethernet (rev 12)
	Subsystem: ASRock Incorporation Device 1752
	Flags: bus master, fast devsel, latency 0, IRQ 29
	Memory at fc13000000 (64-bit, prefetchable) [size=64K]
	Memory at fc11000000 (64-bit, prefetchable) [size=16M]
	Memory at fc13020000 (64-bit, prefetchable) [size=32K]
	Expansion ROM at fc100000 [disabled] [size=512K]
	Capabilities: [48] Power Management version 3
	Capabilities: [50] Vital Product Data
	Capabilities: [a0] MSI-X: Enable+ Count=74 Masked-
	Capabilities: [ac] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [13c] Device Serial Number 9c-6b-00-ff-fe-65-38-6b
	Capabilities: [150] Power Budgeting <?>
	Capabilities: [180] Vendor Specific Information: ID=0000 Rev=0 Len=020 <?>
	Capabilities: [1b8] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [230] Transaction Processing Hints
	Kernel driver in use: bnxt_en
	Kernel modules: bnxt_en

05:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Bus: primary=05, secondary=06, subordinate=0f, sec-latency=0
	I/O behind bridge: 0000c000-0000efff
	Memory behind bridge: fa000000-fb6fffff
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Upstream Port, MSI 00
	Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 3328
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [160] Latency Tolerance Reporting
	Capabilities: [1c0] #19
	Capabilities: [200] L1 PM Substates
	Capabilities: [220] #25
	Capabilities: [240] #26
	Capabilities: [280] #27
	Kernel driver in use: pcieport

06:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Bus: primary=06, secondary=07, subordinate=07, sec-latency=0
	Memory behind bridge: fb600000-fb6fffff
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Downstream Port (Slot+), MSI 00
	Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 3328
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1c0] #19
	Capabilities: [200] L1 PM Substates
	Capabilities: [220] #25
	Capabilities: [240] #26
	Capabilities: [280] #27
	Kernel driver in use: pcieport

06:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 47
	Bus: primary=06, secondary=08, subordinate=08, sec-latency=0
	I/O behind bridge: 0000e000-0000efff
	Memory behind bridge: fb500000-fb5fffff
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Downstream Port (Slot+), MSI 00
	Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 3328
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1c0] #19
	Capabilities: [200] L1 PM Substates
	Capabilities: [220] #25
	Capabilities: [240] #26
	Capabilities: [280] #27
	Kernel driver in use: pcieport

06:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 44
	Bus: primary=06, secondary=09, subordinate=09, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff
	Memory behind bridge: fb400000-fb4fffff
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Downstream Port (Slot+), MSI 00
	Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 3328
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1c0] #19
	Capabilities: [200] L1 PM Substates
	Capabilities: [220] #25
	Capabilities: [240] #26
	Capabilities: [280] #27
	Kernel driver in use: pcieport

06:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 45
	Bus: primary=06, secondary=0a, subordinate=0b, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: fa000000-fb0fffff
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Downstream Port (Slot+), MSI 00
	Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 3328
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1c0] #19
	Capabilities: [200] L1 PM Substates
	Capabilities: [220] #25
	Capabilities: [240] #26
	Capabilities: [280] #27
	Kernel driver in use: pcieport

06:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Bus: primary=06, secondary=0c, subordinate=0c, sec-latency=0
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Downstream Port (Slot+), MSI 00
	Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 3328
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1c0] #19
	Capabilities: [200] L1 PM Substates
	Capabilities: [220] #25
	Capabilities: [240] #26
	Capabilities: [280] #27
	Kernel driver in use: pcieport

06:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Bus: primary=06, secondary=0d, subordinate=0d, sec-latency=0
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Downstream Port (Slot+), MSI 00
	Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 3328
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [1c0] #19
	Capabilities: [200] L1 PM Substates
	Capabilities: [220] #25
	Capabilities: [240] #26
	Capabilities: [280] #27
	Kernel driver in use: pcieport

06:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Bus: primary=06, secondary=0e, subordinate=0e, sec-latency=0
	Memory behind bridge: fb300000-fb3fffff
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Downstream Port (Slot-), MSI 00
	Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 3328
	Capabilities: [100] Advanced Error Reporting
	Kernel driver in use: pcieport

06:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 47
	Bus: primary=06, secondary=0f, subordinate=0f, sec-latency=0
	Memory behind bridge: fb200000-fb2fffff
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Downstream Port (Slot-), MSI 00
	Capabilities: [c0] Subsystem: ASMedia Technology Inc. Device 3328
	Capabilities: [100] Advanced Error Reporting
	Kernel driver in use: pcieport

07:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])
	Subsystem: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Memory at fb600000 (64-bit, non-prefetchable) [size=8K]
	Capabilities: [50] Power Management version 3
	Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [90] MSI-X: Enable- Count=8 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [150] Latency Tolerance Reporting
	Kernel driver in use: pciback
	Kernel modules: xhci_pci

08:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
	Subsystem: ASRock Incorporation Device 1533
	Flags: bus master, fast devsel, latency 0, IRQ 47
	Memory at fb500000 (32-bit, non-prefetchable) [size=512K]
	I/O ports at e000 [size=32]
	Memory at fb580000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 9c-6b-00-ff-ff-65-37-bf
	Capabilities: [1a0] Transaction Processing Hints
	Kernel driver in use: igb
	Kernel modules: igb

09:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
	Subsystem: ASRock Incorporation Device 1533
	Flags: bus master, fast devsel, latency 0, IRQ 44
	Memory at fb400000 (32-bit, non-prefetchable) [size=512K]
	I/O ports at d000 [size=32]
	Memory at fb480000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
	Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [140] Device Serial Number 9c-6b-00-ff-ff-65-38-0f
	Capabilities: [1a0] Transaction Processing Hints
	Kernel driver in use: igb
	Kernel modules: igb

0a:00.0 PCI bridge: ASRock Incorporation Device 1150 (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 255
	Bus: primary=0a, secondary=0b, subordinate=0b, sec-latency=32
	I/O behind bridge: 0000c000-0000cfff
	Memory behind bridge: fa000000-fb0fffff
	Capabilities: [50] MSI: Enable- Count=1/32 Maskable- 64bit+
	Capabilities: [78] Power Management version 3
	Capabilities: [80] Express PCI-Express to PCI/PCI-X Bridge, MSI 00
	Capabilities: [c0] Subsystem: ASRock Incorporation Device 1150
	Capabilities: [100] Virtual Channel
	Capabilities: [200] #23
	Capabilities: [2c0] #23
	Capabilities: [800] Advanced Error Reporting

0b:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 52) (prog-if 00 [VGA controller])
	Subsystem: ASRock Incorporation Device 2000
	Flags: medium devsel, IRQ 255
	Memory at fa000000 (32-bit, non-prefetchable) [size=16M]
	Memory at fb000000 (32-bit, non-prefetchable) [size=256K]
	I/O ports at c000 [size=128]
	[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: [40] Power Management version 3
	Capabilities: [50] MSI: Enable- Count=1/4 Maskable- 64bit+

0e:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset USB 3.2 Controller (rev 01) (prog-if 30 [XHCI])
	Subsystem: ASMedia Technology Inc. Device 1142
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Memory at fb300000 (64-bit, non-prefetchable) [size=32K]
	Capabilities: [50] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [68] MSI-X: Enable+ Count=8 Masked-
	Capabilities: [78] Power Management version 3
	Capabilities: [80] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [160] Latency Tolerance Reporting
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

0f:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller (rev 01) (prog-if 01 [AHCI 1.0])
	Subsystem: ASMedia Technology Inc. Device 1062
	Flags: bus master, fast devsel, latency 0, IRQ 118
	Memory at fb280000 (32-bit, non-prefetchable) [size=1K]
	Expansion ROM at fb200000 [disabled] [size=512K]
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit+
	Capabilities: [70] Power Management version 3
	Capabilities: [80] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Kernel driver in use: ahci
	Kernel modules: ahci

10:00.0 Non-Volatile memory controller: Kingston Technology Company, Inc. FURY Renegade NVMe SSD with heatsink (rev 01) (prog-if 02 [NVM Express])
	Subsystem: Kingston Technology Company, Inc. FURY Renegade NVMe SSD with heatsink
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Memory at fc000000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [80] Express Endpoint, MSI 00
	Capabilities: [d0] MSI-X: Enable+ Count=33 Masked-
	Capabilities: [e0] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [f8] Power Management version 3
	Capabilities: [100] Latency Tolerance Reporting
	Capabilities: [110] L1 PM Substates
	Capabilities: [128] Alternative Routing-ID Interpretation (ARI)
	Capabilities: [1e0] #25
	Capabilities: [200] Advanced Error Reporting
	Capabilities: [290] Device Serial Number 00-00-00-00-00-00-00-00
	Capabilities: [2a0] Power Budgeting <?>
	Capabilities: [300] #19
	Capabilities: [340] #26
	Capabilities: [378] #27
	Kernel driver in use: nvme
	Kernel modules: nvme

11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Phoenix PCIe Dummy Function (rev c1)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 13c0
	Flags: fast devsel
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Legacy Endpoint, MSI 00
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [410] #26
	Capabilities: [450] #27

11:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP
	Subsystem: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP
	Flags: fast devsel, IRQ 255
	Memory at fba00000 (32-bit, non-prefetchable) [disabled] [size=1M]
	Memory at fbb48000 (32-bit, non-prefetchable) [disabled] [size=8K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/2 Maskable- 64bit+
	Capabilities: [c0] MSI-X: Enable- Count=2 Masked-
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [2a0] Access Control Services

11:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b6 (prog-if 30 [XHCI])
	Subsystem: Advanced Micro Devices, Inc. [AMD] Device 15b6
	Flags: bus master, fast devsel, latency 0, IRQ 41
	Memory at fb900000 (64-bit, non-prefetchable) [size=1M]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [2a0] Access Control Services
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

11:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b7 (prog-if 30 [XHCI])
	Subsystem: Advanced Micro Devices, Inc. [AMD] Device 15b6
	Flags: bus master, fast devsel, latency 0, IRQ 42
	Memory at fb800000 (64-bit, non-prefetchable) [size=1M]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [2a0] Access Control Services
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

11:00.5 Multimedia controller: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor (rev 62)
	Subsystem: Advanced Micro Devices, Inc. [AMD] ACP/ACP3X/ACP6x Audio Coprocessor
	Flags: fast devsel, IRQ 255
	Memory at fbb00000 (32-bit, non-prefetchable) [disabled] [size=256K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [2a0] Access Control Services

11:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller
	Subsystem: Advanced Micro Devices, Inc. [AMD] Device d601
	Flags: fast devsel, IRQ 255
	Memory at fbb40000 (32-bit, non-prefetchable) [disabled] [size=32K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [2a0] Access Control Services

12:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8 (prog-if 30 [XHCI])
	Subsystem: Advanced Micro Devices, Inc. [AMD] Device 15b6
	Flags: bus master, fast devsel, latency 0, IRQ 46
	Memory at fbf00000 (64-bit, non-prefetchable) [size=1M]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
	Capabilities: [64] Express Endpoint, MSI 00
	Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
	Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [270] #19
	Capabilities: [2a0] Access Control Services
	Capabilities: [410] #26
	Capabilities: [450] #27
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci

[06:06 xcp-ng-AsRoCk ~]#
          T 1 Reply Last reply Reply Quote 0
          • T Offline
            tuxen Top contributor @steff22
            last edited by

            @steff22 I'm seeing that this new card has large BAR. Could you disable in the BIOS:

            • Disable Re-Size BAR;
            • DIsable Above 4G Decoding;

            Tux

            S 2 Replies Last reply Reply Quote 0
            • S Offline
              steff22 @tuxen
              last edited by

              @tuxen Tried to Disable Re-Size BAR /Above 4G Decoding.

              Vm hangs less then but is still only rainbow screen.

              can't get it to work on vmware esxi either. just gets the same rainbow screen.

              Not sure if that's true but found on level1 forum that others also couldn't get gpu passthrough with rx7800xt and they thought it was something with RDNA3

              Ubuntu 24.4 manages to show the correct image but hangs and can't log in

              1 Reply Last reply Reply Quote 0
              • S Offline
                steff22 @tuxen
                last edited by

                @tuxen

                Is there any possibility to achieve something the same with changing the power pci state in xcp-ng as on proxmox.

                Found this on a forum

                ((
                Splitting IOMMU groups did help but didn't solve the freeze it just reduced how often it occurs and added an output just one time before freezing

                can't change power state from D3cold to D0
                so i kept digging and found this in a post in proxmox forum:

                pcie_port_pm=off

                adding this to my grub config finally got the freeze out of the way big_smile
                this is not a total win tho, my gpu fully passes to the kvm but looks like if the kvm couln't use it at full power but limit itself around 25% of the gpu usage. My guesses now are:

                Is the GPU locked in D3cold state? How to verify/control the state))

                T 1 Reply Last reply Reply Quote 0
                • T Offline
                  tuxen Top contributor @steff22
                  last edited by tuxen

                  @steff22

                  /opt/xensource/libexec/xen-cmdline --set-dom0 "pcie_port_pm=off"
reboot

                  As an another option, you can also test the pcie_aspm=off that we did before for the Nvidia GPU (edit: added the command to remove the pcie_port_pm):

                  /opt/xensource/libexec/xen-cmdline --delete-dom0 "pcie_port_pm"
/opt/xensource/libexec/xen-cmdline --set-dom0 "pcie_aspm=off"
reboot

                  Tux

                  S 3 Replies Last reply Reply Quote 0
                  • S Offline
                    steff22 @tuxen
                    last edited by

                    @tuxen

                    Unfortunately, it didn't work better, maybe a little less xcp-ng host crash with the VM running. But always when the VM shuts down.

                    This is the last entry in kern.log before Xcp-ng restarts

                    Nov 28 12:34:47 xcp-ng-AsRoCk kernel: [ 3313.679697] device vif2.0 left promiscuous mode
Nov 28 12:34:53 xcp-ng-AsRoCk kernel: [ 3319.117908] bnxt_en 0000:04:00.0 eth0: Recovering missed irq [0]: raw_cons: 8a4
Nov 28 12:34:59 xcp-ng-AsRoCk kernel: [ 3325.081449] bnxt_en 0000:04:00.0 eth0: Recovering missed irq [0]: raw_cons: 8a4
Nov 28 12:34:59 xcp-ng-AsRoCk kernel: [ 3325.081451] bnxt_en 0000:04:00.0 eth0: Recovering missed irq [1]: raw_cons: 4a9b

                    There is a red timer that counts up in the middle of the rainbow screen. Does this mean that Xcp-ng has not completely lost control of the dgpu?
                    or is this normal before the VM sends an image to the screen?

                    Is it possible to get live xl dmesg messages like with tail -f /var/log/kern.log so that I see the last entry before Xcp-ng reboots?

                    1 Reply Last reply Reply Quote 0
                    • S Offline
                      steff22 @tuxen
                      last edited by

                      @tuxen Unfortunately I think this is amd reset bug on Radeon rx7800XT too. I have no idea if it can be fixed or if it's just to return the gpu to the store again. And rather find an nvidia card.

                      also found that amd gpu can automatically activate AMD vDisplay by gpu passthrough

                      Is there a list of gpu that others have managed to get to work with gpu gpu passthrough with Xcp-ng?

                      R 1 Reply Last reply Reply Quote 0
                      • S Offline
                        steff22 @tuxen
                        last edited by

                        @tuxen

                        Confirmed that vDisplay is enabled.

                        Windows finds the screen and extends the desktop to the available screen without connecting any physical screen.

                        Then tried with parsec and has access to hardware in the GPU to work playing steam games

                        1 Reply Last reply Reply Quote 0
                        • R Offline
                          ravenet @steff22
                          last edited by

                          @steff22

                          I have a Radeon Pro W7500 working fine on Win11 vm, on xcp 8.3. Windows update even installed the pro series driver for me.
                          Asrock rack x470d4u, Ryzen 1700x.
                          I know it's older platform and agesa though. I may have access to a ryzen 7700x system for some testing next week.

                          12b5bea4-5c61-4677-abb1-7353ad2005c5-image.png

                          S 1 Reply Last reply Reply Quote 1
                          • S Offline
                            steff22 @ravenet
                            last edited by

                            @ravenet Ok I think the pro gpu's work better without so many bugs.

                            I switched to Nvidia RTX 4070. With the Nvidia gpu and this works as it should and gets an image on the screen on the first try.

                            So there must be something wrong with the amd drivers plus that error with the amd reset bug

                            R 1 Reply Last reply Reply Quote 1
                            • R Offline
                              ravenet @steff22
                              last edited by

                              @steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:

                              @ravenet Ok I think the pro gpu's work better without so many bugs.

                              I switched to Nvidia RTX 4070. With the Nvidia gpu and this works as it should and gets an image on the screen on the first try.

                              So there must be something wrong with the amd drivers plus that error with the amd reset bug

                              I've setup a test system with a ryzen 7700x and a regular radeon 7600xt, non-pro gpu. Will let you know results. Shouldn't be an issue, it's same hardware and even the drivers work across them. Though they do have different bios on cards

                              1 Reply Last reply Reply Quote 0
                              • First post
                                Last post