Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work

tuxen

@steff22 I'm seeing that this new card has large BAR. Could you disable in the BIOS:

Disable Re-Size BAR;
DIsable Above 4G Decoding;

Tux

steff22

@tuxen Tried to Disable Re-Size BAR /Above 4G Decoding.

Vm hangs less then but is still only rainbow screen.

can't get it to work on vmware esxi either. just gets the same rainbow screen.

Not sure if that's true but found on level1 forum that others also couldn't get gpu passthrough with rx7800xt and they thought it was something with RDNA3

Ubuntu 24.4 manages to show the correct image but hangs and can't log in

steff22

@tuxen

Is there any possibility to achieve something the same with changing the power pci state in xcp-ng as on proxmox.

Found this on a forum

((
Splitting IOMMU groups did help but didn't solve the freeze it just reduced how often it occurs and added an output just one time before freezing

can't change power state from D3cold to D0
so i kept digging and found this in a post in proxmox forum:

pcie_port_pm=off

adding this to my grub config finally got the freeze out of the way big_smile
this is not a total win tho, my gpu fully passes to the kvm but looks like if the kvm couln't use it at full power but limit itself around 25% of the gpu usage. My guesses now are:

Is the GPU locked in D3cold state? How to verify/control the state))

tuxen

@steff22

/opt/xensource/libexec/xen-cmdline --set-dom0 "pcie_port_pm=off"
reboot

As an another option, you can also test the pcie_aspm=off that we did before for the Nvidia GPU (edit: added the command to remove the pcie_port_pm):

/opt/xensource/libexec/xen-cmdline --delete-dom0 "pcie_port_pm"
/opt/xensource/libexec/xen-cmdline --set-dom0 "pcie_aspm=off"
reboot

Tux

steff22

@tuxen

Unfortunately, it didn't work better, maybe a little less xcp-ng host crash with the VM running. But always when the VM shuts down.

This is the last entry in kern.log before Xcp-ng restarts

Nov 28 12:34:47 xcp-ng-AsRoCk kernel: [ 3313.679697] device vif2.0 left promiscuous mode
Nov 28 12:34:53 xcp-ng-AsRoCk kernel: [ 3319.117908] bnxt_en 0000:04:00.0 eth0: Recovering missed irq [0]: raw_cons: 8a4
Nov 28 12:34:59 xcp-ng-AsRoCk kernel: [ 3325.081449] bnxt_en 0000:04:00.0 eth0: Recovering missed irq [0]: raw_cons: 8a4
Nov 28 12:34:59 xcp-ng-AsRoCk kernel: [ 3325.081451] bnxt_en 0000:04:00.0 eth0: Recovering missed irq [1]: raw_cons: 4a9b

There is a red timer that counts up in the middle of the rainbow screen. Does this mean that Xcp-ng has not completely lost control of the dgpu?
or is this normal before the VM sends an image to the screen?

Is it possible to get live xl dmesg messages like with tail -f /var/log/kern.log so that I see the last entry before Xcp-ng reboots?

steff22

@tuxen Unfortunately I think this is amd reset bug on Radeon rx7800XT too. I have no idea if it can be fixed or if it's just to return the gpu to the store again. And rather find an nvidia card.

also found that amd gpu can automatically activate AMD vDisplay by gpu passthrough

Is there a list of gpu that others have managed to get to work with gpu gpu passthrough with Xcp-ng?

steff22

@tuxen

Confirmed that vDisplay is enabled.

Windows finds the screen and extends the desktop to the available screen without connecting any physical screen.

Then tried with parsec and has access to hardware in the GPU to work playing steam games

ravenet

@steff22

I have a Radeon Pro W7500 working fine on Win11 vm, on xcp 8.3. Windows update even installed the pro series driver for me.
Asrock rack x470d4u, Ryzen 1700x.
I know it's older platform and agesa though. I may have access to a ryzen 7700x system for some testing next week.

steff22

@ravenet Ok I think the pro gpu's work better without so many bugs.

I switched to Nvidia RTX 4070. With the Nvidia gpu and this works as it should and gets an image on the screen on the first try.

So there must be something wrong with the amd drivers plus that error with the amd reset bug

ravenet

@steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:

@ravenet Ok I think the pro gpu's work better without so many bugs.

I switched to Nvidia RTX 4070. With the Nvidia gpu and this works as it should and gets an image on the screen on the first try.

So there must be something wrong with the amd drivers plus that error with the amd reset bug

I've setup a test system with a ryzen 7700x and a regular radeon 7600xt, non-pro gpu. Will let you know results. Shouldn't be an issue, it's same hardware and even the drivers work across them. Though they do have different bios on cards