Installed fine. Didn't get a chance to deep dive and/or test features.

@stormi
Thanks!

@olivierlambert @TeddyAstie

I am unable to test further since I don't have a 'passable'/'assignable usb controller which can be passed through. Have ordered one and would keep you guys posted once I get it and test it.

Though I can confirm that the bug (dunno if it is the driver / xcp-ng) persists.

@olivierlambert @TeddyAstie
Whelp !
I tried using /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:07:00.0)" but now XCP-ng refuses to boot !
Any way I can reverse the command by booting in safe mode ?

UPDATE :
Manually edited /boot/efi/EFI/xenserver/grub.cfg and removed the entry, now the server boots.

Hoping that /opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:07:00.0)" only alters /boot/efi/EFI/xenserver/grub.cfg and not some internal Xen settings.

A clarification on the above would be highly appreciated!

@TeddyAstie @olivierlambert

Thank you sooo much for your prompt response !

FYI a function-level-reset is systematically performed by Xen when doing PCI passthrough, thus your device should be reset before entering another guest (aside reset bugs like you may have).

This is exactly what the problem is. The Dom0 is unable to perform FLR when I also pass vUSB to the Guest. However, if USB is not passed (i mean if it is not attached; even though USB passthrough is enabled in host), the FLR seems to be performed correctly and I am able to restart the guest without problems.

If the FLR is not performed, the Guest (even if it is the same one being restarted) is unable to detect the pass-through cards and also waits for about 136 seconds (about 65.5 seconds for each card) after which it continues without adding the card. This wait time is the kernel default wait time which cannot be changed without rebuilding the kernel I think.

I am trying to find out where the conflict is. (Basically I am unable to understand if this is a driver problem or a Xen problem; since the FLR is being performed correctly when there is no usb being passed through)
Ideally passing USB should not have in impact on PCI-pass through.

Current XAPI uses the approach of having a set of "passthrough-able" devices at boot time by modifying the xen-pciback.hide kernel parameter, which does the same but at boot time.

Will this still work with XCP-ng 8.3 ?

/opt/xensource/libexec/xen-cmdline --set-dom0 "xen-pciback.hide=(0000:07:00.0)"

I am unable to use xe pci-disable-dom0-access uuid=<pci uuid> since UUID for the above pci is not generated (and not visible in xe pci-list)

@olivierlambert @stormi

Another bug I encountered ( I don't know if this is to be mentioned here or whether I should open this as an issue in github )
Also, this bug may be present in previous versions as the current version is the one I have first tried this on:

Here is the summary:

If USB Keyboard & Mouse is passed-through along-with GPU:
The GPU gets stuck in D3 state (on Shutdown/Restart of VM) (Classic GPU reset problem)

If no vUSB is passed but GPU is passed through:
The GPU works correctly and resets correctly (on Shutdown/Restart of VM)

I will try the workaround of passing the whole usb controller to see how it goes; but in my use case that may not be possible for regular usage (I'll just be doing this for testing only)

Update :
When I run :

$> lspci
Extract of Output (Partial):

07:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8

However, this controller does not show up when I run :
xe pci-list

Is it a bug that lspci & xe pci-list have different number of devices ?

How can I pass this controller since xe pci-list does not show it so I can't get the UUID ?
Will kernel parameters (like XCP-ng 8.2) work in this case ?

Is it safe to run on XCP-ng host ?

 echo 1 > /sys/bus/pci/rescan

(I'm trying to find a way where the PCI card is reset by the host without complete reboot, though I am aware that the above command will not reset it.)

Also is it advisable to use :

xl pci-assignable-add 07:00.0

in XCP-ng 8.3 ? or is this method deprecated ?

@tjkreidl
Thanks once again for your help and guidance!
I have seen/read many 'how to' videos/docs. The problem was not the method I was using. I managed to get this working albeit there is a bug in XCG-ng I suppose.

1. If USB Keyboard & Mouse is passed-through along-with GPU:
   The GPU gets stuck in D3 state (Classic GPU reset problem)

2. If no vUSB is passed but GPU is passed through:
   The GPU works correctly and resets correctly

@tjkreidl

Thanks so much for your response !
I agree with you that the drivers may be a problem. I am starting again from scratch and re-installing everything.
Though a question does remain that whether we need to use both options (Attach PCI + GPU Selection) or either option for GPU pass through. Maybe someone in @vates-tech-support can clarify exactly what the option in Xen Orchestra means.

From what I know, the GPU option in the upper menu wasn't there a few version before. Seems to be a new entry.

@olivierlambert

I agree. But just thought I should report nevertheless.

Update: This seems to be related to 7840HS motherboard. I tried it on another AMD Ryzen 7945HX but did not get this error.

Sorry to join the bandwagon late this time, but i got these errors while booting:

[    0.826901] ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.GPP2.WWAN], AE_NOT_FOUND (20180810/dswload2-160)
[    0.826908] ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20180810/psobject-221)
[    0.826910] ACPI Error: Ignore error and continue table load (20180810/psobject-604)
[    0.826912] ACPI Error: Skip parsing opcode OpcodeName unavailable (20180810/psloop-543)
[    0.827236] ACPI BIOS Error (bug): Could not resolve [\_SB.PCI0.GPP2.WWAN], AE_NOT_FOUND (20180810/dswload2-160)
[    0.827239] ACPI Error: AE_NOT_FOUND, During name lookup/catalog (20180810/psobject-221)
[    0.827241] ACPI Error: Ignore error and continue table load (20180810/psobject-604)
[    0.827242] ACPI Error: Skip parsing opcode OpcodeName unavailable (20180810/psloop-543)
[    0.827245] ACPI BIOS Error (bug): Failure creating [\_SB.PCI0.GPP5.EWPM], AE_ALREADY_EXISTS (20180810/dswload2-316)
[    0.827248] ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20180810/psobject-221)
[    0.827250] ACPI BIOS Error (bug): Failure creating [\_SB.PCI0.GPP5._PRW], AE_ALREADY_EXISTS (20180810/dswload2-316)
[    0.827252] ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20180810/psobject-221)
[    0.827254] ACPI Error: Skip parsing opcode OpcodeName unavailable (20180810/psloop-543)
[    0.827257] ACPI BIOS Error (bug): Failure creating [\_SB.PCI0.GPP5.RTL8._S0W], AE_ALREADY_EXISTS (20180810/dswload2-316)
[    0.827259] ACPI Error: AE_ALREADY_EXISTS, During name lookup/catalog (20180810/psobject-221)

I think this is after the new update

Processor : AMD Ryzen 7 7840HS

@MattRC

Thanks for your reply. I am using RTX 4060 and have tried to install new drivers as well. But no matter what I do, I am having problems. And yes I have passed both the pci ids to the vm.

For Linux Guests:

1. Remove Nvidia Drivers -> Guest is able to see the Card in lspci
2. Install nvidia-drivers-575/570 or *-open or -server-open, then the VM cannot detect the card.

For Windows Guests:
I get Code 43 even after installing the latest driver.

I have done the passthrough as per the documentation and @lawrencesystems video.

One thing I noticed is that when I attach the PCI to the VM the Console Display in XO takes longer to load (don't know if this is related)

PS: even I have the same use-case of passing the PCI SSD directly to the VM for gaming but that doesn't seem to work either.

Can you confirm if you used passthrough in the Advanced->GPU Section and set it as passthough or used the 'Attach PCI' section below? or did you enable at both places ?

@MattRC

Were you able to get GPU Passthrough to work ? I had followed all steps, but my card stops getting detected inside the Guest VM (still detected on the host dom0) once I install the Nvidia drivers. Or at times it gives me a Code 43.

@tjkreidl

Once I install graphics drivers (nvidia) and allow pci passthrough of the graphics card, i get:

NVRM: cpuidInfoAMD Unrecognised AMD processor in cpuInfoAMD

and the VM refuses to boot

If I remove the drivers or detach the card, then the VM boots

@tjkreidl said in GPU Passthrough:

After adding the PCI device, you might also need to specifically enable "passthrough" for that device within the VM's settings. It may do it automatically when you add it.

This is one of my confusions. It is not automatically adding it. Whether I should select it or leave it blank is the confusion.

@tjkreidl

You are trying to do passthrough to a specific VM? I don't think that used to be supported, but maybe is now.

Yes. 1 Card -> 1 VM. Nvidia opened the passthrough lock in 2021. (Before that it was locked)

Are NVIDIA drivers installed on the VM, as needed?

I am installing them(tried the open one), but they are causing issues.

Also, is IOMMU supported and enabled in the BIOS?

Yes

@olivierlambert

Can you please confirm if I need to attach PCI AND select the GPU in the upper section or either should work ?

@tjkreidl

Still struggling to make it work. Simple passthrough is 'hanging' the VM and the VM gets stuck at splash screen.

Trying to figure out what to do

Do you know if I need to put the card in attach pci or just select it in the gpu section or both ?

@tjkreidl
I do have one iGPU & another RTX 4060. I was just confirming which option to use

In the new Xen Orchestra there are two places where passthrough can be used in 'Advanced Section'

Here :

When you click on the (+) Sign

And this section :

Which one should be used for passthrough ? Or can either be used ? Or both should be used ?

@ronan-a @dthenot @Team-Storage

Guys, Can you please clarify which method to use for installing XOSTOR in XCP-ng 8.3 ?

Simple :

yum install xcp-ng-linstor
yum install xcp-ng-release-linstor
./install --disks /dev/nvme0n1 --thin

Or the script in the first post ?
Or Some other script ?

@olivierlambert

Since XOSTOR is now supported on XCP-ng 8.3 LTS, should we use the same script or some other method is required ?
Can you remove the heading which states the script is only compatible with 8.2 ?