yannsionneau

yannsionneau

@steff22
I am making good progress on this and I will soon have some packages to share for people to try and give feedback

yannsionneau

@maximsachs Thanks for getting back to me on this.
I think (and hope) that in the mean time we will have released an official new ISO that fixes the issue you are having.
So, you will be better off testing the new official ISO rather than this unofficial one
I'll try to ping here when the new official ISO is out.

Thanks!
Regards,
Yann

yannsionneau

@gleveque Hello

Can you retry with an up-to-date XCP-ng 8.3 host?
PCI expansion ROM are now enabled when doing pass-through.
The guest VM can now access the ROM bar.

Regards,

Yann

yannsionneau

I get out of my silence to confirm that I'm indeed working on AMD MxGPU driver and AMD iGPU passthrough issues.
I confess that I have not communicated since a while on this work, but I'm making progress and I'm very close to publish a call to testing PoC packages that should improve the iGPU passthrough situation.

yannsionneau

Hello @mgr42
I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng.
I am currently (slowly) working on it.

There are other references about this topic in the Forum and on GitHub FYI:

On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table.
I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build).

I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails).

So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR.

Regards,

Yann
Kernel & Hypervisor team

yannsionneau

@yllar I'm not sure, maybe it'll slip to July.
But what I can say is that the ISO is currently being tested in our QA process.
Brace yourself, it's coming soon

yannsionneau

@yllar @mik @Pilow @marcvds Hello, we think we now understand better this issue.
We think it's a Xen issue with regard to PCI devices on segment 0001 (which is your case) and not a dom0 driver issue.

Could you give a try to this ISO installer please?
Please note that this is a custom test ISO and it is therefore not officially supported, do not use it for production, it is just to confirm that it fixes your issue, we will release a supported ISO in the future (target date for now is June).

unsupported test ISO for you to test: https://nextcloud.vates.tech/index.php/s/iHf8sqq39EqE683

Regards,
Yann

yannsionneau

Hello @codemercenary
In theory Xen is NUMA aware (See https://wiki.xenproject.org/wiki/Xen_on_NUMA_Machines) so you would leave memory interleaving option of the BIOS off so that Xen does what it thinks is necessary.

But these kinds of settings are very very workload sensitive and in fact it's usually a good idea to just test what works best for you.

Regards,

Yann

yannsionneau

@dkg Please also note that you should not use @jul1an ISO installer on production environment. Only on test env where you can afford to break things and lose data.

yannsionneau

FYI this is not being forgotten, we still have a card about this assigned to someone, it will just take a bit more time than I thought before we work on this.
I'll send updates here when we have new elements.
Regards,
Yann

yannsionneau

@ovicz with the new edk2 package it should help your VM access the vbios in the rom bar.
Does the VM see the gpu?
Can you share the VM dmesg as attached file?
Also, are you trying to passthrough a sriov virtual function? or the entire physical gpu?
Thanks

yannsionneau

Another possibility for you would be to yum downgrade qemu and the xapi packages, everything except edk2 and try again.

yannsionneau

@ovicz in your case it seems the vbios is in the ROM BAR.
You can dump the vbios by doing something like

cd /sys/bus/pci/devices/0000:00:02.0/
echo 1 > rom
cat rom > /root/vbios.bin
echo 0 > rom

yannsionneau

@ovicz a dit:

intel alder lake cpu

Thanks for giving this a try!
Can you give me the output of lspci -n -vvv -s SBDF_OF_GPU on your dom0 please?

yannsionneau

@antest Hi

I notice in your domU dmesg:

[    2.425816] i915 0000:00:08.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x4556
[    2.425820] i915 0000:00:08.0: [drm] Failed to find VBIOS tables (VBT)

If you are on a non-production machine can you give this a try?
=> https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers

Thanks!
Yann

yannsionneau

@jp13232 @peek have you tried following this: https://docs.xcp-ng.org/compute/#pci-passthrough ?

Also, you can have a look there: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers if you finally manage to do the passthrough but have "vbios issues" like explained in the thread.

yannsionneau

@the_spice @bullerwins @xerxist @hawkpro @vhaelan @ovicz

Hi

I published a post this morning with packages to test iGPU passthrough, if you have a non production host on which you can test it would be awesome: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers
Thanks!

Regards,

Yann

yannsionneau

@mgr42 Hi

I published a post this morning with packages to test, if you have a non production host on which you can test it would be awesome: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers
Thanks!

Regards,

Yann

yannsionneau

DO NOT TEST THIS IN PRODUCTION

I did a Proof-of-Concept to improve the iGPU passthrough situation in the case where the vbios is not in the ROM BAR. This would usually fail in the guest because the guest amdgpu or i915driver would not be able to find the vbios.
I would like to share this PoC with you to gather feedback / testing.

This is not at all a production ready solution, as you will see with the manual steps which are needed to test this.

Who should test this?

If you have an host with an iGPU, and the usual 8.3 PCI passthrough procedure described at https://docs.xcp-ng.org/compute/#pci-passthrough does not work because the guest dmesg prints something like

Invalid PCI ROM header signature: expecting 0xaa55, got 0x4556

or

Failed to find VBIOS tables (VBT)

or

Unable to locate a BIOS ROM

What has been tested so far

I only tested on an AMD Ryzen Embedded V2748 with Radeon Graphics embedded APU system with a Radeon RX Vega series integrated GPU (PCI vendor id 1002 PCI device id 1636). With this config, I tested that the guest VM was able to correctly probe the GPU with the amgdpu driver.

How to test

Install a brand new XCP-ng 8.3 on a test host (or make sure you are up to date with latest 8.3 packages on your test host)
Configure XCP-ng user repositories:

wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

Update your system with the iGPU PoC packages

yum update --enablerepo=xcp-ng-ysionneau1

it will prompt you about importing a new GPG key
it should install the following packages:

=============================================================================================================================================================================================================================================================================================================================
 Package                                                                      Arch                                                            Version                                                                                       Repository                                                                  Size
=============================================================================================================================================================================================================================================================================================================================
Updating:
 edk2                                                                         x86_64                                                          20220801-1.7.11.3~romtest.2.xcpng8.3                                                          xcp-ng-ysionneau1                                                          3.0 M
 forkexecd                                                                    x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          2.5 M
 message-switch                                                               x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          4.6 M
 qcow-stream-tool                                                             x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.4 M
 qemu                                                                         x86_64                                                          2:4.2.1-5.2.18.1.0.fixrom.1.xcpng8.3                                                          xcp-ng-ysionneau1                                                           16 M
 rrdd-plugins                                                                 x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          9.9 M
 sm-cli                                                                       x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          4.9 M
 squeezed                                                                     x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.9 M
 varstored-guard                                                              x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          5.0 M
 vhd-tool                                                                     x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          5.0 M
 wsproxy                                                                      x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.1 M
 xapi-core                                                                    x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                           30 M
 xapi-nbd                                                                     x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          3.1 M
 xapi-rrd2csv                                                                 x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          3.0 M
 xapi-storage-script                                                          x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          2.5 M
 xapi-tests                                                                   x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          5.0 M
 xapi-xe                                                                      x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.5 M
 xcp-networkd                                                                 x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          4.9 M
 xcp-rrdd                                                                     x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          3.6 M
 xenopsd                                                                      x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          1.4 M
 xenopsd-cli                                                                  x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          2.0 M
 xenopsd-xc                                                                   x86_64                                                          26.1.19-1.2.0.romtest.1.xcpng8.3                                                              xcp-ng-ysionneau1                                                          5.3 M

Transaction Summary
=============================================================================================================================================================================================================================================================================================================================
Upgrade  22 Packages

Then you will have to manually dump your iGPU VBIOS to precisely this path: /root/vbios.bin
- for AMD modern iGPU it should be located in an ACPI table called VFCT, you could first call acpidump -b then use this """one liner"""

python3 -c 'import struct,sys;d=open(sys.argv[1],"rb").read();assert d[:4]==b"VFCT",d[:4];n=struct.unpack("<I",d[4:8])[0];off=struct.unpack("<I",d[0x34:0x38])[0];i=0
while off and off+28<=n:
  pcibus,pcidev,pcifunc,vendor,device,ssvid,ssid,rev,imagelen=struct.unpack("<IIIHHHHII",d[off:off+28])
  if imagelen==0 or off+28+imagelen>n:break
  fn="vbios_%02d_%04x_%04x.bin"%(i,vendor,device);open(fn,"wb").write(d[off+28:off+28+imagelen]);print(fn,"bus=%d dev=%d func=%d vid=%04x did=%04x len=%d"%(pcibus,pcidev,pcifunc,vendor,device,imagelen));off+=28+imagelen;i+=1' vfct.dat

Reboot the host
Follow the usual PCI Passthrough procedure to pass-through the iGPU PCI device to a Linux VM (a Debian 13 for instance): https://docs.xcp-ng.org/compute/#pci-passthrough
Make sure you have the corresponding gpu firmware Debian package installed in your VM: firmware-amd-graphics or firmware-intel-graphics
Start the VM
Check in the dmesg if the GPU is correctly detected and probed by the driver (amdgpu for AMD for instance)

What to report
Report as replies to this thread, putting text files as attached-files

If it worked or not
the dmesg of the Linux VM
the xl dmesg of the host (dom0)
the output of grep xen-pci-passthrough /var/log/xensource.log*
the lspci -n -vvv -s SBDF (replace SBDF with the domain:bus:device.function) or your GPU, both in the host (dom0) and in the VM (beware the SBDF will be different!)
hexdump -C /boot/vbios.bin | head -n100 on the host (dom0)

Thank you for your help!

yannsionneau

I get out of my silence to confirm that I'm indeed working on AMD MxGPU driver and AMD iGPU passthrough issues.
I confess that I have not communicated since a while on this work, but I'm making progress and I'm very close to publish a call to testing PoC packages that should improve the iGPU passthrough situation.

yannsionneau

@yannsionneau

DO NOT TEST THIS IN PRODUCTION