@ovicz It seems your gpu works even without vbios since in your dmesg I can see
[ 7.424250] i915 0000:00:08.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[ 7.424259] i915 0000:00:08.0: [drm] Failed to find VBIOS tables (VBT)
So you're saying that anyway your setup was working even without all those new packages?
Thanks for testing anyway!
@ovicz with the new edk2 package it should help your VM access the vbios in the rom bar.
Does the VM see the gpu?
Can you share the VM dmesg as attached file?
Also, are you trying to passthrough a sriov virtual function? or the entire physical gpu?
Thanks
Another possibility for you would be to yum downgrade qemu and the xapi packages, everything except edk2 and try again.
@ovicz in your case it seems the vbios is in the ROM BAR.
You can dump the vbios by doing something like
cd /sys/bus/pci/devices/0000:00:02.0/
echo 1 > rom
cat rom > /root/vbios.bin
echo 0 > rom
intel alder lake cpu
Thanks for giving this a try!
Can you give me the output of lspci -n -vvv -s SBDF_OF_GPU on your dom0 please?
@antest Hi
I notice in your domU dmesg:
[ 2.425816] i915 0000:00:08.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x4556
[ 2.425820] i915 0000:00:08.0: [drm] Failed to find VBIOS tables (VBT)
If you are on a non-production machine can you give this a try?
=> https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers
Thanks!
Yann
@jp13232 @peek have you tried following this: https://docs.xcp-ng.org/compute/#pci-passthrough ?
Also, you can have a look there: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers if you finally manage to do the passthrough but have "vbios issues" like explained in the thread.
@the_spice @bullerwins @xerxist @hawkpro @vhaelan @ovicz
Hi
I published a post this morning with packages to test iGPU passthrough, if you have a non production host on which you can test it would be awesome: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers
Thanks!
Regards,
Yann
@mgr42 Hi
I published a post this morning with packages to test, if you have a non production host on which you can test it would be awesome: https://xcp-ng.org/forum/topic/12523/igpu-pci-passthrough-new-experimental-packages-need-testers
Thanks!
Regards,
Yann
DO NOT TEST THIS IN PRODUCTIONI did a Proof-of-Concept to improve the iGPU passthrough situation in the case where the vbios is not in the ROM BAR. This would usually fail in the guest because the guest amdgpu or i915driver would not be able to find the vbios.
I would like to share this PoC with you to gather feedback / testing.
This is not at all a production ready solution, as you will see with the manual steps which are needed to test this.
Who should test this?
If you have an host with an iGPU, and the usual 8.3 PCI passthrough procedure described at https://docs.xcp-ng.org/compute/#pci-passthrough does not work because the guest dmesg prints something like
Invalid PCI ROM header signature: expecting 0xaa55, got 0x4556
or
Failed to find VBIOS tables (VBT)
or
Unable to locate a BIOS ROM
What has been tested so far
I only tested on an AMD Ryzen Embedded V2748 with Radeon Graphics embedded APU system with a Radeon RX Vega series integrated GPU (PCI vendor id 1002 PCI device id 1636). With this config, I tested that the guest VM was able to correctly probe the GPU with the amgdpu driver.
How to test
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo
yum update --enablerepo=xcp-ng-ysionneau1
=============================================================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================================================================================================================================================================================
Updating:
edk2 x86_64 20220801-1.7.11.3~romtest.2.xcpng8.3 xcp-ng-ysionneau1 3.0 M
forkexecd x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 2.5 M
message-switch x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 4.6 M
qcow-stream-tool x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 1.4 M
qemu x86_64 2:4.2.1-5.2.18.1.0.fixrom.1.xcpng8.3 xcp-ng-ysionneau1 16 M
rrdd-plugins x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 9.9 M
sm-cli x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 4.9 M
squeezed x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 1.9 M
varstored-guard x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 5.0 M
vhd-tool x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 5.0 M
wsproxy x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 1.1 M
xapi-core x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 30 M
xapi-nbd x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 3.1 M
xapi-rrd2csv x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 3.0 M
xapi-storage-script x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 2.5 M
xapi-tests x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 5.0 M
xapi-xe x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 1.5 M
xcp-networkd x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 4.9 M
xcp-rrdd x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 3.6 M
xenopsd x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 1.4 M
xenopsd-cli x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 2.0 M
xenopsd-xc x86_64 26.1.19-1.2.0.romtest.1.xcpng8.3 xcp-ng-ysionneau1 5.3 M
Transaction Summary
=============================================================================================================================================================================================================================================================================================================================
Upgrade 22 Packages
/root/vbios.bin
VFCT, you could first call acpidump -b then use this """one liner"""python3 -c 'import struct,sys;d=open(sys.argv[1],"rb").read();assert d[:4]==b"VFCT",d[:4];n=struct.unpack("<I",d[4:8])[0];off=struct.unpack("<I",d[0x34:0x38])[0];i=0
while off and off+28<=n:
pcibus,pcidev,pcifunc,vendor,device,ssvid,ssid,rev,imagelen=struct.unpack("<IIIHHHHII",d[off:off+28])
if imagelen==0 or off+28+imagelen>n:break
fn="vbios_%02d_%04x_%04x.bin"%(i,vendor,device);open(fn,"wb").write(d[off+28:off+28+imagelen]);print(fn,"bus=%d dev=%d func=%d vid=%04x did=%04x len=%d"%(pcibus,pcidev,pcifunc,vendor,device,imagelen));off+=28+imagelen;i+=1' vfct.dat
firmware-amd-graphics or firmware-intel-graphicsdmesg if the GPU is correctly detected and probed by the driver (amdgpu for AMD for instance)What to report
Report as replies to this thread, putting text files as attached-files
dmesg of the Linux VMxl dmesg of the host (dom0)grep xen-pci-passthrough /var/log/xensource.log*lspci -n -vvv -s SBDF (replace SBDF with the domain:bus:device.function) or your GPU, both in the host (dom0) and in the VM (beware the SBDF will be different!)hexdump -C /boot/vbios.bin | head -n100 on the host (dom0)Thank you for your help! 
I get out of my silence to confirm that I'm indeed working on AMD MxGPU driver and AMD iGPU passthrough issues.
I confess that I have not communicated since a while on this work, but I'm making progress and I'm very close to publish a call to testing PoC packages that should improve the iGPU passthrough situation.
@steff22
I am making good progress on this and I will soon have some packages to share for people to try and give feedback 
Just FYI, we had some issues with some Debian VMs for instance, with old kernel
where the VM would suddenly take 100% CPU and would we completely frozen and unresponsive.
It was in fact Debian enabling the suspend, the guest would suspend and trigger a bug in xen PV driver in the guest kernel which would never be able to wake up anymore.
Newer kernel don't have this issue and are able to wake up.
I'm quoting Debian here but it might very well happen on other distro
@PessimistTech Let's try to see if we have errors in the logs.
Run ls /boot/xen* and note the name of the xen file that ends with -d.gz
Then, can you reboot and , while in the grub menu, edit the XCP-ng menu entry (the first one) and make the following modifications:
debug Xen build by specifying the file that ends with -d.gz instead of /boot/xen.gziommu=debugThen boot (by doing ctrl+X) and once booted and the VM with PCI passthrough started, please issue xl dmesg command in dom0 shell and report the content here.
Also please paste the result of dom0 dmesg.
then also an lspci -vvv -nn -s SBDF from both dom0 and the VM
replace SBDF with the segment bus device function address of the device (beware it won't be the same in dom0 and inside the VM).
Regards,
Yann
Maybe let's ping the entire @Team-Hypervisor-Kernel here
Hi @pessimisttech let me ping @Team-Hypervisor-Kernel here
Hello @mgr42
I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng.
I am currently (slowly) working on it.
There are other references about this topic in the Forum and on GitHub FYI:
On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table.
I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build).
I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails).
So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR.
Regards,
Yann
Kernel & Hypervisor team
@yllar I'm not sure, maybe it'll slip to July.
But what I can say is that the ISO is currently being tested in our QA process.
Brace yourself, it's coming soon
@yllar I'm not sure honestly (I will have to ask the rest of the team), but anyway, even if it were fixed by latest xen package, we still didn't publish a new installer ISO with the fix.
Target is still around June for the new ISO.
Regards,
Yann
@maximsachs Thanks for getting back to me on this.
I think (and hope) that in the mean time we will have released an official new ISO that fixes the issue you are having.
So, you will be better off testing the new official ISO rather than this unofficial one
I'll try to ping here when the new official ISO is out.
Thanks!
Regards,
Yann