@maximsachs Thanks for getting back to me on this.
I think (and hope) that in the mean time we will have released an official new ISO that fixes the issue you are having.
So, you will be better off testing the new official ISO rather than this unofficial one 
I'll try to ping here when the new official ISO is out.
Thanks!
Regards,
Yann
@gleveque Hello
Can you retry with an up-to-date XCP-ng 8.3 host?
PCI expansion ROM are now enabled when doing pass-through.
The guest VM can now access the ROM bar.
Regards,
Yann
Hello @mgr42
I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng.
I am currently (slowly) working on it.
There are other references about this topic in the Forum and on GitHub FYI:
On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table.
I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build).
I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails).
So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR.
Regards,
Yann
Kernel & Hypervisor team
@yllar I'm not sure, maybe it'll slip to July.
But what I can say is that the ISO is currently being tested in our QA process.
Brace yourself, it's coming soon
@yllar @mik @Pilow @marcvds Hello, we think we now understand better this issue.
We think it's a Xen issue with regard to PCI devices on segment 0001 (which is your case) and not a dom0 driver issue.
Could you give a try to this ISO installer please?
Please note that this is a custom test ISO and it is therefore not officially supported, do not use it for production, it is just to confirm that it fixes your issue, we will release a supported ISO in the future (target date for now is June).
unsupported test ISO for you to test: https://nextcloud.vates.tech/index.php/s/iHf8sqq39EqE683
Regards,
Yann
Hello @codemercenary
In theory Xen is NUMA aware (See https://wiki.xenproject.org/wiki/Xen_on_NUMA_Machines) so you would leave memory interleaving option of the BIOS off so that Xen does what it thinks is necessary.
But these kinds of settings are very very workload sensitive and in fact it's usually a good idea to just test what works best for you.
Regards,
Yann
FYI this is not being forgotten, we still have a card about this assigned to someone, it will just take a bit more time than I thought before we work on this.
I'll send updates here when we have new elements.
Regards,
Yann
Just FYI, we had some issues with some Debian VMs for instance, with old kernel
where the VM would suddenly take 100% CPU and would we completely frozen and unresponsive.
It was in fact Debian enabling the suspend, the guest would suspend and trigger a bug in xen PV driver in the guest kernel which would never be able to wake up anymore.
Newer kernel don't have this issue and are able to wake up.
I'm quoting Debian here but it might very well happen on other distro
@PessimistTech Let's try to see if we have errors in the logs.
Run ls /boot/xen* and note the name of the xen file that ends with -d.gz
Then, can you reboot and , while in the grub menu, edit the XCP-ng menu entry (the first one) and make the following modifications:
debug Xen build by specifying the file that ends with -d.gz instead of /boot/xen.gziommu=debugThen boot (by doing ctrl+X) and once booted and the VM with PCI passthrough started, please issue xl dmesg command in dom0 shell and report the content here.
Also please paste the result of dom0 dmesg.
then also an lspci -vvv -nn -s SBDF from both dom0 and the VM
replace SBDF with the segment bus device function address of the device (beware it won't be the same in dom0 and inside the VM).
Regards,
Yann
Maybe let's ping the entire @Team-Hypervisor-Kernel here
Hi @pessimisttech let me ping @Team-Hypervisor-Kernel here
Hello @mgr42
I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng.
I am currently (slowly) working on it.
There are other references about this topic in the Forum and on GitHub FYI:
On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table.
I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build).
I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails).
So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR.
Regards,
Yann
Kernel & Hypervisor team
@yllar I'm not sure, maybe it'll slip to July.
But what I can say is that the ISO is currently being tested in our QA process.
Brace yourself, it's coming soon
@yllar I'm not sure honestly (I will have to ask the rest of the team), but anyway, even if it were fixed by latest xen package, we still didn't publish a new installer ISO with the fix.
Target is still around June for the new ISO.
Regards,
Yann
@maximsachs Thanks for getting back to me on this.
I think (and hope) that in the mean time we will have released an official new ISO that fixes the issue you are having.
So, you will be better off testing the new official ISO rather than this unofficial one
I'll try to ping here when the new official ISO is out.
Thanks!
Regards,
Yann
@yllar @mik @Pilow @marcvds Hello, we think we now understand better this issue.
We think it's a Xen issue with regard to PCI devices on segment 0001 (which is your case) and not a dom0 driver issue.
Could you give a try to this ISO installer please?
Please note that this is a custom test ISO and it is therefore not officially supported, do not use it for production, it is just to confirm that it fixes your issue, we will release a supported ISO in the future (target date for now is June).
unsupported test ISO for you to test: https://nextcloud.vates.tech/index.php/s/iHf8sqq39EqE683
Regards,
Yann
Hello @maximsachs,
on top of trying what @thierryescande told you, could you also give a try to this ISO installer please?
We have done some internal investigations and we think that the issue could be because of an issue in Xen handling of PCI devices that are on 0001 segment which seems to be your case.
Please note that this is a custom test ISO and it is therefore not officially supported, do not use it for production, it is just to confirm that it fixes your issue, we will release a supported ISO in the future (target date for now is June).
unsupported test ISO for you to test: https://nextcloud.vates.tech/index.php/s/iHf8sqq39EqE683
Regards,
Yann