@steff22
I am making good progress on this and I will soon have some packages to share for people to try and give feedback

@maximsachs Thanks for getting back to me on this.
I think (and hope) that in the mean time we will have released an official new ISO that fixes the issue you are having.
So, you will be better off testing the new official ISO rather than this unofficial one
I'll try to ping here when the new official ISO is out.

Thanks!
Regards,
Yann

@gleveque Hello

Can you retry with an up-to-date XCP-ng 8.3 host?
PCI expansion ROM are now enabled when doing pass-through.
The guest VM can now access the ROM bar.

Regards,

Yann

Hello @mgr42
I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng.
I am currently (slowly) working on it.

There are other references about this topic in the Forum and on GitHub FYI:

• https://xcp-ng.org/forum/topic/11918/amd-barcelo-passthrough-issues-any-success-stories
• https://xcp-ng.org/forum/topic/8909/issue-to-load-gpu-passthrough-invalid-pci-rom-header-signature-expecting-0xaa55-got-0x4556
• https://github.com/xcp-ng/xcp/issues/806
• https://github.com/xapi-project/xen-api/commit/930ffb34e42d674b8234c3d8399fdba97efa6b38

On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table.
I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build).

I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails).

So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR.

Regards,

Yann
Kernel & Hypervisor team

@yllar I'm not sure, maybe it'll slip to July.
But what I can say is that the ISO is currently being tested in our QA process.
Brace yourself, it's coming soon

@yllar @mik @Pilow @marcvds Hello, we think we now understand better this issue.
We think it's a Xen issue with regard to PCI devices on segment 0001 (which is your case) and not a dom0 driver issue.

Could you give a try to this ISO installer please?
Please note that this is a custom test ISO and it is therefore not officially supported, do not use it for production, it is just to confirm that it fixes your issue, we will release a supported ISO in the future (target date for now is June).

unsupported test ISO for you to test: https://nextcloud.vates.tech/index.php/s/iHf8sqq39EqE683

Regards,
Yann

Hello @codemercenary
In theory Xen is NUMA aware (See https://wiki.xenproject.org/wiki/Xen_on_NUMA_Machines) so you would leave memory interleaving option of the BIOS off so that Xen does what it thinks is necessary.

But these kinds of settings are very very workload sensitive and in fact it's usually a good idea to just test what works best for you.

Regards,

Yann

@dkg Please also note that you should not use @jul1an ISO installer on production environment. Only on test env where you can afford to break things and lose data.

FYI this is not being forgotten, we still have a card about this assigned to someone, it will just take a bit more time than I thought before we work on this.
I'll send updates here when we have new elements.
Regards,
Yann