Just FYI, we had some issues with some Debian VMs for instance, with old kernel
where the VM would suddenly take 100% CPU and would we completely frozen and unresponsive.
It was in fact Debian enabling the suspend, the guest would suspend and trigger a bug in xen PV driver in the guest kernel which would never be able to wake up anymore.
Newer kernel don't have this issue and are able to wake up.
I'm quoting Debian here but it might very well happen on other distro

@PessimistTech Let's try to see if we have errors in the logs.

Run ls /boot/xen* and note the name of the xen file that ends with -d.gz

Then, can you reboot and , while in the grub menu, edit the XCP-ng menu entry (the first one) and make the following modifications:

• modify the Xen ELF loaded by multiboot2 to load the debug Xen build by specifying the file that ends with -d.gz instead of /boot/xen.gz
• on the same line, at the end, add iommu=debug

Then boot (by doing ctrl+X) and once booted and the VM with PCI passthrough started, please issue xl dmesg command in dom0 shell and report the content here.

Also please paste the result of dom0 dmesg.

then also an lspci -vvv -nn -s SBDF from both dom0 and the VM

replace SBDF with the segment bus device function address of the device (beware it won't be the same in dom0 and inside the VM).

Regards,

Yann

Maybe let's ping the entire @Team-Hypervisor-Kernel here

Hi @pessimisttech let me ping @Team-Hypervisor-Kernel here

Hello @mgr42
I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng.
I am currently (slowly) working on it.

There are other references about this topic in the Forum and on GitHub FYI:

• https://xcp-ng.org/forum/topic/11918/amd-barcelo-passthrough-issues-any-success-stories
• https://xcp-ng.org/forum/topic/8909/issue-to-load-gpu-passthrough-invalid-pci-rom-header-signature-expecting-0xaa55-got-0x4556
• https://github.com/xcp-ng/xcp/issues/806
• https://github.com/xapi-project/xen-api/commit/930ffb34e42d674b8234c3d8399fdba97efa6b38

On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table.
I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build).

I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails).

So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR.

Regards,

Yann
Kernel & Hypervisor team

@yllar I'm not sure, maybe it'll slip to July.
But what I can say is that the ISO is currently being tested in our QA process.
Brace yourself, it's coming soon

@yllar I'm not sure honestly (I will have to ask the rest of the team), but anyway, even if it were fixed by latest xen package, we still didn't publish a new installer ISO with the fix.
Target is still around June for the new ISO.
Regards,
Yann

@maximsachs Thanks for getting back to me on this.
I think (and hope) that in the mean time we will have released an official new ISO that fixes the issue you are having.
So, you will be better off testing the new official ISO rather than this unofficial one
I'll try to ping here when the new official ISO is out.

Thanks!
Regards,
Yann

@yllar @mik @Pilow @marcvds Hello, we think we now understand better this issue.
We think it's a Xen issue with regard to PCI devices on segment 0001 (which is your case) and not a dom0 driver issue.

Could you give a try to this ISO installer please?
Please note that this is a custom test ISO and it is therefore not officially supported, do not use it for production, it is just to confirm that it fixes your issue, we will release a supported ISO in the future (target date for now is June).

unsupported test ISO for you to test: https://nextcloud.vates.tech/index.php/s/iHf8sqq39EqE683

Regards,
Yann

Hello @maximsachs,

on top of trying what @thierryescande told you, could you also give a try to this ISO installer please?
We have done some internal investigations and we think that the issue could be because of an issue in Xen handling of PCI devices that are on 0001 segment which seems to be your case.

Please note that this is a custom test ISO and it is therefore not officially supported, do not use it for production, it is just to confirm that it fixes your issue, we will release a supported ISO in the future (target date for now is June).

unsupported test ISO for you to test: https://nextcloud.vates.tech/index.php/s/iHf8sqq39EqE683

Regards,
Yann

@dkg Oops sorry I thought you were someone else having the same issues.

Hello @codemercenary
In theory Xen is NUMA aware (See https://wiki.xenproject.org/wiki/Xen_on_NUMA_Machines) so you would leave memory interleaving option of the BIOS off so that Xen does what it thinks is necessary.

But these kinds of settings are very very workload sensitive and in fact it's usually a good idea to just test what works best for you.

Regards,

Yann

@dkg Please also note that you should not use @jul1an ISO installer on production environment. Only on test env where you can afford to break things and lose data.

@timemaster5 To give you more feedback, I am currently testing with an AMD Ryzen Embedded V2748 with Radeon Graphics host with a Renoir [Radeon Vega Series / Radeon Vega Mobile Series] iGPU.
I'm also talking to XAPI team to coordinate the effort and try to see what we can/want to do about this.
My first effort is to confirm what works/does not work and give all the information they need to the XAPI team so that they can make the decision on what to do.
I'll update you with my findings.

Can you retry with an up-to-date xcp-ng 8.3 please?

FYI on recent XCP-ng 8.3 versions the pci-passthrough will enable the ROM expansion bar. The guest VM will have access to it, so no need to pass it via qemu anymore.
See my comment on GitHub: https://github.com/xcp-ng/xcp/issues/786#issuecomment-4281846490

Regards,

Yann

@gleveque Hello

Can you retry with an up-to-date XCP-ng 8.3 host?
PCI expansion ROM are now enabled when doing pass-through.
The guest VM can now access the ROM bar.

Regards,

Yann

FYI this is not being forgotten, we still have a card about this assigned to someone, it will just take a bit more time than I thought before we work on this.
I'll send updates here when we have new elements.
Regards,
Yann

Hi,

Someone from the Hypervisor & Kernel team will have a look shortly, we'll get back to you with our findings.

Thanks a lot for the very detailed report!

Yann

@abudef Some quotes from the documentation to clarify the situation: https://docs.xcp-ng.org/compute/#-nested-virtualization

@samuelolavo Thanks for your answer

It's very weird because by seeing the command outputs that you pasted, it looks like everything is behaving as it should be.
Even the PCI ID (segment:bus:device:function) seems to stay correct (0000:03:00.0)

I'll ask others internally.