Hello @mgr42 I can confirm that indeed today AMD iGPU passthrough is not working on XCP-ng. I am currently (slowly) working on it. There are other references about this topic in the Forum and on GitHub FYI: https://xcp-ng.org/forum/topic/11918/amd-barcelo-passthrough-issues-any-success-stories https://xcp-ng.org/forum/topic/8909/issue-to-load-gpu-passthrough-invalid-pci-rom-header-signature-expecting-0xaa55-got-0x4556 https://github.com/xcp-ng/xcp/issues/806 https://github.com/xapi-project/xen-api/commit/930ffb34e42d674b8234c3d8399fdba97efa6b38 On the setup I am working on, the Expansion ROM is provided indeed by firmware via the VFCT ACPI table. I dumped it to a file and provided it to qemu-dm xen-pci-passthrough device via the romfile= argument (via a modified XAPI build). I can now indeed see a ROM bar in the guest but unfortunately its content is not the content of the file therefore nothing works (amdgpu driver is unable to get its ROM and fails). So now I'm debugging qemu to try to understand the issue with the emulated/passed-through ROM BAR. Regards, Yann Kernel & Hypervisor team timemaster5 created this issue in xcp-ng/xcp open Unable to set romfile for xen-pci-passthrough devices #806 0 freddy77 committed to xapi-project/xen-api CA-418993: Enable PCI ROM BAR before attempting to give permissions to the VM Currently the VM is not able to access the ROM as the host ROM BAR is disabled and not set. Signed-off-by: Frediano Ziglio <frediano.ziglio@citrix.com>

@yllar I'm not sure, maybe it'll slip to July. But what I can say is that the ISO is currently being tested in our QA process. Brace yourself, it's coming soon

Meant to update this again last night - as a road map for those who might try similar at some point: This community's documentation led me to some relevant information on mouse/keyboard pass-through. Namely, if you want this to work you have to pass the entire PCI controller through to the guest(s), Won't work with motherboard devices. But I'm out of PCIe slots. What to do... Simple. Chuck the two serial adapters and install two dual USB 2.0 adapters in their places. Connect the keyboard/mouse to one of the ports and a 4/8/16 port USB to serial adapter to the other. Then pass the entire controller through. In this particular server (R810) I have 5 available PCIe slots. Two will be used for the video cards, one for a 4x NVMe to PCIe adapter (which is eventually where the VMs will reside) and the remaining two for the USB interfaces. This effectively gives me two discrete physical systems, a third available VM instance (say, for network management, monitoring or whatever) and the XOA instance. For application/radio control systems hypervisors (nested) I'll probably have 2ea Win11 and whatever flavor of Linux that supports the GT730 GPU on the system - each assigned the pass-through devices - but only two running at any given time. So no resource contention issues. As far as the displays: I have the main interface of the server connected via SIP to a 2160AS KVM. I'll take 2 more SIPs and use these as the keyboard/mouse inputs for the USB cards. Video...I use an HDMI active splitter on other systems in the network so I'll employ that here. One of the ports on the add-in GPUs goes to the splitter, one of the splitter outputs goes to the SIP via HDMI to VGA adapter and the other feeds the main ceiling mount monitor. The Primary display will be replicated to the KVM screen. Resolution isn't an issue as I have a video scaler inline with the KVM for just this scenario. The concept could be expanded. I'm looking at picking up a T630 and building 4 effective "systems" out of it, following the same principle. The 630 has 8 available PCIe slots. Will update when the next batch of hardware arrives - or I get the Linux side of things sorted. That's next.

@niko thanks, that would help a ton!

Good to hear, thanks a lot for your feedback.

That leftover NIC is probably a stale PIF that XAPI is still holding, even though the card is physically gone, and the Refresh button re-scans rather than removing it. The docs have proper "remove a physical NIC" steps that end in forgetting the old PIF with xe pif-forget: https://docs.xcp-ng.org/networking/#remove-a-physical-nic. For the GPU side, the PCI passthrough flow (hiding the device from dom0, then assigning it) is here: https://docs.xcp-ng.org/compute#detaching-a-pci-device. If the GPU still won't show up as assignable once the stale PIF is gone, it might be worth a mention to @Team-Hypervisor-Kernel.

@poddingue Thank you kindly! Honestly, whatever organizational structure you think is best is fine by me.

@maximsachs Thanks for getting back to me on this. I think (and hope) that in the mean time we will have released an official new ISO that fixes the issue you are having. So, you will be better off testing the new official ISO rather than this unofficial one I'll try to ping here when the new official ISO is out. Thanks! Regards, Yann

Thank you all (@olivierlambert , @teddyastie ) for your valuable responses

@tjkreidl I found that some Intel PRO GPU card have SRV-IO without any additional license, but this functionality is supported in Linux kernel 6.12 and up. I hope in new XCP-NG will have 6.12+ and support GPU sharing.

@taghjichte From my testing you would need to passthought a device for audio. If need for professional audio work then yes a pci passthough would be the perferred option. With that said. @dinhngtu As for sound I have never looked much into it but the only vm I heard make sound is from Fedora even while booted from install iso. Kubuntu or windows with xen drivers installed no audio. (find for me at the moment) While audio seems to work in Fedora from iso it seems to be limited to left channel. Local system is connected to 5.1 sound.

Nice!! Thanks for the feedback @mattrc , it's really cool to see that you can fully enjoy your machine

From what I understand, the OS-to-iDRAC pass-through is a direct USB link between the host and the BMC, designed for the hypervisor host itself to talk to the iDRAC, not something that would naturally bridge through Xen's networking layer into a VM. That might be why the PIF shows up in XCP-ng but the VM still can't reach 169.254.0.1; the bare metal case works because the OS sits directly on the hardware with no virtualisation layer in between. If you need the VM to have direct BMC access, USB passthrough might be worth exploring; XCP-ng does support passing USB devices through to VMs (https://docs.xcp-ng.org/compute#usb-passthrough), though I'm not certain the iDRAC virtual USB NIC would show up as a passthrough-able device. Might be worth a mention to @Team-Hypervisor-Kernel to find out; others here will know more than me.

Hello @codemercenary In theory Xen is NUMA aware (See https://wiki.xenproject.org/wiki/Xen_on_NUMA_Machines) so you would leave memory interleaving option of the BIOS off so that Xen does what it thinks is necessary. But these kinds of settings are very very workload sensitive and in fact it's usually a good idea to just test what works best for you. Regards, Yann

@yannsionneau Thank you very much. Please let me know if there’s anything I can help with.

@gleveque Hello Can you retry with an up-to-date XCP-ng 8.3 host? PCI expansion ROM are now enabled when doing pass-through. The guest VM can now access the ROM bar. Regards, Yann

I've seen this happen after a BIOS update or if a CMOS battery starts failing. Sometimes the motherboard resets to 'Legacy' instead of 'UEFI' (or vice versa), and GRUB just loses its mind because it can't find the partition it expects. It’s worth a quick check in your firmware settings to make sure the boot mode and the drive order haven't shifted on you.

@TeddyAstie I hadn't seen anything special was needed in a VM other that a kernel new enough for support which 6.17 has, is there more to the story with adding things to Ubuntu?

@samuelolavo Thanks for your answer It's very weird because by seeing the command outputs that you pasted, it looks like everything is behaving as it should be. Even the PCI ID (segment:bus:device:function) seems to stay correct (0000:03:00.0) I'll ask others internally.