Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work
-
@olivierlambert Hi, I have tried but it is not easy to find things in an Asrock bios. But have enabled svm and Iommu. Are there more things I need to enable?
Is an amd ryzen 9 9950x Cpu -
BIOS is up to date?
-
@olivierlambert Yes but is beta bios to support the new Amd Ryzen9000 series. But there may be some bugs in the bios,
I think the motherboard was launched after April this year. First bios slip 27/7/2024.
I have contacted Asrock support, But there are so many links that the error could lie in.
But there are several with B650D4U3 motherboards that have the same error
-
Yeah, I have the impression it's a buggy motherboard. Let's see if at some point you can manage to get it working
Keep us posted! -
@olivierlambert Yes, I hope so
. I don't need so many cpu cores if I don't have any ext gpuIs it the bios that determines which screen the xcp-ng console should be displayed on?
As soon as I choose passthrough enable, I lose this image on the screen associated with the ext video card.
But it doesn't come back on any of the other screens connected to the motherboard, nor does ipmi show this xcp-ng console.
even though I have selected primary video adapter to int graphics in the bios.
This is the main difference on the Radeon HD 5450 that gpu passthrough worked with. There, the primary video adapter remained on the graphics section of the motherboard.So one suspects that this choice in the bios does not work.
The strange thing is that normal pci passthrough works as it should on both m.2 and pci usb controller card
Asrock suport wanted to try to reproduce the error with them but have no idea if they will choose to use Xcp-ng and xen orchestra. It takes quite a long time to set everything up
-
Tell them that if they want "assistance" to reproduce, we are keen to discuss directly with them
-
@olivierlambert Got it working with proxmox now by hooking on an option (PCI-Exspress).
can one address that there is a pci-e device in the same way in Xcp-ng with a command line?
As soon as I removed this option it stopped working and I got the same error 43 back
-
Pinging @Teddy-Astie
-
Hello @steff22 ,
Tested with another PC with Xcp-ng 8.3 and everything worked immediately. So think it must be Asrock rack B650D4U3-2L2Q bios related or ipmi trying to use the same gpu.
What are the specs of the hardware where it works ?
-
@Teddy-Astie It's an asus prime h370f with an old 8 core intel core i7 9700k
-
I have gpu pcie passthrough working on Asrock rack wrx80d8-2t (Threadripper), and x470d4u2 (Ryzen) to passthrough multiple Radeon Pro 7000 series gpus
Rodney
-
@ravenet Do you remember if you only enabled svm and Iommu? Or did you take the whole bunch with SR-iov, ACS,AER,ARI support, resize bar Support
-
@steff22
Far as I know, SVM, IOMMU, SR-IOV and POSSIBLY resize bar were enabled.
Can't check the threadripper system until the weekend as it runs production loads.I'll reboot one of the Ryzen boxes tonight to verify what I enabled there, but was basically defaults other than making sure the 3 or 4 items above were enabled.
-
@ravenet ok thanks
Didn't work here.
But there may be some error in the bios since it is beta and the first bios that supports the Amd Ryzen9000 series.But the strange thing is that it works stably with Proxmox without any errors of any kind.
-
It's not strange, those a two VERY different system in how they work.
-
@olivierlambert Is there a slight software difference, yes But both are hypervisors.
So the fact that one of them works means that it should be possible to make it happen with my hardware.
Isn't that much you do with proxmox rather than enable iommu in grub and blacklist nvidia drivers plus disable_vga out on the Gpu card.
Is it possible to blacklist the nvidia drivers with Xcp-ng?
And there is no difference to enable pci passthrough in xen-orchestra compared to command-line?I think I have tried both parts but not sure if I had enabled everything in the bios then
-
No, it's not a slight difference: it's a completely different design. In XCP-ng, you start first to boot on Xen, a kind of microkernel. Then only you boot on specific VM, the Dom0, which is a PV guest. Then, from this guest, you have the API etc. But if the Dom0 has access to the hardware for I/O (NICs, disks, GPUs…), it's still a VM and Dom0 doesn't have access to all CPUs and memory of the physical machine.
This means, Xen will always be "in the middle" to control who takes what (on cores and RAM). This is a great design from a security perspective because you always have a small piece of software (Xen) that is controlling what's going on.
In KVM, it's vastly different. You boot on a full Linux kernel, and then you load a small module (KVM). The "host" is really the host, accessing all the memory and CPUs. There's a lot less isolation (and therefore security), however the plus is that there's nobody in the middle to deal with.
I can assure you that PCI passthrough is really complex and also depends on many factors, even how the BIOS is configured and how many things are done by the motherboard manufacturer.
So knowing it works with Proxmox is as saying "it works on Windows", it's that different. It doesn't mean it's not possible in Xen, but if it does, it will be vastly different on how it's done.
-
@olivierlambert Ok, didn't realize there was such a big difference. I actually like everything better with Xcp-ng and xen orchestra.
All testing has been done with UEFI BIOS. Is it just a waste of time to try with legacy bios on Xcp-ng and vm os?
Does it still make sense to try to blacklist the nvidia drivers with Xcp-ng Dom0 to try to isolate the Gpu even more.
But is Xcp-ng more like Vmware esxi in the way everything is handled? doesn't work with vmware esxi either.
-
Yes, you could say XCP-ng is a lot closer to VMware than KVM
(even if VMware is more advanced on some aspects, the architecture is roughly similar).Anyway, it doesn't mean that will never work; However, it might be less straightforward than with KVM until we find exactly what's causing the problem.
-
@steff22 Assuming the
xen-pciback.hidewas previously set, could you try this workaround (no guarantee that'll work, since each motherboard and BIOSes have their quirks):/opt/xensource/libexec/xen-cmdline --set-dom0 pci=realloc reboot
