@tuxen said (https://xcp-ng.org/forum/topic/3652/no-free-virtual-function-found-vgpu-s7150/4?_=1731502751059)
After some digging, could be the case of a GPU firmware being incompatible with UEFI. Do you have any spare server for testing XCP-ng boot in legacy/BIOS with this GPU?
Perhaps it is the issue ?
Nov 13 11:30:21 xen03 kernel: [10188.720655] AMD IOMMUv2 driver by Joerg Roedel jroedel@suse.de Nov 13 11:30:21 xen03 kernel: [10188.720656] AMD IOMMUv2 functionality not available on this system
This is expected, Dom0 Kernel (Linux) is not supposed to access the IOMMU when it is already used by Xen. To check if AMD-Vi is working, you need to check xl dmesg instead.
I took a quick look at kern_gim_compiled.txt, and it look likes it timed-out somewhere
Oct 23 20:49:32 xen03 kernel: [ 80.657394] gim error:(wait_cmd_complete:2387) wait_cmd_complete -- time out after 0.003004460 sec
Oct 23 20:49:32 xen03 kernel: [ 80.657408] gim error:(wait_cmd_complete:2390) Cmd = 0x17, Status = 0x0, cmd_Complete=0
3ms looks like a short timeout for me, but aside that, it looks like a driver(gim) or hardware issue
PCIe AER needs proper PCIe, which in practice needs Q35 chipset in the guest (or some other guest type/PCI passthrough way).
Q35 support is currently work in progress
I think it is the same MSI-X/PBA issues that may be partially fixed with https://gitlab.com/xen-project/xen/-/commit/b2cd07a0447bfa25e96ae13e190225b61a3670cb
However, with this device, MSI-X vector table and PBA are in a same page (vector table in 46800 and PBA in 46068) though, which is threated a bit differently
If PBA lives on the same page, discard writes and log a message.
Technically, writes outside of PBA could be allowed, but at this moment
the precise location of PBA isn't saved, and also no known device abuses
the spec in this way (at least yet).
But Coral appears to abuse this according to DKMS driver by having more than just MSI-X and PBA on a single page
https://github.com/google/gasket-driver/blob/main/src/apex_driver.c#L103-L140
Hello @hvm,
Can you give the output of xl dmesg in XCP-ng and of dmesg in the guest that has the issues ?
I have the impression that something is going wrong with reserved regions related to the SATA controller.
PCIe AER needs proper PCIe, which in practice needs Q35 chipset in the guest (or some other guest type/PCI passthrough way).
Q35 support is currently work in progress
@brodiecyber said in Question on CPU masking with qemu and xen:
Hello all
I have a question but dot know enough about Xen hypervisor to answer it.
So it simple why cant XCP-ng mask the type of CPU that the vms running on.
Take for example the KVM on proxmox you can set host pass-through to send all CPU functions to the vm or mask it to an intel Xeon scalable or AMD Epyc.
I see that host pass-through is what XCP-ng does but was wondering why.
It doesn't exactly work this way.
With XCP-ng, it works at pool level, it basically takes the lowest CPU feature level of your hosts of the pool, and sets all VM to use it. That way you should always be able to migrate to another machine of the same pool, while not having to manually set the feature levels to improve performance.
Doesnt masking the cpu make the vm more portable as the profile can be set to for example QEMU-xen or something making it possible to say migrate a vm live from AMD to intel n different pools or a heterogeneous cluster.
In practice, you can't reliably live migrate between Intel and AMD architecture (even with QEMU) due to various subtle architecture differences. It may somewhat work with luck if the guest is using a ancient processing model (like 32-bits) but that's not really something we support.
And one last thing if it is possible is there no way to emulate other architecture supported by qemu such as arm64 or does xen hypervisor still need more features built out on arm before that i possible. Asking because my use case is i cant afford a raspberry pi but have alot of compute on x86 why not emulate arm 64 on the hypervisor rather than in a vm
Emulating a completely different CPU architecture while technically possible, is a completely different story.
@tuxen said (https://xcp-ng.org/forum/topic/3652/no-free-virtual-function-found-vgpu-s7150/4?_=1731502751059)
After some digging, could be the case of a GPU firmware being incompatible with UEFI. Do you have any spare server for testing XCP-ng boot in legacy/BIOS with this GPU?
Perhaps it is the issue ?
Nov 13 11:30:21 xen03 kernel: [10188.720655] AMD IOMMUv2 driver by Joerg Roedel jroedel@suse.de Nov 13 11:30:21 xen03 kernel: [10188.720656] AMD IOMMUv2 functionality not available on this system
This is expected, Dom0 Kernel (Linux) is not supposed to access the IOMMU when it is already used by Xen. To check if AMD-Vi is working, you need to check xl dmesg instead.
I took a quick look at kern_gim_compiled.txt, and it look likes it timed-out somewhere
Oct 23 20:49:32 xen03 kernel: [ 80.657394] gim error:(wait_cmd_complete:2387) wait_cmd_complete -- time out after 0.003004460 sec
Oct 23 20:49:32 xen03 kernel: [ 80.657408] gim error:(wait_cmd_complete:2390) Cmd = 0x17, Status = 0x0, cmd_Complete=0
3ms looks like a short timeout for me, but aside that, it looks like a driver(gim) or hardware issue
@steff22 said in Gpu passthrough on Asrock rack B650D4U3-2L2Q will not work:
@tuxen Got hold of custom bios with G4 available now. But disabling 4g didn't help.. It was hidden but always enabled
But see that in Proxmox the pci slot does not change. it stays at slot 1 with Xcp-ng it changes to slot 8.
Is this slot change in the guest or in XCP-ng itself ?
I would not be supprised to see it chaging in the guest (it's a part of what QEMU pci passthrough can do) but that would be weird for Dom0.
Proxmox user uses hex pci id for gpu passthrough does this matter?
I don't think it's a guarantee that you see the same BDF in guest and host regarding in proxmox (more likely a coincidence). But that's a behavior that would change in XCP-ng with Q35/proper PCIe support.
Aside that, it "appears" to work in a Linux VM (albeit with no display), have you managed to make CUDA running there (to see if the device actually works) ? And if the BAR window message still exist.
@steff22 It appears to work, though it appears to had to make extra steps for that.
[ 0.773716] pci 0000:00:06.0: can't claim BAR 1 [mem 0xfce0000000-0xfcefffffff 64bit pref]: no compatible bridge window
[ 0.773722] pci 0000:00:06.0: can't claim BAR 3 [mem 0xfcf0000000-0xfcf1ffffff 64bit pref]: no compatible bridge window
...
[ 0.797979] pci 0000:00:06.0: BAR 1: assigned [mem 0xe0000000-0xefffffff 64bit pref]
[ 0.799824] pci 0000:00:06.0: BAR 3: assigned [mem 0xf0000000-0xf1ffffff 64bit pref]
AFAIU, the device was mapped initially outside of the 4G range, and Linux had to remap it elsewhere, I think it is due to a mix of QEMU PIIX3 limitations and (host) BIOS configuration (Above 4G Decoding or something like that) so the device has its BAR mapped somewhere invalid from guest point of view.
I would suppose that is "works" in the Linux case but fails in the Windows case.
If you find a above 4g decoding parameter in the host BIOS, you should try to disable it and check if it fixes (workarround) the issue.
@steff22 I was more thinking of regular dmesg in the PopOS VM.
Do you have something different if you try to passthrough the same device into a Linux VM.
I suppose something would show up in dmesg in this case.
@slavox The patch I linked is not applied to current XCP-ng.
But even if it was, it would still not work due to the MSI-X/PBA/registers issue in a same page I quoted previously.
It's not a simple issue to tackle on, but upstream Xen is aware of that and it may be solved in the future (difficult to put an ETA though).
I think it is the same MSI-X/PBA issues that may be partially fixed with https://gitlab.com/xen-project/xen/-/commit/b2cd07a0447bfa25e96ae13e190225b61a3670cb
However, with this device, MSI-X vector table and PBA are in a same page (vector table in 46800 and PBA in 46068) though, which is threated a bit differently
If PBA lives on the same page, discard writes and log a message.
Technically, writes outside of PBA could be allowed, but at this moment
the precise location of PBA isn't saved, and also no known device abuses
the spec in this way (at least yet).
But Coral appears to abuse this according to DKMS driver by having more than just MSI-X and PBA on a single page
https://github.com/google/gasket-driver/blob/main/src/apex_driver.c#L103-L140