XCP-ng 8.3 & AMD Firepro S7150x2

TeddyAstie

Nov 13 11:30:21 xen03 kernel: [10188.720655] AMD IOMMUv2 driver by Joerg Roedel jroedel@suse.de
Nov 13 11:30:21 xen03 kernel: [10188.720656] AMD IOMMUv2 functionality not available on this system

This is expected, Dom0 Kernel (Linux) is not supposed to access the IOMMU when it is already used by Xen. To check if AMD-Vi is working, you need to check xl dmesg instead.

I took a quick look at kern_gim_compiled.txt, and it look likes it timed-out somewhere

Oct 23 20:49:32 xen03 kernel: [   80.657394]        gim error:(wait_cmd_complete:2387)  wait_cmd_complete -- time out after 0.003004460 sec
Oct 23 20:49:32 xen03 kernel: [   80.657408]        gim error:(wait_cmd_complete:2390)   Cmd = 0x17, Status = 0x0, cmd_Complete=0

3ms looks like a short timeout for me, but aside that, it looks like a driver(gim) or hardware issue

TeddyAstie

@tuxen said (https://xcp-ng.org/forum/topic/3652/no-free-virtual-function-found-vgpu-s7150/4?_=1731502751059)

After some digging, could be the case of a GPU firmware being incompatible with UEFI. Do you have any spare server for testing XCP-ng boot in legacy/BIOS with this GPU?

Perhaps it is the issue ?

ohajek

Hi Teddy,
thanks for the analysis and your brief explanation.
IOMMU should be properly activated:

(XEN) [ 0.221843] AMD-Vi: IOMMU Extended Features:
(XEN) [ 0.222606] - Peripheral Page Service Request
(XEN) [ 0.223366] - NX bit
(XEN) [ 0.224123] - Guest APIC Physical Processor Interrupt
(XEN) [ 0.224889] - Invalidate All Command
(XEN) [ 0.225649] - Guest APIC
(XEN) [ 0.226412] - Performance Counters
(XEN) [ 0.227178] - Host Address Translation Size: 0x2
(XEN) [ 0.227940] - Guest Address Translation Size: 0
(XEN) [ 0.228681] - Guest CR3 Root Table Level: 0x1
(XEN) [ 0.229416] - Maximum PASID: 0xf
(XEN) [ 0.230140] - SMI Filter Register: 0x1
(XEN) [ 0.230867] - SMI Filter Register Count: 0x1
(XEN) [ 0.231596] - Guest Virtual APIC Modes: 0x1
(XEN) [ 0.232316] - Dual PPR Log: 0x2
(XEN) [ 0.233024] - Dual Event Log: 0x2
(XEN) [ 0.233727] - Secure ATS
(XEN) [ 0.234424] - User / Supervisor Page Protection
(XEN) [ 0.235126] - Device Table Segmentation: 0x3
(XEN) [ 0.235826] - PPR Log Overflow Early Warning
(XEN) [ 0.236514] - PPR Automatic Response
(XEN) [ 0.237198] - Memory Access Routing and Control: 0x1
(XEN) [ 0.237881] - Block StopMark Message
(XEN) [ 0.238558] - Performance Optimization
(XEN) [ 0.239234] - MSI Capability MMIO Access
(XEN) [ 0.239906] - Guest I/O Protection
(XEN) [ 0.240570] - Enhanced PPR Handling
(XEN) [ 0.241231] - Invalidate IOTLB Type
(XEN) [ 0.241886] - VM Table Size: 0x2
(XEN) [ 0.242537] - Guest Access Bit Update Disable
(XEN) [ 0.252988] AMD-Vi: IOMMU 0 Enabled.
(XEN) [ 0.253820] I/O virtualisation enabled

I have also read about the problem with UEFI several times.
Now I am going to try to find a system with "legacy" boot mode.

rtjdamen

@ohajek did u ever managed to get this to work? We upgraded from 8.2 where it worked to 8.3, the card is detected but when u start the vm u get virtual function not available

ohajek

@rtjdamen No, unfortunately not! I tried everything possible and finally gave up ...
I wish you more luck getting it to work!

rtjdamen

@ohajek i am affraid then it’s a lost case with this one, @olivierlambert do you have any knowledge of this going to work on xcp?

olivierlambert

I'm completely E_BUSY at the moment (at the Xen Winter Meetup), so I can't do much on long/complex problems on non-urgent situation

mohammadm

Too bad, so at this point we have 4/5 GPU's that are pretty much useless. Any other alternatives for GPU?

When using dGPU, PCI-e passthrough to the VM itself, it does work but after a while the whole screen turn black. They have to disconnect and connect to the RDP session again to see the screen again.

TeddyAstie

@mohammadm said in XCP-ng 8.3 & AMD Firepro S7150x2:

Too bad, so at this point we have 4/5 GPU's that are pretty much useless. Any other alternatives for GPU?

When using dGPU, PCI-e passthrough to the VM itself, it does work but after a while the whole screen turn black. They have to disconnect and connect to the RDP session again to see the screen again.

Looks like a screen suspend issue, have you tried to disable it ?

john

@mohammadm

Could you share information about your configuration.

What system do you use on guest VM?
What drivers you are using on guest VM?
On VM you are using bios or uefi?

I am trying to get to the point where VM can detect GPU but I have some problems. Recreating your guest configuration can simplify things.