XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    iGPU PCI passthrough: new experimental packages need testers

    Scheduled Pinned Locked Moved Hardware
    12 Posts 2 Posters 37 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • O
      ovicz
      last edited by

      Hi. I only have this after acpidump -b on a igpu from intel alder lake cpu.

      ls -l | grep dat
      -rw-r--r-- 1 root root 476 Oct 8 11:35 apic.dat
      -rw-r--r-- 1 root root 92 Oct 8 11:35 dbg2.dat
      -rw-r--r-- 1 root root 52 Oct 8 11:35 dbgp.dat
      -rw-r--r-- 1 root root 480216 Oct 8 11:35 dsdt.dat
      -rw-r--r-- 1 root root 276 Oct 8 11:35 facp.dat
      -rw-r--r-- 1 root root 64 Oct 8 11:35 facs.dat
      -rw-r--r-- 1 root root 156 Oct 8 11:35 fidt.dat
      -rw-r--r-- 1 root root 68 Oct 8 11:35 fpdt.dat
      -rw-r--r-- 1 root root 56 Oct 8 11:35 hpet.dat
      -rw-r--r-- 1 root root 204 Oct 8 11:35 lpit.dat
      -rw-r--r-- 1 root root 60 Oct 8 11:35 mcfg.dat
      -rw-r--r-- 1 root root 45 Oct 8 11:35 nhlt.dat
      -rw-r--r-- 1 root root 1553 Oct 8 11:35 phat.dat
      -rw-r--r-- 1 root root 136 Oct 8 11:35 rmad.dat
      -rw-r--r-- 1 root root 1850 Oct 8 11:35 ssdt10.dat
      -rw-r--r-- 1 root root 15082 Oct 8 11:35 ssdt11.dat
      -rw-r--r-- 1 root root 14810 Oct 8 11:35 ssdt12.dat
      -rw-r--r-- 1 root root 324 Oct 8 11:35 ssdt13.dat
      -rw-r--r-- 1 root root 916 Oct 8 11:35 ssdt14.dat
      -rw-r--r-- 1 root root 419 Oct 8 11:35 ssdt15.dat
      -rw-r--r-- 1 root root 427 Oct 8 11:35 ssdt16.dat
      -rw-r--r-- 1 root root 7087 Oct 8 11:35 ssdt17.dat
      -rw-r--r-- 1 root root 4937 Oct 8 11:35 ssdt18.dat
      -rw-r--r-- 1 root root 4027 Oct 8 11:35 ssdt19.dat
      -rw-r--r-- 1 root root 908 Oct 8 11:35 ssdt1.dat
      -rw-r--r-- 1 root root 23819 Oct 8 11:35 ssdt2.dat
      -rw-r--r-- 1 root root 10549 Oct 8 11:35 ssdt3.dat
      -rw-r--r-- 1 root root 13271 Oct 8 11:35 ssdt4.dat
      -rw-r--r-- 1 root root 54407 Oct 8 11:35 ssdt5.dat
      -rw-r--r-- 1 root root 7962 Oct 8 11:35 ssdt6.dat
      -rw-r--r-- 1 root root 10883 Oct 8 11:35 ssdt7.dat
      -rw-r--r-- 1 root root 9047 Oct 8 11:35 ssdt8.dat
      -rw-r--r-- 1 root root 6425 Oct 8 11:35 ssdt9.dat
      -rw-r--r-- 1 root root 76 Oct 8 11:35 tpm2.dat
      -rw-r--r-- 1 root root 72 Oct 8 11:35 uefi.dat
      -rw-r--r-- 1 root root 40 Oct 8 11:35 wsmt.dat

      And the python script throws an error :

      python3 -c 'import struct,sys;d=open(sys.argv[1],"rb").read();assert d[:4]==b"VFCT",d[:4];n=struct.unpack("<I",d[4:8])[0];off=struct.unpack("<I",d[0x34:0x38])[0];i=0
      while off and off+28<=n:
      pcibus,pcidev,pcifunc,vendor,device,ssvid,ssid,rev,imagelen=struct.unpack("<IIIHHHHII",d[off:off+28])
      if imagelen==0 or off+28+imagelen>n:break
      fn="vbios_%02d_%04x_%04x.bin"%(i,vendor,device);open(fn,"wb").write(d[off+28:off+28+imagelen]);print(fn,"bus=%d dev=%d func=%d vid=%04x did=%04x len=%d"%(pcibus,pcidev,pcifunc,vendor,device,imagelen));off+=28+imagelen;i+=1' vfct.dat
      Traceback (most recent call last):
      File "<string>", line 1, in <module>
      FileNotFoundError: [Errno 2] No such file or directory: 'vfct.dat'

      Y O 2 Replies Last reply
      Reply Quote 0
      • Y
        yannsionneau Vates 🪐 XCP-ng Team @ovicz
        last edited by

        @ovicz a dit:

        intel alder lake cpu

        Thanks for giving this a try!
        Can you give me the output of lspci -n -vvv -s SBDF_OF_GPU on your dom0 please?

        1 Reply Last reply
        Reply Quote 0
        • O
          ovicz @ovicz
          last edited by ovicz

          lspci -n -vvv -s 00:02.0
          00:02.0 0300: 8086:46d1 (prog-if 00 [VGA controller])
          Subsystem: 8086:7270
          Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
          Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
          Interrupt: pin A routed to IRQ 16
          Region 0: Memory at 60e7000000 (64-bit, non-prefetchable) [disabled] [size=16M]
          Region 2: Memory at 4000000000 (64-bit, prefetchable) [disabled] [size=256M]
          Region 4: I/O ports at 3000 [disabled] [size=64]
          [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
          Capabilities: [40] Vendor Specific Information: Len=0c <?>
          Capabilities: [70] Express (v2) Root Complex Integrated Endpoint, MSI 00
          DevCap: MaxPayload 128 bytes, PhantFunc 0
          ExtTag- RBE+
          DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
          RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
          MaxPayload 128 bytes, MaxReadReq 128 bytes
          DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
          DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
          DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
          Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit-
          Address: 00000000 Data: 0000
          Masking: 00000000 Pending: 00000000
          Capabilities: [d0] Power Management version 2
          Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
          Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
          Capabilities: [100 v1] Process Address Space ID (PASID)
          PASIDCap: Exec- Priv-, Max PASID Width: 14
          PASIDCtl: Enable- Exec- Priv-
          Capabilities: [200 v1] Address Translation Service (ATS)
          ATSCap: Invalidate Queue Depth: 00
          ATSCtl: Enable-, Smallest Translation Unit: 00
          Capabilities: [300 v1] Page Request Interface (PRI)
          PRICtl: Enable- Reset-
          PRISta: RF- UPRGI- Stopped+
          Page Request Capacity: 00008000, Page Request Allocation: 00000000
          Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
          IOVCap: Migration-, Interrupt Message Number: 000
          IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
          IOVSta: Migration-
          Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
          VF offset: 1, stride: 1, Device ID: 46d1
          Supported Page Size: 00000553, System Page Size: 00000001
          Region 0: Memory at 00000060e0000000 (64-bit, non-prefetchable)
          Region 2: Memory at 0000006000000000 (64-bit, prefetchable)
          VF Migration: offset: 00000000, BIR: 0
          Kernel driver in use: pciback

          Now the VMs don't start at all.

          vm.start
          {
          "id": "6783fdf9-3569-0224-fdc0-0082323305fe",
          "bypassMacAddressesCheck": false,
          "force": false
          }
          {
          "code": "INTERNAL_ERROR",
          "params": [
          "xenopsd internal error: Cannot_add(0000:00:02.0, Device_common.QMP_Error(5, "{\"error\":{\"class\":\"GenericError\",\"desc\":\"failed to find romfile \\\"/root/vbios.bin\\\"\",\"data\":{}},\"id\":\"qmp-000015-5\"}"))"
          ],
          "call": {
          "duration": 4031,
          "method": "VM.start",
          "params": [
          "* session id *",
          "OpaqueRef:2877507a-7ad9-f1e6-2b9c-e5690cb21e76",
          false,
          false
          ]
          },
          "message": "INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:00:02.0, Device_common.QMP_Error(5, "{\"error\":{\"class\":\"GenericError\",\"desc\":\"failed to find romfile \\\"/root/vbios.bin\\\"\",\"data\":{}},\"id\":\"qmp-000015-5\"}")))",
          "name": "XapiError",
          "stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:00:02.0, Device_common.QMP_Error(5, "{\"error\":{\"class\":\"GenericError\",\"desc\":\"failed to find romfile \\\"/root/vbios.bin\\\"\",\"data\":{}},\"id\":\"qmp-000015-5\"}")))
          at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202610080923/packages/xen-api/_XapiError.mjs:16:12)
          at file:///opt/xo/xo-builds/xen-orchestra-202610080923/packages/xen-api/transports/json-rpc.mjs:38:21
          at runNextTicks (node:internal/process/task_queues:65:5)
          at processImmediate (node:internal/timers:541:9)"
          }

          Y 1 Reply Last reply
          Reply Quote 0
          • Y
            yannsionneau Vates 🪐 XCP-ng Team @ovicz
            last edited by

            @ovicz in your case it seems the vbios is in the ROM BAR.
            You can dump the vbios by doing something like

            cd /sys/bus/pci/devices/0000:00:02.0/
echo 1 > rom
cat rom > /root/vbios.bin
echo 0 > rom
            Y O 2 Replies Last reply
            Reply Quote 0
            • Y
              yannsionneau Vates 🪐 XCP-ng Team @yannsionneau
              last edited by

              Another possibility for you would be to yum downgrade qemu and the xapi packages, everything except edk2 and try again.

              O 1 Reply Last reply
              Reply Quote 0
              • O
                ovicz @yannsionneau
                last edited by

                @yannsionneau cd /sys/bus/pci/devices/0000:00:02.0/
                [11:55 xcp-ng-akz 0000:00:02.0]# echo 1 > rom
                [11:55 xcp-ng-akz 0000:00:02.0]# cat rom > /root/vbios.bin
                cat: rom: Input/output error
                [11:55 xcp-ng-akz 0000:00:02.0]# echo 0 > rom

                1 Reply Last reply
                Reply Quote 0
                • O
                  ovicz @yannsionneau
                  last edited by

                  @yannsionneau Reverted to stable and I upgraded only that package from your repo. VMs start now. At what should i look for now ?

                  Y O 2 Replies Last reply
                  Reply Quote 0
                  • Y
                    yannsionneau Vates 🪐 XCP-ng Team @ovicz
                    last edited by yannsionneau

                    @ovicz with the new edk2 package it should help your VM access the vbios in the rom bar.
                    Does the VM see the gpu?
                    Can you share the VM dmesg as attached file?
                    Also, are you trying to passthrough a sriov virtual function? or the entire physical gpu?
                    Thanks

                    1 Reply Last reply
                    Reply Quote 0
                    • O
                      ovicz @ovicz
                      last edited by

                      Yes it does see it.
                      dmesg.txt
                      Passthrough was working and without that file from your repo.

                      Y 1 Reply Last reply
                      Reply Quote 0
                      • Y
                        yannsionneau Vates 🪐 XCP-ng Team @ovicz
                        last edited by

                        @ovicz It seems your gpu works even without vbios since in your dmesg I can see

                        [    7.424250] i915 0000:00:08.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[    7.424259] i915 0000:00:08.0: [drm] Failed to find VBIOS tables (VBT)

                        So you're saying that anyway your setup was working even without all those new packages?
                        Thanks for testing anyway!

                        O 1 Reply Last reply
                        Reply Quote 0
                        • O
                          ovicz @yannsionneau
                          last edited by

                          @yannsionneau Yes it works with the stock packages.

                          1 Reply Last reply
                          Reply Quote 0

                          Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                          Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                          With your input, this post could be even better 💗

                          Register Login
                          • First post
                            Last post