iGPU PCI passthrough: new experimental packages need testers

[ovicz]

lspci -n -vvv -s 00:02.0
00:02.0 0300: 8086:46d1 (prog-if 00 [VGA controller])
Subsystem: 8086:7270
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 16
Region 0: Memory at 60e7000000 (64-bit, non-prefetchable) [disabled] [size=16M]
Region 2: Memory at 4000000000 (64-bit, prefetchable) [disabled] [size=256M]
Region 4: I/O ports at 3000 [disabled] [size=64]
[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express (v2) Root Complex Integrated Endpoint, MSI 00
DevCap: MaxPayload 128 bytes, PhantFunc 0
ExtTag- RBE+
DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
MaxPayload 128 bytes, MaxReadReq 128 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
Capabilities: [ac] MSI: Enable- Count=1/1 Maskable+ 64bit-
Address: 00000000 Data: 0000
Masking: 00000000 Pending: 00000000
Capabilities: [d0] Power Management version 2
Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [100 v1] Process Address Space ID (PASID)
PASIDCap: Exec- Priv-, Max PASID Width: 14
PASIDCtl: Enable- Exec- Priv-
Capabilities: [200 v1] Address Translation Service (ATS)
ATSCap: Invalidate Queue Depth: 00
ATSCtl: Enable-, Smallest Translation Unit: 00
Capabilities: [300 v1] Page Request Interface (PRI)
PRICtl: Enable- Reset-
PRISta: RF- UPRGI- Stopped+
Page Request Capacity: 00008000, Page Request Allocation: 00000000
Capabilities: [320 v1] Single Root I/O Virtualization (SR-IOV)
IOVCap: Migration-, Interrupt Message Number: 000
IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy-
IOVSta: Migration-
Initial VFs: 7, Total VFs: 7, Number of VFs: 0, Function Dependency Link: 00
VF offset: 1, stride: 1, Device ID: 46d1
Supported Page Size: 00000553, System Page Size: 00000001
Region 0: Memory at 00000060e0000000 (64-bit, non-prefetchable)
Region 2: Memory at 0000006000000000 (64-bit, prefetchable)
VF Migration: offset: 00000000, BIR: 0
Kernel driver in use: pciback

Now the VMs don't start at all.

vm.start
{
"id": "6783fdf9-3569-0224-fdc0-0082323305fe",
"bypassMacAddressesCheck": false,
"force": false
}
{
"code": "INTERNAL_ERROR",
"params": [
"xenopsd internal error: Cannot_add(0000:00:02.0, Device_common.QMP_Error(5, "{\"error\":{\"class\":\"GenericError\",\"desc\":\"failed to find romfile \\\"/root/vbios.bin\\\"\",\"data\":{}},\"id\":\"qmp-000015-5\"}"))"
],
"call": {
"duration": 4031,
"method": "VM.start",
"params": [
"* session id *",
"OpaqueRef:2877507a-7ad9-f1e6-2b9c-e5690cb21e76",
false,
false
]
},
"message": "INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:00:02.0, Device_common.QMP_Error(5, "{\"error\":{\"class\":\"GenericError\",\"desc\":\"failed to find romfile \\\"/root/vbios.bin\\\"\",\"data\":{}},\"id\":\"qmp-000015-5\"}")))",
"name": "XapiError",
"stack": "XapiError: INTERNAL_ERROR(xenopsd internal error: Cannot_add(0000:00:02.0, Device_common.QMP_Error(5, "{\"error\":{\"class\":\"GenericError\",\"desc\":\"failed to find romfile \\\"/root/vbios.bin\\\"\",\"data\":{}},\"id\":\"qmp-000015-5\"}")))
at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202610080923/packages/xen-api/_XapiError.mjs:16:12)
at file:///opt/xo/xo-builds/xen-orchestra-202610080923/packages/xen-api/transports/json-rpc.mjs:38:21
at runNextTicks (node:internal/process/task_queues:65:5)
at processImmediate (node:internal/timers:541:9)"
}

[yannsionneau]

@ovicz in your case it seems the vbios is in the ROM BAR.
You can dump the vbios by doing something like

cd /sys/bus/pci/devices/0000:00:02.0/
echo 1 > rom
cat rom > /root/vbios.bin
echo 0 > rom

[yannsionneau]

Another possibility for you would be to yum downgrade qemu and the xapi packages, everything except edk2 and try again.

[ovicz]

@yannsionneau cd /sys/bus/pci/devices/0000:00:02.0/
[11:55 xcp-ng-akz 0000:00:02.0]# echo 1 > rom
[11:55 xcp-ng-akz 0000:00:02.0]# cat rom > /root/vbios.bin
cat: rom: Input/output error
[11:55 xcp-ng-akz 0000:00:02.0]# echo 0 > rom

[ovicz]

@yannsionneau Reverted to stable and I upgraded only that package from your repo. VMs start now. At what should i look for now ?

[yannsionneau]

@ovicz with the new edk2 package it should help your VM access the vbios in the rom bar.
Does the VM see the gpu?
Can you share the VM dmesg as attached file?
Also, are you trying to passthrough a sriov virtual function? or the entire physical gpu?
Thanks

[ovicz]

Yes it does see it.
dmesg.txt
Passthrough was working and without that file from your repo.

[yannsionneau]

@ovicz It seems your gpu works even without vbios since in your dmesg I can see

[    7.424250] i915 0000:00:08.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x0000
[    7.424259] i915 0000:00:08.0: [drm] Failed to find VBIOS tables (VBT)

So you're saying that anyway your setup was working even without all those new packages?
Thanks for testing anyway!

[ovicz]

@yannsionneau Yes it works with the stock packages.

[yannsionneau]

@ovicz Thanks a lot for the feedbacks and tests