Google Coral TPU PCIe Passthrough Woes
-
Definitely frustrating and no fault of xcp-ng - I have a lot of spare cpu cycles so it isn't majorly impacting me that I know of. I'm still available to test fixes though.
Looks like most of the Proxmox users have got this working in an LXC container by installing the drivers on the host itself and passing through the actual Apex devices. Not a route that's applicable to us but just a datapoint.
-
@jjgg it would be great if we could get this working. My CPU utilisation is fine too, but when I shut down my Zoneminder VM things go a lot quieter (fans) so I'm sure there would be a benefit CPU and power wise.
- 9 months later
-
@jmccoy555 // @jjgg
Did anyone of you get your Coral USB TPU working and passthrough to a VM?
-
@Nornode hey, nope I did not. I ended up moving my infrastructure to Proxmox.
Honestly this is no fault of XCP-ng and XCP-ng suits my hardware / setup a lot better, but it was either that or I had two servers that needed to be bare metal.
-
PCI passthrough might cause problems with this device, but USB could work.
- 25 days later
-
@olivierlambert Seems like a reasonable place to ask as any - I am currently using a USB Coral over IP (Virtualhere) but would rather load it into my VM directly - what's the current status of snapshots/backups with a vUSB?
I've been reading that XO can now support disk exclusions with
[NOBAK]
but this probably doesn't apply to a Coral. Is an offline backup still the best available method? -
For NOBAK and on 8.3 yes, but I'm not sure it will be related to USB. You should use offline, that should work. Alternatively, we have plans to detect the error, to unplug the USB device, do the snap and replug it just after.
-
@olivierlambert Thanks that's good to know. That functionality would be great down the line!
I do have a spare M.2 E-key on my XCP host running the VM Coral is needed for, but seems like I'd have trouble going by this thread. Might even have trouble with the USB Coral, it hasn't been much better so far in terms of whacky non-standard behavior...
- 6 months later
-
So I eventually got round to trying the USB Coral via passthrough, which worked great, but the TPU itself exhibited some behavior that made it nonviable which sucks. The USB was actually detected by XO as
Google Inc.
and Frigate actually loaded the TPU, but the inference speed was in excess of 180ms (it should be around 10, USB over IP it's 40). So it worked but, didn't.The normal procedure with a Coral is to run a
make reset
from their utilities which switches the TPU back to runtime mode. This worked under my current (and now reverted) system of VirtualHere USB over IP, but it didn't work when passed through.Output of
make reset
:dfu-util: Warning: Invalid DFU suffix signature dfu-util: A valid DFU suffix will be required in a future dfu-util release dfu-util: No DFU capable USB device available
It should look like this:
Opening DFU capable USB device... Device ID 1a6e:089a Device DFU version 0101 Claiming USB DFU Interface... Setting Alternate Interface #0 ... Determining device status... DFU state(2) = dfuIDLE, status(0) = No error condition is present DFU mode device DFU version 0101 Device returned transfer size 256 Copying data from PC to DFU device Download [=========================] 100% 10783 bytes Download done. DFU state(2) = dfuIDLE, status(0) = No error condition is present Done! Resetting USB to switch back to Run-Time mode
Sorry to ping you @olivierlambert but would you happen to know what might cause this in XCP/XO? Is there something going on when the device is made into a vUSB that would cause it to error out/be inaccessible in DFU (I assume this means Device Firmware Update)?
-
Hi,
I don't know internal mechanism of the vUSB thing and why it cause this on your device (which is really a special device, with its own quirks).
I don't remember if you already tried to passthrough a PCIe USB adapter card, then plug the USB device on it and see if it's better than vUSB?
-
@olivierlambert Thanks, don't worry in that case, was just to see if there was something like "oh yeah XCP does [something] with vUSBs when passing through which could explain it". The server is a mini PC so no PCIe card slots or capability unfortunately.
I'll just live with 40ms via VirtualHere (don't know why that's so high either as others have 15-20 with that method)! It works well enough.
- 3 months later
-
hey @andSmv @olivierlambert
I have a PCI coral TPU and have the same issue from this thread. It doesn't look like anyone confirmed if the patch is working.Anything I can do to help test here? I have just switched away from proxmox so would prefer to get it working in XCP
I'm currently on 8.3 and the alt kernel. But happy to test with whatever, I have some spare hardware to setup a dedicated test if needed.uname -a Linux xcp-long 4.19.316+1 #1 SMP Mon Aug 19 14:31:42 CEST 2024 x86_64 x86_64 x86_64 GNU/Linux
xl dmesg
(XEN) [ 3010.009205] d12v5 EPT violation 0x1aa (-w-/r-x) gpa 0x000000f1846800 mfn 0x90246 type 5 (XEN) [ 3010.009207] d12v5 Walking EPT tables for GFN f1846: (XEN) [ 3010.009209] d12v5 epte 9c00000cb3924007 (XEN) [ 3010.009210] d12v5 epte 9c0000084c552007 (XEN) [ 3010.009211] d12v5 epte 9c00000847e9d007 (XEN) [ 3010.009212] d12v5 epte 9c50000090246845 (XEN) [ 3010.009214] d12v5 --- GLA 0xffffaea6c0d8d800 (XEN) [ 3010.009219] domain_crash called from vmx_vmexit_handler+0xa8d/0x1ab0 (XEN) [ 3010.009221] Domain 12 (vcpu#5) crashed on cpu#17: (XEN) [ 3010.009225] ----[ Xen-4.17.5-3 x86_64 debug=n Not tainted ]---- (XEN) [ 3010.009226] CPU: 17 (XEN) [ 3010.009227] RIP: 0010:[<ffffffff8dd86326>] (XEN) [ 3010.009228] RFLAGS: 0000000000010286 CONTEXT: hvm guest (d12v5) (XEN) [ 3010.009231] rax: ffffaea6c0d8d800 rbx: ffff88c634a53800 rcx: 0000000000000000 (XEN) [ 3010.009232] rdx: 00000000fee87000 rsi: 0000000000000000 rdi: 0000000000000000 (XEN) [ 3010.009234] rbp: ffffaea6c0b0f448 rsp: ffffaea6c0b0f410 r8: 0000000000000000 (XEN) [ 3010.009235] r9: 0000000000000000 r10: 0000000000000000 r11: 0000000000000000 (XEN) [ 3010.009236] r12: ffffaea6c0b0f464 r13: 0000000000000011 r14: ffff88c6022860c8 (XEN) [ 3010.009238] r15: 0000000000000087 cr0: 0000000080050033 cr4: 00000000001006f0 (XEN) [ 3010.009239] cr3: 0000000105aca000 cr2: 00007b3046869000 (XEN) [ 3010.009240] fsb: 000079ea9326d8c0 gsb: ffff88cb07280000 gss: 0000000000000000 (XEN) [ 3010.009242] ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: 0018 cs: 0010
lspci -vvv -s
lspci -vvv -s 86:00.0 86:00.0 System peripheral: Global Unichip Corp. Coral Edge TPU (prog-if ff) Subsystem: Global Unichip Corp. Coral Edge TPU Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 56 Region 0: Memory at 901fc000 (64-bit, prefetchable) [size=16K] Region 2: Memory at 90200000 (64-bit, prefetchable) [size=1M] Capabilities: [80] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <64ns, L1 <1us ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W DevCtl: Report errors: Correctable- Non-Fatal+ Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 256 bytes, MaxReadReq 4096 bytes DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend- LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [d0] MSI-X: Enable- Count=128 Masked- Vector table: BAR=2 offset=00046800 PBA: BAR=2 offset=00046068 Capabilities: [e0] MSI: Enable- Count=1/32 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [f8] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [100 v1] Vendor Specific Information: ID=1556 Rev=1 Len=008 <?> Capabilities: [108 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [110 v1] L1 PM Substates L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ PortCommonModeRestoreTime=10us PortTPowerOnTime=10us Capabilities: [200 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES- TLP+ FCP+ CmpltTO+ CmpltAbrt- UnxCmplt- RxOF- MalfTLP+ ECRC+ UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- CEMsk: RxErr+ BadTLP+ BadDLLP+ Rollover+ Timeout+ NonFatalErr+ AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Kernel driver in use: pciback
-
@Teddy-Astie if you have some bandwidth, can you take a look?
-
I think it is the same MSI-X/PBA issues that may be partially fixed with https://gitlab.com/xen-project/xen/-/commit/b2cd07a0447bfa25e96ae13e190225b61a3670cb
However, with this device, MSI-X vector table and PBA are in a same page (vector table in 46800 and PBA in 46068) though, which is threated a bit differently
If PBA lives on the same page, discard writes and log a message. Technically, writes outside of PBA could be allowed, but at this moment the precise location of PBA isn't saved, and also no known device abuses the spec in this way (at least yet).
But Coral appears to abuse this according to DKMS driver by having more than just MSI-X and PBA on a single page
https://github.com/google/gasket-driver/blob/main/src/apex_driver.c#L103-L140 -
@Teddy-Astie Is this patch already in the current kernel or do i need to manually apply it?
-
@slavox The patch I linked is not applied to current XCP-ng.
But even if it was, it would still not work due to the MSI-X/PBA/registers issue in a same page I quoted previously.
It's not a simple issue to tackle on, but upstream Xen is aware of that and it may be solved in the future (difficult to put an ETA though). -
@Teddy-Astie
I think that is the patch i tested here:
https://xcp-ng.org/forum/topic/7066/coral-tpu-pci-passthrough/26?_=1730909872550And no it made no difference...
I don't know if @andSmv has any more info? - ref. the thread above.There seems to be a lot of work in the Xen repo on MSI stuff but i could not figure out what would be relevant for the Coral.
-
Having the same issue, tried all different kernels and patches with no luck. Found this thread here and have some hope. I have both the m.2 and usb versions of the coral. So I have a pcie usb 3.2 adapter card coming tomorrow and I am going to try to pass the card through instead of the m.2 and hook up the usb version see if I can get some performance. I'll let ya guys know if it is acceptable speed.
-
Passing through the usb adapter I am getting roughly 30ms response times which is acceptable. Definitely not as fast as the m.2 version but good enough to keep me from buying a mini-pc or SBC. I would love to switch to the m.2 if someone could post here if they get a success. Just a strange side note, I couldn't get it to list in xoa and use the gui to do the passthrough. I had to manually hide it from the dom and pass the usb adapter to the vm. I couldn't figure out how to refresh the pci list and tried doing reboots to see if it would pick up the new device but no luck. If anyone knows how to refresh that list I'd be interested.
Thanks,
SFD