Coral TPU PCI Passthrough

logical.systems

@olivierlambert I saw that post in my initial search but it doesn't look like the OP replied with the PCI dump. Is there any hope for a workaround?

olivierlambert

You can continue here to provide details, maybe we'll see something obvious

logical.systems

@olivierlambert Aside from the dump in my original post would you like me to run any additional commands to gather more data?

olivierlambert

@andSmv will take a look when he's around

olivierlambert

What's your exact model of Coral TPU by the way?

olivierlambert

For reference: https://github.com/google-coral/edgetpu/issues/343#issuecomment-1287251821

dakota created this issue in google-coral/edgetpu

open Apex failing with error -110 (No /dev/apex_0) #343

logical.systems

@olivierlambert M.2 Accelerator B+M Key
https://coral.ai/products/m2-accelerator-bm#description

olivierlambert

So I've heard Qubes OS people did some patches to workaround the broken PCI spec for the device, I need to ask around more details about this.

olivierlambert

@andSmv I discussed with Marek from Qubes, he told me that might be relevant (or not): https://lore.kernel.org/xen-devel/20221114192100.1539267-2-marmarek@invisiblethingslab.com/

What do you think?

andSmv

Hello, sorry for late response (just discovered the topic)

With regards of Marek patches, I'm actually think it can worth a try (at least the patch seems to treat the problem where MSI-x PBA page is shared with other regs of the device), but there's some cons too:

the patches are quite new (doesn't seems to be integrated yet).
the patches can be applied to more recent Xen (not XCP-ng Xen), and even we could probably backport them, it potentially will require some significant work
we are not 100% sure it's the issue (or the only issue)

So If this is a must have, we can go and do some digging to make it work (but still in the scope of "exeperimental" platform, not the production platform)

olivierlambert

We could probably try on a non-XCP-ng platform with a very recent "vanilla" Xen (+Marek patches) and see if it's fixed. If it is, then we could think about a potential backport when 8.3 will include a more recent Xen version

andSmv

@logical-systems I will check which Xen version the patches are easily applied and If you want I could give you a hand (if needed) to build and install your builded XEN, so you can test if this resolve your issue.

Unfortunatly we don't have the related HW (Coral TPU) to test it by ourselves.

UPDATE: the both patches apply to xen 4.17 (tag RELEASE-4.17.0)