Xen Guru

So, we've had reports on xen-devel which look a little like this.

@BlueBadger are you able to switch back to your 7950x and try booting Xen with x2apic_phys=true ? It appears that the -X processors are missing a feature in their IOMMU and Xen was getting confused when setting up interrupt handling.

https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=0d2686f6b66b4b1b3c72c3525083b0ce02830054 is at least part of the fix, but so far feedback on the mailing lists suggests it's not a complete fix.

This is way way outside of a normal-ish looking server usecase. I'm honestly surprised you've got anything to function...

To start with, you're probably booting Xen with console=vga (because that's the default). It will be handed over to dom0 too, so start by going through the bootloader configuration and making sure that neither Xen nor dom0 are trying to use the display at all.

I suspect this is the root cause of the display going periodically back to black.

@jevan223 Well, if you confirm it worked well on i440fx that probably the hypothesis is wrong. Whas it kvm-qemu virtualization?

@jevan223 This is not about the real hardware. This is about the emulated chipset offered by QEMU to HVM guests (which is the case with Windows VM)

QEMU actually emulates 2 chipset to its guests

• i440fx: basic PCI bus with CAM access

• Q35: enhanced PCI bus with ECAM access (and thus access to PCI-e capabiliites).

The problem is that Q35 is not supported by xen-dependant parts in QEMU code, so only i440fx is emulated for XEN HVM guests. We are actually working to enable Q35 in XEN, but this is a work in progress.

Well, this is a hypothesis which needs to be confirmed, but by the look of a lspci output, there's a good chance that's the reason

@logical-systems I will check which Xen version the patches are easily applied and If you want I could give you a hand (if needed) to build and install your builded XEN, so you can test if this resolve your issue.

Unfortunatly we don't have the related HW (Coral TPU) to test it by ourselves.

UPDATE: the both patches apply to xen 4.17 (tag RELEASE-4.17.0)

Yes. Some of the PCI capabilities are beyond the "standard" PCI configuration space of 256 bytes per BDF (PCI device). And unfortunatly the "enhanced" configuration access method is not provided yet (it's ongoing work) for HVM guests by XEN. It would require from QEMU (xen related part) the chipset emulation which offers an access to such method, such as Q35.

Very probably, windows drivers for these devices are not happy to not access these fields, so this is potentially the reason of malfunctionning for these devices.

The good way to confirm this would be to try to passthrough these devices to Linux guests, so we could possibly add some extended traces. And possibly passthrough these devices to PVH Linux guest and see how they are handled (PVH guest do not use QEMU for PCI bus emulation)

@jevan223 can you please provide a lspci -vvv output (in dom0) ?

Hmm, at first glance looks to me as a real use case for q35 chipset emulation support on XEN ?

Hello, sorry for late response (just discovered the topic)

With regards of Marek patches, I'm actually think it can worth a try (at least the patch seems to treat the problem where MSI-x PBA page is shared with other regs of the device), but there's some cons too:

• the patches are quite new (doesn't seems to be integrated yet).
• the patches can be applied to more recent Xen (not XCP-ng Xen), and even we could probably backport them, it potentially will require some significant work
• we are not 100% sure it's the issue (or the only issue)

So If this is a must have, we can go and do some digging to make it work (but still in the scope of "exeperimental" platform, not the production platform)