How many guest IRQs is too many?
-
Ran into an interesting issue today. We do a lot of hardware passthrough for some of our XCP-ng configurations. One of our use cases is a guest running a networking tool we built that we PCI passthrough NICs to. Typically we have been using about 6 to 8 NICs but today we put together a configuration that required us to passthrough 17 NICs.
Because we do a lot of passthrough configs our standard config is to set the "extra_guest_irqs=128" parameter on all of our installs. Well long story short, I ran into some MSI-X interrupt errors on the guest with the 17 passthrough NICs and the RHEL guest couldn't initialize some of my NICs. I took a look at /proc/interrupts and with the extra 128 IRQ setting I was at 192 IRQ and they seemed to be all allocated to the NICs that were functional. It looked like each NIC needed 9 IRQ, so I calculated I needed at least 40 more and then bumped the "extra_guest_irqs" to 192 for a total of 256. That solved my issues and everything is working great now.
So my question is, what is the downside to allocating too many guest IRQ and is there an upper limit before I start seeing issues? Also that is the only guest VM on this system with that many passthrough devices, is there a way to set the IRQ on a per guest basis?
There isn't really a whole lot of documentation on how to best size guest IRQ and how to determine what settings i may need when passing through multiple PCI devices. If anyone has any insights or could point me to some documentation I would be very appreciative! Thanks!
-
Hi,
Let me ask @Team-Hypervisor-Kernel
