Soo..... the problem was the NIC.

I passed the NIC through to a Windows vm and it worked fine.
I passed the NIC through to a Linux VM and it was UNCLAIMED and unusable.

Checking the event logs I found the following entries :

igb 0000:00:05.0: The NVM Checksum Is Not Valid
igb 0000:00:05.0: probe with driver igb failed with error -5

For some reason Linux seems to no like that check sum failing and Windows seems to not care.

Digging around I found some people trying to re-flash firmware to resolve check sum errors and that was a dead end.
Other seems to get things working by changing some boot firmware settings.

Attaching it back to the Windows machine I ran the intel boot utility and disabled PXE boot.

BOOTUTILW64E.EXE -BOOTENABLE- DISABLED -ALL

To my surprise, I reattached to the Linux VM and was able to see and use the NIC!

Disabled the PCI pass through for the NIC and did a PIFs refresh and I now can see the device in as ETH0.

Use it on a test VM and performance is what it should be.

I suspect the boot util properly set the parameters and and updated the check sum info making Linux happy.

I am also sure this is a long way to resolve the issue and there would have been a much faster process but it was bit of a random moment of stumbling on a solution.

Posting here in hopes that it helps somebody else some day.