Soo..... the problem was the NIC.
I passed the NIC through to a Windows vm and it worked fine.
I passed the NIC through to a Linux VM and it was UNCLAIMED and unusable.
Checking the event logs I found the following entries :
igb 0000:00:05.0: The NVM Checksum Is Not Valid
igb 0000:00:05.0: probe with driver igb failed with error -5
For some reason Linux seems to no like that check sum failing and Windows seems to not care.
Digging around I found some people trying to re-flash firmware to resolve check sum errors and that was a dead end.
Other seems to get things working by changing some boot firmware settings.
Attaching it back to the Windows machine I ran the intel boot utility and disabled PXE boot.
BOOTUTILW64E.EXE -BOOTENABLE- DISABLED -ALL
To my surprise, I reattached to the Linux VM and was able to see and use the NIC!
Disabled the PCI pass through for the NIC and did a PIFs refresh and I now can see the device in as ETH0.
Use it on a test VM and performance is what it should be.
I suspect the boot util properly set the parameters and and updated the check sum info making Linux happy.
I am also sure this is a long way to resolve the issue and there would have been a much faster process but it was bit of a random moment of stumbling on a solution.
Posting here in hopes that it helps somebody else some day.