That leftover NIC is probably a stale PIF that XAPI is still holding, even though the card is physically gone, and the Refresh button re-scans rather than removing it.
The docs have proper "remove a physical NIC" steps that end in forgetting the old PIF with xe pif-forget: https://docs.xcp-ng.org/networking/#remove-a-physical-nic.
For the GPU side, the PCI passthrough flow (hiding the device from dom0, then assigning it) is here: https://docs.xcp-ng.org/compute#detaching-a-pci-device.
If the GPU still won't show up as assignable once the stale PIF is gone, it might be worth a mention to @Team-Hypervisor-Kernel.
