From what I understand, the OS-to-iDRAC pass-through is a direct USB link between the host and the BMC, designed for the hypervisor host itself to talk to the iDRAC, not something that would naturally bridge through Xen's networking layer into a VM.
That might be why the PIF shows up in XCP-ng but the VM still can't reach 169.254.0.1; the bare metal case works because the OS sits directly on the hardware with no virtualisation layer in between. If you need the VM to have direct BMC access, USB passthrough might be worth exploring; XCP-ng does
support passing USB devices through to VMs (https://docs.xcp-ng.org/compute#usb-passthrough), though I'm not certain the iDRAC virtual USB NIC would show up as a passthrough-able device.
Might be worth a mention to @Team-Hypervisor-Kernel to find out; others here will know more than me.