The grub setup is rather simple and not very flexible. There's just one file to modify, as you found out (/etc/grub.cfg in BIOS mode, /etc/grub-efi.cfg in EFI mode, both being symbolic links to the actual file location).

You can add an entry to it, but there's a small chance this doesn't play well with scripts from either XenServer or ourselves which may want to update the file and get confused. It's usually better to just modify the existing entries, ideally using /opt/xensource/libexec/xen-cmdline, so that the file structure remains unchanged.

Ah that makes sense then 🙂

You need a fully up to date XO (check that first). If you are on XOA Free, you might need a trial (send me a private chat message with your email registered to this XOA, so I can unlock you the free trial). Alternatively, you can also use XO from the sources 🙂

Just updating for anyone that has the same issue, we ended up just rebooting the master and like @JamuelStarkey said everything just automatically fell in place. Did have to exit maintenance mode on the master and replug the PBD but everything else went back to normal immediately.

Still frustrating to experience and would really love to know what caused this. If there's any logs I can pull to figure this out do let me know @olivierlambert

Perfect! 🍾

So I managed to get the VM to see both drives.

In dmesg I can see something like this

[ 1.160149] ahci 0000:00:08.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x20 impl SATA mode [ 1.161043] ahci 0000:00:08.0: flags: 64bit ncq sntf ilck led clo only pmp fbs pio slum part [ 1.187888] ahci 0000:00:09.0: SSS flag set, parallel bus scan disabled [ 2.186211] ahci 0000:00:09.0: controller reset failed (0xffffffff) [ 2.187307] ahci: probe of 0000:00:09.0 failed with error -5

So I removed the pci and did a rescan with

root@mars-test-uefi:~# echo "1" > /sys/bus/pci/devices/0000\:00\:09.0/remove root@mars-test-uefi:~# echo "1" > /sys/bus/pci/rescan

Now when I checked dmesg it shows up properly

[ 795.538214] ahci 0000:00:09.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x4 impl SATA mode [ 795.538219] ahci 0000:00:09.0: flags: 64bit ncq sntf ilck led clo only pmp fbs pio slum part

fdisk -l also shows up both drives now too

root@mars-test-uefi:~# fdisk -l Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors Disk model: ST8000DM004-2U91 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/sdb: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors Disk model: WDC WD80EFZZ-68B Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes

It seems like on boot it tries to find the device but it fails to, but on a remove/rescan it seems like its fine?

The question now becomes, how do I make this work ON boot.

Thanks a lot for your feedback!

Really, you should because it would have prevent you this situation in the first place, but also some other future questions or problem you might have That's one of the few capital rules in XCP-ng world. Your master must ALWAYS be more recent than the slaves (or at the same level obviously), otherwise slaves won't be able to connect. I think you really need some assistance on best practices and such, please contact us so we can assist more generally. If this basic requirement is not understand, you might have other/deeper issues.

Any of the host: if they are in the same pool, that's logical. Only the master is needed to be reach. XAPI will probably be in "Starting state" as long as all SR aren't plugged. If you have the SR on a VM on another host than the master, reboot the master only, you should be able to connect sooner Alternatively, check https://docs.xcp-ng.org/troubleshooting/

On this one VM yeah, all other VM's have been fine with the Citrix tools. I am not sure if the tools change anything on the VM level once installed, as the problem did continue after attaching a fresh blank virtual disk to the problem VM.

@tuxen

Absolutely no idea how to do this in Windows. I looked for any MTU setting but couldn't find any. This is not a viable workaround for me, maybe it would be useful to pin the issue to the xen PV driver, maybe I'll do some more testing on spare hardware. I read this, but I don't know how to test it. I didn't have any manual MTUs set so I don't know what values were before the update.

What most definitely fixed the issue for me was using PCIe passthrough for the WAN interface. I used a 10 GbE NIC. It uses the ix driver (ix0) so IDK if this is related. Somehow PPPoE + WG + Windows Client on the virtual interface (Xen PV driver) in OPNsense produces this issue.
At the moment I am happy with this mitigation.

I'm a little spread thin with free time at the moment. Anyone care to test this further?

How long have you waited for it? Like is it truly stuck or maybe just really slow to boot?