Now we have 76 VMs running on a 3 host pool. Each server has 320 GB of RAM. Our scenario doesn't need big CPU resources so everything works fine.

anyone has any other ideas ?? I seem to be lost at what to do.

@wilsonqanda Yes, BIOS as well, assuming I am remembering right haha. If you just pick whatever is closest and go with it I doubt you'll run into issues, if you do just make a post here and I'm sure someone will be willing to help out or work on a new template or something.

@wilsonqanda Downgrading the EKD2 package fix it for now, as posted: yum downgrade edk2-20180522git4b8552d-1.5.1.xcpng8.3

The grub setup is rather simple and not very flexible. There's just one file to modify, as you found out (/etc/grub.cfg in BIOS mode, /etc/grub-efi.cfg in EFI mode, both being symbolic links to the actual file location). You can add an entry to it, but there's a small chance this doesn't play well with scripts from either XenServer or ourselves which may want to update the file and get confused. It's usually better to just modify the existing entries, ideally using /opt/xensource/libexec/xen-cmdline, so that the file structure remains unchanged.

Ah that makes sense then

You need a fully up to date XO (check that first). If you are on XOA Free, you might need a trial (send me a private chat message with your email registered to this XOA, so I can unlock you the free trial). Alternatively, you can also use XO from the sources

Just updating for anyone that has the same issue, we ended up just rebooting the master and like @JamuelStarkey said everything just automatically fell in place. Did have to exit maintenance mode on the master and replug the PBD but everything else went back to normal immediately. Still frustrating to experience and would really love to know what caused this. If there's any logs I can pull to figure this out do let me know @olivierlambert

Perfect!

So I managed to get the VM to see both drives. In dmesg I can see something like this [ 1.160149] ahci 0000:00:08.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x20 impl SATA mode [ 1.161043] ahci 0000:00:08.0: flags: 64bit ncq sntf ilck led clo only pmp fbs pio slum part [ 1.187888] ahci 0000:00:09.0: SSS flag set, parallel bus scan disabled [ 2.186211] ahci 0000:00:09.0: controller reset failed (0xffffffff) [ 2.187307] ahci: probe of 0000:00:09.0 failed with error -5 So I removed the pci and did a rescan with root@mars-test-uefi:~# echo "1" > /sys/bus/pci/devices/0000\:00\:09.0/remove root@mars-test-uefi:~# echo "1" > /sys/bus/pci/rescan Now when I checked dmesg it shows up properly [ 795.538214] ahci 0000:00:09.0: AHCI 0001.0301 32 slots 1 ports 6 Gbps 0x4 impl SATA mode [ 795.538219] ahci 0000:00:09.0: flags: 64bit ncq sntf ilck led clo only pmp fbs pio slum part fdisk -l also shows up both drives now too root@mars-test-uefi:~# fdisk -l Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors Disk model: ST8000DM004-2U91 Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes Disk /dev/sdb: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors Disk model: WDC WD80EFZZ-68B Units: sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 4096 bytes I/O size (minimum/optimal): 4096 bytes / 4096 bytes It seems like on boot it tries to find the device but it fails to, but on a remove/rescan it seems like its fine? The question now becomes, how do I make this work ON boot.

Thanks a lot for your feedback!

Really, you should because it would have prevent you this situation in the first place, but also some other future questions or problem you might have That's one of the few capital rules in XCP-ng world. Your master must ALWAYS be more recent than the slaves (or at the same level obviously), otherwise slaves won't be able to connect. I think you really need some assistance on best practices and such, please contact us so we can assist more generally. If this basic requirement is not understand, you might have other/deeper issues.

Any of the host: if they are in the same pool, that's logical. Only the master is needed to be reach. XAPI will probably be in "Starting state" as long as all SR aren't plugged. If you have the SR on a VM on another host than the master, reboot the master only, you should be able to connect sooner Alternatively, check https://docs.xcp-ng.org/troubleshooting/

On this one VM yeah, all other VM's have been fine with the Citrix tools. I am not sure if the tools change anything on the VM level once installed, as the problem did continue after attaching a fresh blank virtual disk to the problem VM.

How long have you waited for it? Like is it truly stuck or maybe just really slow to boot?