I have found a work round for this, another problem i found is the host 2&3 server have wrong credentials setup so can't access them locally, so a rebuild on them will be starting shortly.
Thanks @olivierlambert i looked into the networking a bit more, I was able to get successful pings between the servers and more looking into the networking on these 2 servers I had an IP address but was showing disconnected, so i started to look at forcefully removing the unresponsive hosts and found the following article.
-
I ran this from the master (still working) to find the list of hosts.
xe host-list -
Get a list of all the VMs that are resident on the unresponsive host(s) to get their uuid's.
xe vm-list resident-on=aaaaaaaaa-bbbb-cccc-ddddddddddd
3.Force powerstate reset for each of the VMs on the host(s).
xe vm-reset-powerstate uuid=eeeeeee-dddd-cccc-bbbb-aaaaaaaaaaaa --force
- Force the removing of the host(s).
xe host-forget uuid=aaaaaaaaa-bbbb-cccc-ddddddddddd
This has now allowed me to spin up the VMs on the working host while I get the other rebuilt.
Thanks you @olivierlambert and @Gheppy for advice and pointers for going some solution for this.