No VMs Will Boot - No Hosts Available Error in XOA

kagbasi-ngc

Good-day Folks,

Happy New Year to you all. So, today I came back to my air-gapped test lab (after about a week away), to find all systems in the lab without power. So I gradually powered everything up, including my 3-node XCP-ng pool. When HOST #1 came up and I confirmed that XOA had come up, I logged into it then brought up the remaining two hosts.

Once all three hosts were up, I attempted to bring up the VMs (after observing that they did not power up automatically, as they should have). That's when I noticed the "NO HOSTS AVAILABLE" error in XOA. Every VM I attempted to start, would fail with that error. After checking over the ADVANCED tab of all the VMs I was looking to start and not noticing anything out of the ordinary, I turned my attention to the STORAGE REPOSITORIES. That's when I immediately noticed that none of my NFS SRs have reconnected. This now explained why the VMs won't boot; their VDIs existed on the NFS SRs.

So I went to the respective NFS SR and clicked on the "Reconnect to All Hosts" button, and was met with the error: POST_ATTACH_SCAN_FAILED. I then went to each host, from within XOA and clicked on the "Connect" button - that's when I saw a different error message popup: "NFS Service Not Detected on Host".

Has anyone seen this before?

I've already rebooted all hosts, to no avail. And I can confirm that the NFS server is up, running, and the NFS exports are available and can be accessed from other systems within the lab environment.

Danp

Could be a routing issue. Have you checked to make sure that your switch is working?

What do these commands return when run from your XOA VM?

rpcinfo -p <IP address of NFS device>
nmap <IP address of NFS device>
showmount -e <IP address of NFS device>

kagbasi-ngc

@Danp Yes sir, I verified that all infrastructure services (including networking was up) were up and functional.

I log into XOA with AD Integration, so the fact that I could login, the SMB SRs being available (pointing to the same server that hosts the NFS Server role, by the way), and being able to access the same NFS exports from another physical Linux host, made me think that whatever is happening must be local to all three hosts.

I'm home now, but once the kids go to sleep I'll run out to the lab and run those commands and report the output. Thanks.

UPDATE:
Now that I think of it, I wonder if this same issue that affected me in this thread has read its head again. I'll check when I go in.

kagbasi-ngc

@Danp Confirmed - it was the Windows Firewall issue again. I applied the fix from the last time it happened, and things are working again.

For anyone interested - the solution is here: https://xcp-ng.org/forum/post/86828

@olivierlambert This issue highlighted to me how vague that "NO HOSTS AVAILABLE" error message is in XO. Any chance this could be improved upon?

olivierlambert

@kagbasi-ngc Sadly XAPI isn't providing more info. In that case, we need to specifically ask XAPI the reason why. Adding @julien-f in the convo so we discuss how.

kagbasi-ngc

@olivierlambert Thanks, for at least acknowledging that I'm not crazy. I'm willing to help out with testing, just let me know what's needed of me. I'm not a developer by any stretch of the imagination, but I am a seasoned SysAdmin with just enough Linux knowledge to be a little dangerous...lol.