Can't start or shutdown VM's after storage was powered off

dgleba

This post will be less narrow and specific than we would all like - But it is my situation.

I have :

Pool of 3 XCP-ng v8.0. on 3 Dell T5500's. Recently converted from ESXi - 3 months ago. Approx 50 VM's.
2 separate FreeNAS storage computers.

I have used XenServer 6 for years in a business environment.

I Left HA and VM's running.
I Shutdown one FreeNAS box due to disk error to investigate.

Problem: (trio of servers)

After turning FreeNAS back on, I could not do much with Xcp-ng.

I can ssh.
I can now use the XCP-ng center.
Most VM's are starting, but not started.

I tried to shut down VM's using command line xe commands per https://support.citrix.com/article/CTX220777
I can not shut most of them down, I can't start them.
I could not do much with the system.

I used commands "like" these and others.

 96  xe pool-list
101  /etc/init.d/xapissl restart
104  service xapi status
105  service xapi restart
107  netstat -an | grep 172.16.2.30
111  cat network.conf
112  cat xapi-ssl.pem
113  hostname -f
116  xe host-list params=uuid,name-label,host-metrics-live
117  xe host-is-in-emergency-mode
121  xe host-emergency-ha-disable --force
122  xe sr-list
123  xe host-list params=uuid,name-label,host-metrics-live

Test attempt:

Not being able to use any particular XCP-ng server, I powered them all off. Then I set up a 4th fresh XCP-ng on another computer - Dell R620.

Problem: (Dell R620)

mounting exiting SR's takes much longer than before. Instead of 10 seconds, it's two minutes for example.
The windows 2012 w/exchange 2013 VM takes hours to start and exchange services don't start. The network and sharing dialog won't load.

I realize this sounds like a soap opera, but it is my situation. It probably sounds like a country song, I lost my job, then my car, then my wife, then my.....

I will post the last error messages from the XCP-ng log later today.

Many thanks for your patience with me.

Update 12:50pm:

I edited the post above in many places. Read it again if you already read it.
There are no errors in the XCP-ng center log.

Update 7:50pm:

I clarified the post title and that the original three servers are powered off. The R620 is the only one running now.

Forza

Did you restart the entire pool? Sounds like you might have NFS issues like stale mounts lingering. This could make things very slow on the XCP-ng side. Do you export your NFS with fsid option? You should.

dgleba

@S-Pam said in Can't start or shutdown VM's after power outage:

export your NFS with fsid option

I am not certain what you mean by "export your NFS with fsid option"?
Can you explain?

dgleba

Where would the stale mounts be? .. in XCP-ng? or in Freenas?

Forza

In FreeNAS. exports file you should have fsid=123, with a unique number for each exported path. If not it can cause a problem if the nfs server is rebooted while the pool is up.

https://linux.die.net/man/5/exports
https://www.suse.com/support/kb/doc/?id=000017897

The stale mounts are in xcp-ng.
But this is just a theory.

During the power outage. Did all xcp-ng hosts and the FreeNAS as boxes go down? How did you boot up? You wrote you set up fresh servers?