Host not running VMs after update

Tim-PT

I've made the mistake of restarting an unresponsive host, its the pool master on a single host pool running 8.3 RC1

I've tried to update it so I could add another host to the pool, but after the updates I lost all contact via XO. The VMs were still running, until I restarted the host - now I can't get any VMs up and running again.

The host comes back up and I can ssh to it, but things aren't running right, If I try to view VMs via xsconsole I am see the error: "(''NoneType' object has no attribute 'xenapi'')", but even this is inconsistent, I get the error but then the VM list appears.

However staying on the All VMs section alternates between showing me a list of VMs and telling me there are no VMs present.

Checking the VM that should be running XP tells me that it is running, but it isn't.

I'm a bit stuck - can anyone point me in the right direction please?

olivierlambert

Your XAPI seems stopped. Check the usual logs provided in our troubleshoot section of our official doc.

Tim-PT

@olivierlambert Thanks very much for the reply. I'll check there now.

I may need to start the trial of your support if I can't get this up

Tim-PT

@Tim-PT
xapi is not happy

olivierlambert

Try to check if you have made the updates correctly (yum update) and then if it's the case, try a xe-toolstack-restart command.

Tim-PT

@olivierlambert yum update is telling me that no packages are marked for update, xe-toolstack-restart hasn't made a difference. At this stage I've restarted the host multiple times too and no difference

DNS was originally pointing at a VM which is no longer available, I've changed this on the host now but I'm getting a message about the mirror being excluded:

Tim-PT

@Tim-PT
OK - I'm stuck and need to get this moving - I've got a second host that I was trying to add to the pool, is there any way to get the VMs that are on an NFS share working on the new host?

Is that likely to be any easier/quicker to get things up and running again than trying with this host that hasn't updated cleanly?

flakpyro

@Tim-PT Maybe worth trying to do an in place upgrade to 8.3 RC2 or final via ISO and see if that helps?

Tim-PT

@flakpyro
Thanks for your suggestion. I have tried with the release of 8.3 but I'm having just as many problems with that and XAPI is still bouncing up and down

stormi

@Tim-PT Please share the contents of xensource.log for a timeframe that contains a XAPI startup and failure.

Tim-PT

@stormi
Thanks for getting back to me.

Here's the last 500 lines...xensource.txt

stormi

@Tim-PT That's what I feared. Your XAPI database still contains a reference to certificate sdn-controller-ca.pem but the file was deleted on disk (manually?), and XAPI chokes on that. That's an issue I already reported upstream and they're working on a fix.

Thankfully, there's a way to fix it.

What's the output of rpm -q xapi-core?

Tim-PT

@stormi
Thanks for this, I've not deleted anything manually (certainly not knowingly!), I'm pleased to hear there's a way to resolve it.

The output is:
xapi-core-24.19.2-1.4.xcpng8.3.x86_64

stormi

So we need to downgrade XAPI so that it can start and we can fix the root cause, then update it again.

This should work, but tell if it doesn't go as expected:

# Downgrade XAPI. It's made of many components so it's a bit involved.
export VER_FROM=$(rpm -q xapi-core --qf "%{version}-%{release}")
export ESCAPED_VER_FROM=$(echo "$VER_FROM" | sed 's/[.[\*^$]/\\&/g')
export VER_TO=24.19.2-1.2.xcpng8.3
export ESCAPED_VER_TO=$(echo "$VER_TO" | sed 's/[.[\*^$]/\\&/g')
rpm -qa | grep "$VER_FROM" | sed "s/$ESCAPED_VER_FROM/$VER_TO/" | xargs yum downgrade -y

# restart the toolstack so that XAPI works again
xe-toolstack-restart

# delete the problematic certificate from XAPI. To do so we must first create a dummy cert so that `xe` will accept to remove it from database (it fails if it can't find the file for which you're trying to remove the database entry)
touch /etc/stunnel/certs/sdn-controller-ca.pem
xe pool-certificate-uninstall name=sdn-controller-ca.pem

# update again the downgraded packages
rpm -qa | grep "$VER_TO" | sed "s/$ESCAPED_VER_TO/$VER_FROM/" | xargs yum update-to -y

# restart the toolstack again. XAPI's startup should not loop anymore.
xe-toolstack-restart

Tim-PT

@stormi
Hi - thanks very much for the detailed steps. That's definitely made progress.

I think the emergency network reset that was recommended yesterday will need resolving, but xsconsole has settled down already (I'll check the logs shortly)

Tim-PT

We are now live again with all the VMs - which is a relief.

The above steps from @stormi were great, the one thing I notced was the /etc/stunnel/certs directory didn't exist. I created that manually and everything else seemed happy.

I did have to mess about a bit with the fibre NICs, but I've repeated the previous renaming and that's now running fine.

Thanks very much for all the help - I appreciate it.