Rolling Updates Failed
-
Greetings!
Rolling Updates Failed 3x for me this morning. I was attempting to apply updates announced here:
https://xcp-ng.org/forum/topic/9964/xcp-ng-8-3-updates-announcements-and-testing/350
I didn't want to clutter that thread as my case may just be related to how my test system is set up.
Setup:
2x Hosts (up to date prior to the release this morning)
XOA - up to date on Latest channel, Premium Trial active
I have 2 FC attached, multipath SR's where my VDI's live
My ISO store is an SR on local storage on one of my Hosts (the Master)Related Question: why can't I create an ISO store on FC SAN disk? It looks like NFS/SMB/Local are the only options.
First Attempt at Rolling Pool Update:
- Failed immediately, unable to evac Master Host
- The failure was mostly silent. There was a raw log in Tasks that wasn't very useful (to me anyways) that it cannot evac host.
- I believe this was due to my testing Self-Service yesterday, I created 3 VM's and attached a RHEL ISO and booted into the installer. Since I was just testing the ability to create VM's and permissions, I left them in this state. Since the ISO SR is attached to this Host, I am not surprised that the migration failed, but it seems like the Rolling Update should have been able to detect this before starting.
- I was able to Migrate these 3 VM's manually without unmounting the ISO. I didn't look before migrating, but the ISO is no longer mounted after the migrations.
Second Attempt:
- Master Host evac proceeded, but failed to migrate 2 VM's after 17min. It looks like I left Guest Tools ISO mounted on these.
- Unmounted tools and did not Migrate VM's, but proceeded to attempt 3
Third Attempt:
- Master Host completed the evac of the last 2 VM's, applied updates and rebooted. Yay!
- After the Host came back up, Host2 evac started immediately
- Evac eventually failed on 2 VM's. One was one of the VM's that had the RHEL ISO mounted in attempt 1, but no longer had it mounted (according to XOA), The other VM was one of the 2 VM's that failed in attempt 2. Again, there was no longer anything showing as mounted. Not sure why these 2 failed to Migrate back to Host1.
Since the Master Host was updated, I could no longer perform a Rolling Update. I migrated the 2 VM's (no changes to the guests - manual migrated completed normally), disabled HA on the Pool, and applied updates and then Rebooted Host. Host came back up normally and I was able to enable HA and move VM's to it.
So, my cluster is back to normal, but I wanted to put my experience out there in case it is useful. Working with various vendors over the years, I have found that I am VERY good at finding corner cases.
Overall, it seems like the Rolling Update functionality is a bit fragile, at least in my case. Maybe my ISO setup is unusual? It seems problematic that someone using Self-Service could leave a VM in a state that would cause headaches for Updates without the Updater detecting it and letting the XOA Admin know what the problem is. If I had hundreds of VM's created by multiple other people, it would have been hard to figure out what the issue was.
Let me know if I can provide any other useful info.
Thanks!