Rolling Pool Update Failed
-
Hi,
I just tried performing a Rolling Pool Update which has 2 hosts and in the XO tasks, I'm seeing the following failed entries (see below).
Not sure how best to proceed. All running VMs have been evacuated from pool master and are running on the second host. XO now shows the pool master is "Disabled":
Any recommendations on what I should do next?
Here are the XO task that failed during the rolling update:
Rolling pool update 2024-08-12 20:45
id "0lzrpb9pj" properties poolId "e52fca9f-4a2f-b24c-8c37-b7891946a82a" poolName "Production" name "Rolling pool update" userId "6356451e-8fab-4588-a2f0-682b34f8f684" start 1723509930727 status "failure" updatedAt 1723510634068 tasks 0 id "hsb0p4ig9nc" properties name "Listing missing patches" total 2 progress 100 done 2 start 1723509930730 status "success" tasks 0 id "swldkb837u" properties name "Listing missing patches for host 8d6e2b03-54c1-4523-b45b-d572a268f2cc" hostId "8d6e2b03-54c1-4523-b45b-d572a268f2cc" hostName "XCP25" start 1723509930731 status "success" end 1723509930731 1 id "ui4qgc07lk" properties name "Listing missing patches for host a2c8f50f-0555-44e0-bcd4-1454b6e407a1" hostId "a2c8f50f-0555-44e0-bcd4-1454b6e407a1" hostName "XCP35" start 1723509930731 status "success" end 1723509930731 end 1723509930731 1 id "m2f4j80sayd" properties name "Updating and rebooting" start 1723509930732 status "failure" tasks 0 id "kp5is2v7nm" properties name "Restarting hosts" total 2 progress 0 done 0 start 1723509930945 status "failure" tasks 0 id "szi12dqrq" properties name "Restarting host 8d6e2b03…-4523-b45b-d572a268f2cc" hostId "8d6e2b03-54c1-4523-b45b-d572a268f2cc" hostName "XCP25" start 1723509930946 status "failure" tasks 0 {…} 1 {…} end 1723510634067 result message "Text data outside of roo…nColumn: 21393\nChar: }" name "Error" stack "Error: Text data outside…-mixins/api.mjs:366:20)" end 1723510634067 result message "Text data outside of root node.\nLine: 0\nColumn: 21393\nChar: }" name "Error" stack "Error: Text data outside of root node.\nLine: 0\nColumn: 21393\nChar: }\n at error (/opt/xen-orchestra/node_modules/sax/lib/sax.js:652:10)\n at strictFail (/opt/xen-orchestra/node_modules/sax/lib/sax.js:678:7)\n at SAXParser.write (/opt/xen-orchestra/node_modules/sax/lib/sax.js:1036:15)\n at parseXmlTree (/opt/xen-orchestra/@vates/xml/parse.js:20:10)\n at Xapi._xcpUpdate (file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/patching.mjs:314:58)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/patching.mjs:545:17\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/pool.mjs:113:17\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/pool.mjs:99:13\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at Xapi.rollingPoolReboot (file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/pool.mjs:90:5)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/patching.mjs:530:7\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at XenServers.rollingPoolUpdate (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/xen-servers.mjs:689:5)\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:366:20)" end 1723510634067 result message "Text data outside of root node.\nLine: 0\nColumn: 21393\nChar: }" name "Error" stack "Error: Text data outside of root node.\nLine: 0\nColumn: 21393\nChar: }\n at error (/opt/xen-orchestra/node_modules/sax/lib/sax.js:652:10)\n at strictFail (/opt/xen-orchestra/node_modules/sax/lib/sax.js:678:7)\n at SAXParser.write (/opt/xen-orchestra/node_modules/sax/lib/sax.js:1036:15)\n at parseXmlTree (/opt/xen-orchestra/@vates/xml/parse.js:20:10)\n at Xapi._xcpUpdate (file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/patching.mjs:314:58)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/patching.mjs:545:17\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/pool.mjs:113:17\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/pool.mjs:99:13\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at Xapi.rollingPoolReboot (file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/pool.mjs:90:5)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/patching.mjs:530:7\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at XenServers.rollingPoolUpdate (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/xen-servers.mjs:689:5)\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:366:20)" end 1723510634068 result message "Text data outside of root node.\nLine: 0\nColumn: 21393\nChar: }" name "Error" stack "Error: Text data outside of root node.\nLine: 0\nColumn: 21393\nChar: }\n at error (/opt/xen-orchestra/node_modules/sax/lib/sax.js:652:10)\n at strictFail (/opt/xen-orchestra/node_modules/sax/lib/sax.js:678:7)\n at SAXParser.write (/opt/xen-orchestra/node_modules/sax/lib/sax.js:1036:15)\n at parseXmlTree (/opt/xen-orchestra/@vates/xml/parse.js:20:10)\n at Xapi._xcpUpdate (file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/patching.mjs:314:58)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/patching.mjs:545:17\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/pool.mjs:113:17\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/pool.mjs:99:13\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at Xapi.rollingPoolReboot (file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/pool.mjs:90:5)\n at file:///opt/xen-orchestra/packages/xo-server/src/xapi/mixins/patching.mjs:530:7\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at XenServers.rollingPoolUpdate (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/xen-servers.mjs:689:5)\n at Task.runInside (/opt/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/opt/xen-orchestra/@vates/task/index.js:158:20)\n at Api.#callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:366:20)"Logged into pool master and it seems XO installed all of the pending updates:
[21:14 XCP25 ~]# yum update Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org No packages marked for updateAnd /var/log/yum.log shows the following packages were installed:
Aug 12 20:55:06 Updated: xen-libs-4.13.5-9.40.2.xcpng8.2.x86_64 Aug 12 20:55:06 Updated: xcp-ng-release-presets-8.2.1-11.x86_64 Aug 12 20:55:06 Updated: openssh-7.4p1-23.2.1.xcpng8.2.x86_64 Aug 12 20:55:06 Updated: message-switch-1.23.2-17.1.xcpng8.2.x86_64 Aug 12 20:55:07 Updated: xen-hypervisor-4.13.5-9.40.2.xcpng8.2.x86_64 Aug 12 20:55:07 Updated: xen-dom0-libs-4.13.5-9.40.2.xcpng8.2.x86_64 Aug 12 20:55:07 Updated: xen-tools-4.13.5-9.40.2.xcpng8.2.x86_64 Aug 12 20:55:07 Updated: xen-dom0-tools-4.13.5-9.40.2.xcpng8.2.x86_64 Aug 12 20:55:07 Updated: xenopsd-0.150.19-3.1.xcpng8.2.x86_64 Aug 12 20:55:11 Updated: 2:qemu-4.2.1-4.6.4.1.xcpng8.2.x86_64 Aug 12 20:55:11 Updated: forkexecd-1.18.3-10.1.xcpng8.2.x86_64 Aug 12 20:55:12 Updated: vhd-tool-0.43.0-18.1.xcpng8.2.x86_64 Aug 12 20:55:12 Updated: libcurl-8.6.0-2.1.xcpng8.2.x86_64 Aug 12 20:55:12 Updated: curl-8.6.0-2.1.xcpng8.2.x86_64 Aug 12 20:55:15 Updated: xapi-core-1.249.36-1.2.xcpng8.2.x86_64 Aug 12 20:55:16 Updated: xenopsd-xc-0.150.19-3.1.xcpng8.2.x86_64 Aug 12 20:55:16 Updated: xenopsd-cli-0.150.19-3.1.xcpng8.2.x86_64 Aug 12 20:55:16 Updated: squeezed-0.27.0-18.1.xcpng8.2.x86_64 Aug 12 20:55:17 Updated: xapi-tests-1.249.36-1.2.xcpng8.2.x86_64 Aug 12 20:55:17 Updated: gpumon-0.18.0-18.1.xcpng8.2.x86_64 Aug 12 20:55:17 Updated: xcp-rrdd-1.33.4-4.1.xcpng8.2.x86_64 Aug 12 20:55:18 Updated: rrdd-plugins-1.10.9-12.1.xcpng8.2.x86_64 Aug 12 20:55:18 Updated: openssh-server-7.4p1-23.2.1.xcpng8.2.x86_64 Aug 12 20:55:19 Updated: openssh-clients-7.4p1-23.2.1.xcpng8.2.x86_64 Aug 12 20:55:19 Updated: xcp-ng-release-8.2.1-11.x86_64 Aug 12 20:55:22 Updated: xcp-ng-release-config-8.2.1-11.x86_64 Aug 12 20:55:23 Updated: sm-2.30.8-12.1.xcpng8.2.x86_64 Aug 12 20:55:23 Updated: sm-cli-0.23.0-61.1.xcpng8.2.x86_64 Aug 12 20:55:23 Updated: varstored-guard-0.6.2-15.xcpng8.2.x86_64 Aug 12 20:55:23 Updated: xapi-storage-11.19.0_sxm2-17.xcpng8.2.x86_64 Aug 12 20:55:24 Updated: xcp-networkd-0.56.2-15.xcpng8.2.x86_64 Aug 12 20:55:29 Updated: linux-firmware-20190314-11.1.xcpng8.2.noarch Aug 12 20:55:29 Updated: xapi-xe-1.249.36-1.2.xcpng8.2.x86_64 Aug 12 20:55:29 Updated: sudo-1.9.15-2.1.xcpng8.2.x86_64 Aug 12 20:55:29 Updated: xapi-nbd-1.11.0-17.1.xcpng8.2.x86_64 Aug 12 20:55:30 Updated: xapi-storage-script-0.34.1-16.1.xcpng8.2.x86_64 Aug 12 20:55:30 Updated: 2:microcode_ctl-2.1-26.xs29.2.xcpng8.2.x86_64 Aug 12 20:55:31 Updated: rrd2csv-1.2.6-15.1.xcpng8.2.x86_64 Aug 12 20:55:31 Updated: tzdata-2024a-1.el7.noarch Aug 12 20:55:31 Updated: xsconsole-10.1.13-1.2.xcpng8.2.x86_64 Aug 12 20:55:31 Updated: sm-rawhba-2.30.8-12.1.xcpng8.2.x86_64 Aug 12 20:55:31 Updated: wsproxy-1.12.0-19.xcpng8.2.x86_64Thank You,
SW
-
Hi,
In order to assist, we need more details on your setup. XOA or XO sources? Fully up to date? Etc.
-
Please make sure that you are running from the latest sources as this issue was fixed in XOA 5.95.1 (commit f5a49b6).
-
@stevewest15 Looks to me like all your hosts might not have the same updates/patches applied. I'd check to make sure they are all up-to-date and the same on all your hosts.
-
Thank you all! I was on an older release of XOCE. I haven't wanted to update as the last few times I did, it broke the nightly backups. So stuck with an older version of XOCE that seems to have been stable in our environment.
I updated XOCE to latest build:
Xen Orchestra, commit 32228
Master, commit 32228I was able to manually update the second host (by running yum update). Then restarted the pool master and now I'm trying to vacate VMs from second host back to master so I can reboot the second host.
However the live migration seems to be taking much longer from the second host back to master. All VMs use shared storage connected via NFS to this pool. I was thinking it should just take a few minutes to do a live migrate but so far it's been 12+ minutes and it's only at 12%:
The network I selected for the live migration is a 20Gbe (bond of two 10 Gbe).
UPDATE: The live migration progress sped up and it was completed in about 16 minutes instead of the 1hr it was estimating.
