Backups failing since the last 2 days.
-
I have 2 VMs that are now failing and have been for the last two days, the three day old worked. Just sorting through it now.
Both VMs are "bigger" than the rest, both have a disk that are 500gb and more than a lot of empty space in them. Compression is off, no updates have been performed to the XCP hosts for a while, and MS updates were applied 8 days ago. Running Xenserver 9.4.2 agent and drivers, same as the VMs that are still working. There are no snapshots for any of the VMs.
XO sources with Xen Orchestra, commit c5b7e and Master, commit e3a58 currently 31 commits behind so I guess I'll be updating it to see if something got changed.
The only error listed is a timeout error. Going to reboot and try manually running them again to see what happens. Including the the error report
{ "data": { "mode": "full", "reportWhen": "failure" }, "id": "1779369775885", "jobId": "05f70792-9624-4150-9597-5917c045bea6", "jobName": "DC2-backup", "message": "backup", "scheduleId": "5ec0bd0c-b537-4da9-afff-3f1eb0fbd3e9", "start": 1779369775885, "status": "failure", "tasks": [ { "id": "0mpfitnx5-n8d213qn7md", "start": 1779369778265, "status": "failure", "tasks": [ { "id": "0mpfitogj-nllocq70f6q", "start": 1779369778963, "status": "success", "end": 1779369780410, "result": "4353adcc-9de6-1ff1-8172-f600c7e0cb4c", "message": "snapshot" }, { "id": "0mpfitpl7-j4loeu6sjhj", "start": 1779369780427, "status": "success", "tasks": [ { "id": "0mpfitplr-condtv8ouhw", "start": 1779369780447, "status": "success", "end": 1779370618949, "result": { "size": 151409389568 }, "message": "transfer" } ], "end": 1779370618976, "message": "export", "data": { "id": "0c6ed546-4ff5-49b8-b28e-80bbd3513e52", "type": "remote", "isFull": true } }, { "id": "0mpfjbp3d-wpp55k5n2b", "start": 1779370619593, "status": "failure", "tasks": [ { "id": "0mpfjbp3i-c4eg55kbkjn", "start": 1779370619598, "status": "success", "end": 1779371188852, "result": { "size": 151409389568, "id": "e6c3c4c5-f272-a475-8f7a-d9715c257557" }, "message": "transfer" }, { "id": "0mpfjnwc5-sqxh8e4sfsr", "start": 1779371188853, "status": "failure", "end": 1779371789013, "result": { "message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)" }, "message": "vmstart" } ], "end": 1779371792327, "result": { "message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)" }, "message": "health check" }, { "id": "0mpfk0tzk-vg8hq80to3g", "start": 1779371792336, "status": "success", "end": 1779371792896, "result": { "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1779371792897, "result": { "message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)" }, "message": "backup VM", "data": { "id": "1c163183-d863-f8cc-159a-5c5281aafee3", "type": "VM", "name_label": "BMC-DC2" } } ], "end": 1779371792898, "infos": [ { "data": { "vms": [ "1c163183-d863-f8cc-159a-5c5281aafee3" ] }, "message": "vms" } ] }The above suggests the drivers are a problem, Xenserver has 9.6.0 available which requires removing current, then install new. I'll have to plan that upgrade to make sure the AD VMs stay functional, probably a job for tomorrow.
-
@Greg_E Looks to me like the timeouts are occurring during the Health Check phase of the backup job. Thus the backup was successfully transferred to the remote, but the subsequent restoration and startup of the VMs failed because the VMs didn't respond in time.
Are you performing the Health Check on every backup run?
-
Yes, I'm doing a health check on each backup job. I updated XO sources but it looks like the manual backup I started is going to fail, it is almost at 30 minutes for a 22 minute process. I'll reboot the VMs next and see if there is just something dirty in the state that needs to be cleaned up.
-
Both VMs had a reboot condition that they needed, but even after reboot I have a fail.
I watched this one, snapshot and transfer all went fine, during the import stage it only got to around 30% and then failed out and all the IO dropped off.
These are the only two VMs that I can't backup with compression due to the large amount of empty space on the disk. It's an old problem and I'm not sure it has been fixed. I may try setting zstd after lunch and trying again. If that doesn't work, it's a tomorrow problem.
-
I didn't wait long enough for XO to catch up, the manual backup I did after rebooting the VM worked, I'll hit the other VM before I start some other work and see what I see. If I don't respond back toady, then it was a "dirty" VM because of some odd reboot condition. I've just never seen that before and odd that it happened 8 days after installing MS updates.
-
@Greg_E Would you know the version of the installed xen tools?
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login