random backups failing
-
I'm getting an increasing number of random backup failures lately, and almost always at the same step. They almost always fail at the health check while waiting for the VM to start. The storage is NFS on a server that has a 10g connection, same as the production storage so it should be fine, not detecting issues with the storage used for backup so not sure what's going on. Here is the short log of one of these, storage reports no errors on the Truenas server.
XCP-NG and XO (sources) are fairly up to date, I think XO was updated about a week ago and XCP is current patch level. I had a different backup job that was routinely failing, but I think it was happening when a scub in the storage was happening. Changed the time and it was OK after that.
{ "data": { "mode": "full", "reportWhen": "failure" }, "id": "1792559700001", "jobId": "05f70792-9624-4150-9597-5917c045bea6", "jobName": "DC2-backup", "message": "backup", "scheduleId": "5ec0bd0c-b537-4da9-afff-3f1eb0fbd3e9", "start": 1792559700001, "status": "failure", "tasks": [ { "id": "0mvhnriw7-cddb6rumww7", "start": 1792559702887, "status": "failure", "tasks": [ { "id": "0mvhnrjkn-ibpty24qqpg", "start": 1792559703767, "status": "success", "end": 1792559705242, "result": "05a07f3b-2c62-13ef-3405-17166901cefb", "message": "snapshot" }, { "id": "0mvhnrkq7-v610f7ynxw", "start": 1792559705263, "status": "success", "tasks": [ { "id": "0mvhnrkrm-un28zyxt91k", "start": 1792559705314, "status": "success", "end": 1792560387346, "result": { "size": 153946460672 }, "message": "transfer", "data": { "progress": 0 } } ], "end": 1792560387560, "message": "export", "data": { "id": "0c6ed546-4ff5-49b8-b28e-80bbd3513e52", "type": "remote", "isFull": true } }, { "id": "0mvho67u3-ty880j4lxuk", "start": 1792560388395, "status": "failure", "tasks": [ { "id": "0mvho67w9-okbayxed8k", "start": 1792560388473, "status": "success", "end": 1792560962736, "result": { "size": 153946460672, "id": "4ac820a4-133b-e314-a6fd-d1817051ccad", "liveMounts": [] }, "message": "transfer", "data": { "progress": 100 } }, { "id": "0mvhoij02-se35olnw9kf", "start": 1792560962738, "status": "failure", "end": 1792561562889, "result": { "message": "timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:126:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)" }, "message": "vmstart" } ], "end": 1792561565797, "result": { "message": "timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:126:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)" }, "message": "health check", "data": { "progress": 0 } }, { "id": "0mvhovgc2-ih7l8hmrak", "start": 1792561565810, "status": "success", "end": 1792561567438, "result": { "removedFiles": [], "changedFiles": [], "merge": false, "size": 0 }, "message": "clean-vm" } ], "end": 1792561567439, "result": { "message": "timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:126:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)" }, "message": "backup VM", "data": { "id": "1c163183-d863-f8cc-159a-5c5281aafee3", "type": "VM", "name_label": "BMC-DC2", "progress": 0 } } ], "end": 1792561567595, "infos": [ { "data": { "vms": [ "1c163183-d863-f8cc-159a-5c5281aafee3" ] }, "message": "vms" } ] } -
@Greg_E When was the last time you updated the guest drivers?
-
Replying with an update. As usual, a manually trigger backup completes. 12 minutes to send to the remote, 10 minutes to bring it back. But with the domain controller, it look like I may need to update the management agent, it's taking a long time to get that running.
When a DC comes up, it reaches out to the network to check with it's peer and sync data while it was gone. During the health check the network is not there so it takes 5+ minutes to fully boot. it's now sitting waiting. took about 10 minutes total for the MA to report and then shut the health check down. Not sure if there is a better way to do this for a domain controller or not, mostly just a data point for other people that think their health check may be stuck.
Time to go fix the other VM that failed last night.
[edit] it said it failed again, though I watched the console for about 4 minutes and clearly it booted. I'll have to update the MA and drivers and see what happens. But it looks like 10 minutes waiting for the MA is too long now.
-
Probably too long ago, mentioned it on my follow up post while you were posting.
-
@Greg_E Worth remembering that DC backups need to be completed by something that is database aware, as far as I know, XO doesn't currently have this functionality, and if so, you are risking your entire environment....
-
@DustinB Hmmm... They have always completed in the past, but I'll admit that I haven't tried to restore one and see if it really works. too busy to perform the test the last couple years.
but now I know the time out is after the machine boots, and I've seen the console come up and wait longer than 4 minutes, so probably it is OK if it could connect to the other DC and compare the database.
-
@Greg_E Right, the backup operations are occurring outside of the VM, at the hypervisor layer. The guest drivers, don't stop or flush any database writes to disk etc.
So while the VM will likely boot, you might lose records or changes.
Depending on the size of your organization, it may be moot, but still worth keeping an eye on it.
I've seen other people use Veeam Backup and Replication to backup their Domain Controllers outside of the XO backup mechanism, and now that it's officially integrated, it might be a good contender here (and for others). https://www.veeam.com/products/free/backup-recovery.html for things that need to be handled "carefully".
-
Ok, generally I'm doing backups after everyone has gone home for the night, so changes to the DB should be minimal.
I'll have to look into this more deeply and maybe use the Server Backup tool to schedule both DC going forward. Used to use that when bare metal, and had to restore a couple times too.
-
I'm guessing this might be having updates applied, and now I need new Xen Tools. I have a feeling tomorrow will be filled with updating the tools and checking things because I bet all my backups fail tonight. And I bet all fail after they try to start.
-
I'm guessing this might be having updates applied, and now I need new Xen Tools. I have a feeling tomorrow will be filled with updating the tools and checking things because I bet all my backups fail tonight. And I bet all fail after they try to start.
Are you performing nightly healthy checks? Seems obsessive to need to check every system makes it to a login screen, but on the other hand, I understand.
Just curious.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login