<![CDATA[random backups failing]]>I'm getting an increasing number of random backup failures lately, and almost always at the same step. They almost always fail at the health check while waiting for the VM to start. The storage is NFS on a server that has a 10g connection, same as the production storage so it should be fine, not detecting issues with the storage used for backup so not sure what's going on. Here is the short log of one of these, storage reports no errors on the Truenas server.

XCP-NG and XO (sources) are fairly up to date, I think XO was updated about a week ago and XCP is current patch level. I had a different backup job that was routinely failing, but I think it was happening when a scub in the storage was happening. Changed the time and it was OK after that.

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1792559700001",
  "jobId": "05f70792-9624-4150-9597-5917c045bea6",
  "jobName": "DC2-backup",
  "message": "backup",
  "scheduleId": "5ec0bd0c-b537-4da9-afff-3f1eb0fbd3e9",
  "start": 1792559700001,
  "status": "failure",
  "tasks": [
    {
      "id": "0mvhnriw7-cddb6rumww7",
      "start": 1792559702887,
      "status": "failure",
      "tasks": [
        {
          "id": "0mvhnrjkn-ibpty24qqpg",
          "start": 1792559703767,
          "status": "success",
          "end": 1792559705242,
          "result": "05a07f3b-2c62-13ef-3405-17166901cefb",
          "message": "snapshot"
        },
        {
          "id": "0mvhnrkq7-v610f7ynxw",
          "start": 1792559705263,
          "status": "success",
          "tasks": [
            {
              "id": "0mvhnrkrm-un28zyxt91k",
              "start": 1792559705314,
              "status": "success",
              "end": 1792560387346,
              "result": {
                "size": 153946460672
              },
              "message": "transfer",
              "data": {
                "progress": 0
              }
            }
          ],
          "end": 1792560387560,
          "message": "export",
          "data": {
            "id": "0c6ed546-4ff5-49b8-b28e-80bbd3513e52",
            "type": "remote",
            "isFull": true
          }
        },
        {
          "id": "0mvho67u3-ty880j4lxuk",
          "start": 1792560388395,
          "status": "failure",
          "tasks": [
            {
              "id": "0mvho67w9-okbayxed8k",
              "start": 1792560388473,
              "status": "success",
              "end": 1792560962736,
              "result": {
                "size": 153946460672,
                "id": "4ac820a4-133b-e314-a6fd-d1817051ccad",
                "liveMounts": []
              },
              "message": "transfer",
              "data": {
                "progress": 100
              }
            },
            {
              "id": "0mvhoij02-se35olnw9kf",
              "start": 1792560962738,
              "status": "failure",
              "end": 1792561562889,
              "result": {
                "message": "timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.",
                "name": "Error",
                "stack": "Error: timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:126:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)"
              },
              "message": "vmstart"
            }
          ],
          "end": 1792561565797,
          "result": {
            "message": "timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.",
            "name": "Error",
            "stack": "Error: timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:126:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)"
          },
          "message": "health check",
          "data": {
            "progress": 0
          }
        },
        {
          "id": "0mvhovgc2-ih7l8hmrak",
          "start": 1792561565810,
          "status": "success",
          "end": 1792561567438,
          "result": {
            "removedFiles": [],
            "changedFiles": [],
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        }
      ],
      "end": 1792561567439,
      "result": {
        "message": "timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.",
        "name": "Error",
        "stack": "Error: timeout reached while waiting for OpaqueRef:a451ca09-fb0f-06c1-5ee5-70cd22f9fc51 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202610130947/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:126:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202610130947/@vates/task/index.js:188:20)"
      },
      "message": "backup VM",
      "data": {
        "id": "1c163183-d863-f8cc-159a-5c5281aafee3",
        "type": "VM",
        "name_label": "BMC-DC2",
        "progress": 0
      }
    }
  ],
  "end": 1792561567595,
  "infos": [
    {
      "data": {
        "vms": [
          "1c163183-d863-f8cc-159a-5c5281aafee3"
        ]
      },
      "message": "vms"
    }
  ]
}
]]>https://xcp-ng.org/forum/topic/12544/random-backups-failingRSS for NodeWed, 21 Oct 2026 15:35:31 GMTWed, 21 Oct 2026 13:38:17 GMT60<![CDATA[Reply to random backups failing on Wed, 21 Oct 2026 15:30:57 GMT]]>@DustinB

Ok, generally I'm doing backups after everyone has gone home for the night, so changes to the DB should be minimal.

I'll have to look into this more deeply and maybe use the Server Backup tool to schedule both DC going forward. Used to use that when bare metal, and had to restore a couple times too.

]]>https://xcp-ng.org/forum/post/109219https://xcp-ng.org/forum/post/109219Wed, 21 Oct 2026 15:30:57 GMT<![CDATA[Reply to random backups failing on Wed, 21 Oct 2026 15:29:53 GMT]]>@Greg_E Right, the backup operations are occurring outside of the VM, at the hypervisor layer. The guest drivers, don't stop or flush any database writes to disk etc.

So while the VM will likely boot, you might lose records or changes.

Depending on the size of your organization, it may be moot, but still worth keeping an eye on it.

I've seen other people use Veeam Backup and Replication to backup their Domain Controllers outside of the XO backup mechanism, and now that it's officially integrated, it might be a good contender here (and for others). https://www.veeam.com/products/free/backup-recovery.html for things that need to be handled "carefully".

]]>https://xcp-ng.org/forum/post/109218https://xcp-ng.org/forum/post/109218Wed, 21 Oct 2026 15:29:53 GMT<![CDATA[Reply to random backups failing on Wed, 21 Oct 2026 15:24:19 GMT]]>@DustinB Hmmm... They have always completed in the past, but I'll admit that I haven't tried to restore one and see if it really works. too busy to perform the test the last couple years.

but now I know the time out is after the machine boots, and I've seen the console come up and wait longer than 4 minutes, so probably it is OK if it could connect to the other DC and compare the database.

]]>https://xcp-ng.org/forum/post/109217https://xcp-ng.org/forum/post/109217Wed, 21 Oct 2026 15:24:19 GMT<![CDATA[Reply to random backups failing on Wed, 21 Oct 2026 15:21:22 GMT]]>@Greg_E Worth remembering that DC backups need to be completed by something that is database aware, as far as I know, XO doesn't currently have this functionality, and if so, you are risking your entire environment....

]]>https://xcp-ng.org/forum/post/109216https://xcp-ng.org/forum/post/109216Wed, 21 Oct 2026 15:21:22 GMT<![CDATA[Reply to random backups failing on Wed, 21 Oct 2026 15:17:30 GMT]]>@DustinB

Probably too long ago, mentioned it on my follow up post while you were posting.

]]>https://xcp-ng.org/forum/post/109215https://xcp-ng.org/forum/post/109215Wed, 21 Oct 2026 15:17:30 GMT<![CDATA[Reply to random backups failing on Wed, 21 Oct 2026 15:21:30 GMT]]>Replying with an update. As usual, a manually trigger backup completes. 12 minutes to send to the remote, 10 minutes to bring it back. But with the domain controller, it look like I may need to update the management agent, it's taking a long time to get that running.

When a DC comes up, it reaches out to the network to check with it's peer and sync data while it was gone. During the health check the network is not there so it takes 5+ minutes to fully boot. it's now sitting waiting. took about 10 minutes total for the MA to report and then shut the health check down. Not sure if there is a better way to do this for a domain controller or not, mostly just a data point for other people that think their health check may be stuck.

Time to go fix the other VM that failed last night.

[edit] it said it failed again, though I watched the console for about 4 minutes and clearly it booted. I'll have to update the MA and drivers and see what happens. But it looks like 10 minutes waiting for the MA is too long now.

]]>https://xcp-ng.org/forum/post/109213https://xcp-ng.org/forum/post/109213Wed, 21 Oct 2026 15:21:30 GMT<![CDATA[Reply to random backups failing on Wed, 21 Oct 2026 15:14:21 GMT]]>@Greg_E When was the last time you updated the guest drivers?

]]>https://xcp-ng.org/forum/post/109212https://xcp-ng.org/forum/post/109212Wed, 21 Oct 2026 15:14:21 GMT