<![CDATA[Backups failing since the last 2 days.]]>I have 2 VMs that are now failing and have been for the last two days, the three day old worked. Just sorting through it now.

Both VMs are "bigger" than the rest, both have a disk that are 500gb and more than a lot of empty space in them. Compression is off, no updates have been performed to the XCP hosts for a while, and MS updates were applied 8 days ago. Running Xenserver 9.4.2 agent and drivers, same as the VMs that are still working. There are no snapshots for any of the VMs.

XO sources with Xen Orchestra, commit c5b7e and Master, commit e3a58 currently 31 commits behind so I guess I'll be updating it to see if something got changed.

The only error listed is a timeout error. Going to reboot and try manually running them again to see what happens. Including the the error report

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1779369775885",
  "jobId": "05f70792-9624-4150-9597-5917c045bea6",
  "jobName": "DC2-backup",
  "message": "backup",
  "scheduleId": "5ec0bd0c-b537-4da9-afff-3f1eb0fbd3e9",
  "start": 1779369775885,
  "status": "failure",
  "tasks": [
    {
      "id": "0mpfitnx5-n8d213qn7md",
      "start": 1779369778265,
      "status": "failure",
      "tasks": [
        {
          "id": "0mpfitogj-nllocq70f6q",
          "start": 1779369778963,
          "status": "success",
          "end": 1779369780410,
          "result": "4353adcc-9de6-1ff1-8172-f600c7e0cb4c",
          "message": "snapshot"
        },
        {
          "id": "0mpfitpl7-j4loeu6sjhj",
          "start": 1779369780427,
          "status": "success",
          "tasks": [
            {
              "id": "0mpfitplr-condtv8ouhw",
              "start": 1779369780447,
              "status": "success",
              "end": 1779370618949,
              "result": {
                "size": 151409389568
              },
              "message": "transfer"
            }
          ],
          "end": 1779370618976,
          "message": "export",
          "data": {
            "id": "0c6ed546-4ff5-49b8-b28e-80bbd3513e52",
            "type": "remote",
            "isFull": true
          }
        },
        {
          "id": "0mpfjbp3d-wpp55k5n2b",
          "start": 1779370619593,
          "status": "failure",
          "tasks": [
            {
              "id": "0mpfjbp3i-c4eg55kbkjn",
              "start": 1779370619598,
              "status": "success",
              "end": 1779371188852,
              "result": {
                "size": 151409389568,
                "id": "e6c3c4c5-f272-a475-8f7a-d9715c257557"
              },
              "message": "transfer"
            },
            {
              "id": "0mpfjnwc5-sqxh8e4sfsr",
              "start": 1779371188853,
              "status": "failure",
              "end": 1779371789013,
              "result": {
                "message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
                "name": "Error",
                "stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
              },
              "message": "vmstart"
            }
          ],
          "end": 1779371792327,
          "result": {
            "message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
            "name": "Error",
            "stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
          },
          "message": "health check"
        },
        {
          "id": "0mpfk0tzk-vg8hq80to3g",
          "start": 1779371792336,
          "status": "success",
          "end": 1779371792896,
          "result": {
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        }
      ],
      "end": 1779371792897,
      "result": {
        "message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
        "name": "Error",
        "stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n    at new Promise (<anonymous>)\n    at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n    at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n    at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n    at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n    at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
      },
      "message": "backup VM",
      "data": {
        "id": "1c163183-d863-f8cc-159a-5c5281aafee3",
        "type": "VM",
        "name_label": "BMC-DC2"
      }
    }
  ],
  "end": 1779371792898,
  "infos": [
    {
      "data": {
        "vms": [
          "1c163183-d863-f8cc-159a-5c5281aafee3"
        ]
      },
      "message": "vms"
    }
  ]
}

The above suggests the drivers are a problem, Xenserver has 9.6.0 available which requires removing current, then install new. I'll have to plan that upgrade to make sure the AD VMs stay functional, probably a job for tomorrow.

]]>https://xcp-ng.org/forum/topic/12234/backups-failing-since-the-last-2-days.RSS for NodeThu, 21 May 2026 16:29:36 GMTThu, 21 May 2026 14:09:58 GMT60<![CDATA[Reply to Backups failing since the last 2 days. on Thu, 21 May 2026 15:47:08 GMT]]>Both VMs had a reboot condition that they needed, but even after reboot I have a fail.

I watched this one, snapshot and transfer all went fine, during the import stage it only got to around 30% and then failed out and all the IO dropped off.

These are the only two VMs that I can't backup with compression due to the large amount of empty space on the disk. It's an old problem and I'm not sure it has been fixed. I may try setting zstd after lunch and trying again. If that doesn't work, it's a tomorrow problem.

]]>https://xcp-ng.org/forum/post/105666https://xcp-ng.org/forum/post/105666Thu, 21 May 2026 15:47:08 GMT<![CDATA[Reply to Backups failing since the last 2 days. on Thu, 21 May 2026 14:56:21 GMT]]>@Danp

Yes, I'm doing a health check on each backup job. I updated XO sources but it looks like the manual backup I started is going to fail, it is almost at 30 minutes for a 22 minute process. I'll reboot the VMs next and see if there is just something dirty in the state that needs to be cleaned up.

]]>https://xcp-ng.org/forum/post/105663https://xcp-ng.org/forum/post/105663Thu, 21 May 2026 14:56:21 GMT<![CDATA[Reply to Backups failing since the last 2 days. on Thu, 21 May 2026 14:47:43 GMT]]>@Greg_E Looks to me like the timeouts are occurring during the Health Check phase of the backup job. Thus the backup was successfully transferred to the remote, but the subsequent restoration and startup of the VMs failed because the VMs didn't respond in time.

Are you performing the Health Check on every backup run?

]]>https://xcp-ng.org/forum/post/105662https://xcp-ng.org/forum/post/105662Thu, 21 May 2026 14:47:43 GMT