Backup Issue: "timeout reached while waiting for OpaqueRef"
-
I have a Windows 2016 server that backs up successfully however does not complete the heath check. Watching the console the VM does boot and the management tools are installed and recognised.
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1728615190190", "jobId": "73df5884-3e18-4ce9-9833-53be5a6e0a34", "jobName": "Production Servers", "message": "backup", "scheduleId": "dae62e27-1434-45d5-930b-489e58fd7909", "start": 1728615190190, "status": "failure", "infos": [ { "data": { "vms": [ "af2b36be-d1cf-4e67-e1a7-37f0c94300d5" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "af2b36be-d1cf-4e67-e1a7-37f0c94300d5", "name_label": "SERVER01" }, "id": "1728615191235", "message": "backup VM", "start": 1728615191235, "status": "failure", "tasks": [ { "id": "1728615191463", "message": "snapshot", "start": 1728615191463, "status": "success", "end": 1728615193884, "result": "6ca2b876-1da3-f1cb-3f96-9999be9a8bda" }, { "data": { "id": "eb9076c3-13a5-f9a7-43e3-ff1146e5599d", "isFull": false, "name_label": "SVR10533 ZFS", "type": "SR" }, "id": "1728615193885", "message": "export", "start": 1728615193885, "status": "failure", "tasks": [ { "id": "1728615195468", "message": "transfer", "start": 1728615195468, "status": "success", "end": 1728615290406, "result": { "size": 10089412096 } }, { "id": "1728615303194", "message": "health check", "start": 1728615303194, "status": "failure", "tasks": [ { "id": "1728615303201", "message": "cloning-vm", "start": 1728615303201, "status": "success", "end": 1728615305528, "result": "OpaqueRef:3afcabf3-50ad-ef9d-70d1-e591adac64ba" }, { "id": "1728615305532", "message": "vmstart", "start": 1728615305532, "status": "failure", "end": 1728615905736, "result": { "message": "timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:269:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:253:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1728615918927, "result": { "message": "timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:269:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:253:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1728615918927, "result": { "message": "timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:269:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:253:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1728615918962, "result": { "message": "timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:269:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:253:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1728615918963 }Other VMs on the same host backup correctly.
-
Maybe because the VM takes too long to get up? Can you raise the timeout?
-
The VM boot process is definitely completed within 10 mins however I will extend the timeout to 15 to be safe.
Should this be edited in config.toml or via the GUI in XO?
If I edit config.toml, how do I load the update settings?
-
Edit the config file and restart xo-server
-
During that boot, can you witness the guest tools visible in XO? (to see if it's XO being "blind" or the tools not reported correctly)
-
The Health Check VM shows the tools as being recognised by XO, is this what you mean?
-
So the tools are visible but XO can't find them. Thanks for the feedback, we'll dig.
-
What was the fix for this as one of our Windows 10 VMs backups have started to fail and rebooting the VM only takes less than 1 minute so not sure increasing the timeout is the issue. Here is the backup log entries:
{ "data": { "type": "VM", "id": "07767a82-3cc0-b2b7-e399-90793896a574", "name_label": "Windows10C" }, "id": "1760322641022", "message": "backup VM", "start": 1760322641022, "status": "failure", "tasks": [ { "id": "1760322641048", "message": "clean-vm", "start": 1760322641048, "status": "success", "tasks": [ { "id": "1760322641516", "message": "merge", "start": 1760322641516, "status": "success", "end": 1760323012896 } ], "warnings": [ { "data": { "path": "/xo-vm-backups/07767a82-3cc0-b2b7-e399-90793896a574/20251012T023042Z.json", "actual": 5475663872, "expected": 5477068288 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1760323013334, "result": { "merge": true } }, { "id": "1760323014314", "message": "snapshot", "start": 1760323014314, "status": "success", "end": 1760323016106, "result": "0b79d24f-a579-b669-50e1-412516276886" }, { "data": { "id": "2a4bc998-972d-4132-83ba-f3c67b860676", "isFull": false, "type": "remote" }, "id": "1760323016106:0", "message": "export", "start": 1760323016106, "status": "failure", "tasks": [ { "id": "1760323018513", "message": "transfer", "start": 1760323018513, "status": "success", "end": 1760323462425, "result": { "size": 7688159232 } }, { "id": "1760323472984", "message": "health check", "start": 1760323472984, "status": "failure", "tasks": [ { "id": "1760323480406", "message": "transfer", "start": 1760323480406, "status": "success", "end": 1760324685177, "result": { "size": 0, "id": "490d3623-4b1f-e4a4-4947-8062aa0e757c" } }, { "id": "1760324685178", "message": "vmstart", "start": 1760324685178, "status": "failure", "end": 1760325285289, "result": { "message": "timeout reached while waiting for OpaqueRef:5c4eef85-eebb-4480-826b-7bbe596f4349 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:5c4eef85-eebb-4480-826b-7bbe596f4349 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:275:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:259:12)\n at file:///opt/xen-orchestra/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)" } } ], "end": 1760325289511, "result": { "message": "timeout reached while waiting for OpaqueRef:5c4eef85-eebb-4480-826b-7bbe596f4349 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:5c4eef85-eebb-4480-826b-7bbe596f4349 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:275:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:259:12)\n at file:///opt/xen-orchestra/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)" } } ], "end": 1760325289511, "result": { "message": "timeout reached while waiting for OpaqueRef:5c4eef85-eebb-4480-826b-7bbe596f4349 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:5c4eef85-eebb-4480-826b-7bbe596f4349 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:275:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:259:12)\n at file:///opt/xen-orchestra/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)" } }, { "data": { "id": "349e7130-da43-6550-e7c0-aa0a84102e0f", "isFull": false, "name_label": "Local storage", "type": "SR" }, "id": "1760323016107", "message": "export", "start": 1760323016107, "status": "failure", "tasks": [ { "id": "1760323017919", "message": "transfer", "start": 1760323017919, "status": "success", "end": 1760323463982, "result": { "size": 7688159232 } }, { "id": "1760323472984:0", "message": "health check", "start": 1760323472984, "status": "failure", "tasks": [ { "id": "1760323473002", "message": "copying-vm", "start": 1760323473002, "status": "success", "end": 1760325038087, "result": "OpaqueRef:0233c593-da90-4813-959b-46e4e679a662" }, { "id": "1760325038108", "message": "vmstart", "start": 1760325038108, "status": "failure", "end": 1760325638322, "result": { "message": "timeout reached while waiting for OpaqueRef:ca1c86dd-aafc-4f25-bd8c-89399771c004 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:ca1c86dd-aafc-4f25-bd8c-89399771c004 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:275:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:259:12)\n at file:///opt/xen-orchestra/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)" } } ], "end": 1760325643426, "result": { "message": "timeout reached while waiting for OpaqueRef:ca1c86dd-aafc-4f25-bd8c-89399771c004 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:ca1c86dd-aafc-4f25-bd8c-89399771c004 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:275:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:259:12)\n at file:///opt/xen-orchestra/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)" } } ], "end": 1760325643426, "result": { "message": "timeout reached while waiting for OpaqueRef:ca1c86dd-aafc-4f25-bd8c-89399771c004 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:ca1c86dd-aafc-4f25-bd8c-89399771c004 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:275:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:259:12)\n at file:///opt/xen-orchestra/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)" } } ], "end": 1760325644170, "result": { "errors": [ { "message": "timeout reached while waiting for OpaqueRef:5c4eef85-eebb-4480-826b-7bbe596f4349 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:5c4eef85-eebb-4480-826b-7bbe596f4349 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:275:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:259:12)\n at file:///opt/xen-orchestra/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)" }, { "message": "timeout reached while waiting for OpaqueRef:ca1c86dd-aafc-4f25-bd8c-89399771c004 to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:ca1c86dd-aafc-4f25-bd8c-89399771c004 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:275:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xen-orchestra/@xen-orchestra/xapi/index.mjs:259:12)\n at file:///opt/xen-orchestra/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)" } ], "message": "all targets have failed, step: writer.healthCheck()", "name": "Error", "stack": "Error: all targets have failed, step: writer.healthCheck()\n at IncrementalXapiVmBackupRunner._callWriters (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:64:13)\n at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n at async IncrementalXapiVmBackupRunner._healthCheck (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:94:5)\n at async IncrementalXapiVmBackupRunner.run (file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:407:5)\n at async file:///opt/xen-orchestra/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38" } }, -
It's been a year, so I don't remember. Adding @florent in case it rings a bell
-
This post is deleted! -
I believe this issue was resolved when the health check system was changed to detect network connectivity at startup so it did not need to wait for then entire VM to boot. Needs the Xen tools to be installed. I have not had an issue since this change.
