Backup Issue: "timeout reached while waiting for OpaqueRef"
-
I have a Windows 2016 server that backs up successfully however does not complete the heath check. Watching the console the VM does boot and the management tools are installed and recognised.
{ "data": { "mode": "delta", "reportWhen": "failure" }, "id": "1728615190190", "jobId": "73df5884-3e18-4ce9-9833-53be5a6e0a34", "jobName": "Production Servers", "message": "backup", "scheduleId": "dae62e27-1434-45d5-930b-489e58fd7909", "start": 1728615190190, "status": "failure", "infos": [ { "data": { "vms": [ "af2b36be-d1cf-4e67-e1a7-37f0c94300d5" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "af2b36be-d1cf-4e67-e1a7-37f0c94300d5", "name_label": "SERVER01" }, "id": "1728615191235", "message": "backup VM", "start": 1728615191235, "status": "failure", "tasks": [ { "id": "1728615191463", "message": "snapshot", "start": 1728615191463, "status": "success", "end": 1728615193884, "result": "6ca2b876-1da3-f1cb-3f96-9999be9a8bda" }, { "data": { "id": "eb9076c3-13a5-f9a7-43e3-ff1146e5599d", "isFull": false, "name_label": "SVR10533 ZFS", "type": "SR" }, "id": "1728615193885", "message": "export", "start": 1728615193885, "status": "failure", "tasks": [ { "id": "1728615195468", "message": "transfer", "start": 1728615195468, "status": "success", "end": 1728615290406, "result": { "size": 10089412096 } }, { "id": "1728615303194", "message": "health check", "start": 1728615303194, "status": "failure", "tasks": [ { "id": "1728615303201", "message": "cloning-vm", "start": 1728615303201, "status": "success", "end": 1728615305528, "result": "OpaqueRef:3afcabf3-50ad-ef9d-70d1-e591adac64ba" }, { "id": "1728615305532", "message": "vmstart", "start": 1728615305532, "status": "failure", "end": 1728615905736, "result": { "message": "timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:269:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:253:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1728615918927, "result": { "message": "timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:269:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:253:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1728615918927, "result": { "message": "timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:269:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:253:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1728615918962, "result": { "message": "timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.", "name": "Error", "stack": "Error: timeout reached while waiting for OpaqueRef:9d2c7b95-8227-2790-1326-0fe5dc72ab0d to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:269:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/xapi/index.mjs:253:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202410101438/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1728615918963 }Other VMs on the same host backup correctly.
-
Maybe because the VM takes too long to get up? Can you raise the timeout?
-
The VM boot process is definitely completed within 10 mins however I will extend the timeout to 15 to be safe.
Should this be edited in config.toml or via the GUI in XO?
If I edit config.toml, how do I load the update settings?
-
Edit the config file and restart xo-server
-
During that boot, can you witness the guest tools visible in XO? (to see if it's XO being "blind" or the tools not reported correctly)
-
The Health Check VM shows the tools as being recognised by XO, is this what you mean?
-
So the tools are visible but XO can't find them. Thanks for the feedback, we'll dig.
