I have 2 VMs that are now failing and have been for the last two days, the three day old worked. Just sorting through it now.
Both VMs are "bigger" than the rest, both have a disk that are 500gb and more than a lot of empty space in them. Compression is off, no updates have been performed to the XCP hosts for a while, and MS updates were applied 8 days ago. Running Xenserver 9.4.2 agent and drivers, same as the VMs that are still working. There are no snapshots for any of the VMs.
XO sources with Xen Orchestra, commit c5b7e and Master, commit e3a58 currently 31 commits behind so I guess I'll be updating it to see if something got changed.
The only error listed is a timeout error. Going to reboot and try manually running them again to see what happens. Including the the error report
{
"data": {
"mode": "full",
"reportWhen": "failure"
},
"id": "1779369775885",
"jobId": "05f70792-9624-4150-9597-5917c045bea6",
"jobName": "DC2-backup",
"message": "backup",
"scheduleId": "5ec0bd0c-b537-4da9-afff-3f1eb0fbd3e9",
"start": 1779369775885,
"status": "failure",
"tasks": [
{
"id": "0mpfitnx5-n8d213qn7md",
"start": 1779369778265,
"status": "failure",
"tasks": [
{
"id": "0mpfitogj-nllocq70f6q",
"start": 1779369778963,
"status": "success",
"end": 1779369780410,
"result": "4353adcc-9de6-1ff1-8172-f600c7e0cb4c",
"message": "snapshot"
},
{
"id": "0mpfitpl7-j4loeu6sjhj",
"start": 1779369780427,
"status": "success",
"tasks": [
{
"id": "0mpfitplr-condtv8ouhw",
"start": 1779369780447,
"status": "success",
"end": 1779370618949,
"result": {
"size": 151409389568
},
"message": "transfer"
}
],
"end": 1779370618976,
"message": "export",
"data": {
"id": "0c6ed546-4ff5-49b8-b28e-80bbd3513e52",
"type": "remote",
"isFull": true
}
},
{
"id": "0mpfjbp3d-wpp55k5n2b",
"start": 1779370619593,
"status": "failure",
"tasks": [
{
"id": "0mpfjbp3i-c4eg55kbkjn",
"start": 1779370619598,
"status": "success",
"end": 1779371188852,
"result": {
"size": 151409389568,
"id": "e6c3c4c5-f272-a475-8f7a-d9715c257557"
},
"message": "transfer"
},
{
"id": "0mpfjnwc5-sqxh8e4sfsr",
"start": 1779371188853,
"status": "failure",
"end": 1779371789013,
"result": {
"message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
"name": "Error",
"stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
},
"message": "vmstart"
}
],
"end": 1779371792327,
"result": {
"message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
"name": "Error",
"stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
},
"message": "health check"
},
{
"id": "0mpfk0tzk-vg8hq80to3g",
"start": 1779371792336,
"status": "success",
"end": 1779371792896,
"result": {
"merge": false,
"size": 0
},
"message": "clean-vm"
}
],
"end": 1779371792897,
"result": {
"message": "timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.",
"name": "Error",
"stack": "Error: timeout reached while waiting for OpaqueRef:ec13b4c5-cf50-4a2e-a4a5-bdfc3fbdf8b9 to report the driver version through the Xen tools. Please check or update the Xen tools.\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:295:23\n at new Promise (<anonymous>)\n at Xapi.waitObjectState (file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/xapi/index.mjs:279:12)\n at file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/HealthCheckVmBackup.mjs:63:20\n at process.processTicksAndRejections (node:internal/process/task_queues:104:5)\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)\n at async file:///opt/xo/xo-builds/xen-orchestra-202605080919/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:127:13\n at async Task.runInside (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:204:22)\n at async Task.run (/opt/xo/xo-builds/xen-orchestra-202605080919/@vates/task/index.js:188:20)"
},
"message": "backup VM",
"data": {
"id": "1c163183-d863-f8cc-159a-5c5281aafee3",
"type": "VM",
"name_label": "BMC-DC2"
}
}
],
"end": 1779371792898,
"infos": [
{
"data": {
"vms": [
"1c163183-d863-f8cc-159a-5c5281aafee3"
]
},
"message": "vms"
}
]
}
The above suggests the drivers are a problem, Xenserver has 9.6.0 available which requires removing current, then install new. I'll have to plan that upgrade to make sure the AD VMs stay functional, probably a job for tomorrow.
