Delta backups failing
-
The output of the failure is below.
Background: attempting to run delta back ups results in failure. Full backups do work to the smb share. I get a cryptic error "NO_HOSTS_AVAILABLE() butthat doesnt make any sense and not very verbose.{ "data": { "mode": "delta", "reportWhen": "always" }, "id": "1704531600012", "jobId": "a1a87352-9d15-4b5e-aee5-d879b80cff6b", "jobName": "Netadmin server backup", "message": "backup", "scheduleId": "a0fac597-1de9-419f-8079-51cbb5ff380a", "start": 1704531600012, "status": "failure", "infos": [ { "data": { "vms": [ "c22541d3-c75a-5768-5097-ab86703d30b6" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "c22541d3-c75a-5768-5097-ab86703d30b6", "name_label": "management" }, "id": "1704531606002", "message": "backup VM", "start": 1704531606002, "status": "failure", "tasks": [ { "id": "1704531606049", "message": "clean-vm", "start": 1704531606049, "status": "success", "end": 1704531606242, "result": { "merge": false } }, { "id": "1704531606397", "message": "snapshot", "start": 1704531606397, "status": "success", "end": 1704531610234, "result": "da60ca68-caf2-a9a7-2289-91c8ad30c8b6" }, { "data": { "id": "3078d075-61d7-40dc-998e-a8832acd2652", "isFull": false, "type": "remote" }, "id": "1704531610256", "message": "export", "start": 1704531610256, "status": "failure", "tasks": [ { "id": "1704531615131", "message": "transfer", "start": 1704531615131, "status": "success", "end": 1704531819574, "result": { "size": 3475913728 } }, { "id": "1704531820693", "message": "health check", "start": 1704531820693, "status": "failure", "tasks": [ { "id": "1704531821609", "message": "transfer", "start": 1704531821609, "status": "success", "end": 1704532921353, "result": { "size": 42960242688, "id": "2d435412-3e03-aa5e-34e8-a3edd175a42a" } }, { "id": "1704532921357", "message": "vmstart", "start": 1704532921357, "status": "failure", "end": 1704532921697, "result": { "code": "NO_HOSTS_AVAILABLE", "params": [], "task": { "uuid": "162f8d78-bc6a-7c6b-91f0-3cccb06d9a59", "name_label": "Async.VM.start", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20240106T09:22:01Z", "finished": "20240106T09:22:01Z", "status": "failure", "resident_on": "OpaqueRef:931133f4-3d38-4e7e-9075-6c15727bd6f6", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "NO_HOSTS_AVAILABLE" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 104))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1199))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1356))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1191))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1567))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))" }, "message": "NO_HOSTS_AVAILABLE()", "name": "XapiError", "stack": "XapiError: NO_HOSTS_AVAILABLE()\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/_getTaskResult.mjs:11:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:998:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1032:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1022:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1195:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1704532922582, "result": { "code": "NO_HOSTS_AVAILABLE", "params": [], "task": { "uuid": "162f8d78-bc6a-7c6b-91f0-3cccb06d9a59", "name_label": "Async.VM.start", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20240106T09:22:01Z", "finished": "20240106T09:22:01Z", "status": "failure", "resident_on": "OpaqueRef:931133f4-3d38-4e7e-9075-6c15727bd6f6", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "NO_HOSTS_AVAILABLE" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 104))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1199))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1356))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1191))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1567))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))" }, "message": "NO_HOSTS_AVAILABLE()", "name": "XapiError", "stack": "XapiError: NO_HOSTS_AVAILABLE()\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/_getTaskResult.mjs:11:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:998:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1032:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1022:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1195:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1704532922582, "result": { "code": "NO_HOSTS_AVAILABLE", "params": [], "task": { "uuid": "162f8d78-bc6a-7c6b-91f0-3cccb06d9a59", "name_label": "Async.VM.start", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20240106T09:22:01Z", "finished": "20240106T09:22:01Z", "status": "failure", "resident_on": "OpaqueRef:931133f4-3d38-4e7e-9075-6c15727bd6f6", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "NO_HOSTS_AVAILABLE" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 104))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1199))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1356))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1191))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1567))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))" }, "message": "NO_HOSTS_AVAILABLE()", "name": "XapiError", "stack": "XapiError: NO_HOSTS_AVAILABLE()\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/_getTaskResult.mjs:11:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:998:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1032:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1022:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1195:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1704532922625, "result": { "code": "NO_HOSTS_AVAILABLE", "params": [], "task": { "uuid": "162f8d78-bc6a-7c6b-91f0-3cccb06d9a59", "name_label": "Async.VM.start", "name_description": "", "allowed_operations": [], "current_operations": {}, "created": "20240106T09:22:01Z", "finished": "20240106T09:22:01Z", "status": "failure", "resident_on": "OpaqueRef:931133f4-3d38-4e7e-9075-6c15727bd6f6", "progress": 1, "type": "<none/>", "result": "", "error_info": [ "NO_HOSTS_AVAILABLE" ], "other_config": {}, "subtask_of": "OpaqueRef:NULL", "subtasks": [], "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 104))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1199))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1356))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1191))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1567))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))" }, "message": "NO_HOSTS_AVAILABLE()", "name": "XapiError", "stack": "XapiError: NO_HOSTS_AVAILABLE()\n at XapiError.wrap (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/_XapiError.mjs:16:12)\n at default (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/_getTaskResult.mjs:11:29)\n at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:998:24)\n at file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1032:14\n at Array.forEach (<anonymous>)\n at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1022:12)\n at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202401051644/packages/xen-api/index.mjs:1195:14)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)" } } ], "end": 1704532922629 } -
It seems that the error is occurring during the "health check" stage, where the VM is temporarily restored and started on one of your servers. You could try disabling the health check on the backup schedule to see if the job will then complete successfully.
FWIW, it would be beneficial if you included additional details about your environment, ie --
- XO commit or XOA version
- Number of hosts
- Version of XS / XCP-ng
- Patch levels
- etc
Without those details, we can only make (un)educated guesses at the issue.
-
@michmoor0725 Take a look at this post. An update was just made to XO, so make sure you update yours. It may not be the same issue, so try @Danp recommendation.
-
A Andrew referenced this topic on
-
@michmoor0725 Did you ever found a solution? I have the same problem. I think it has someting to do with running it on a remote NAS. @Andrew you linked the same post we're in. Was this your intention? Or is there another post on this topic i didn't find?
-
@Hannes_5253 No, my mistake.... I think it was this post. (off buy a digit... 8117 vs 8177)
