Some backups failing

RickyH

Most backups working fine. 5 not working. End with "SR_Backend_Failure_44 (There is unsufficient space)". Those 5 x VMs will not snapshot either. The host is 8.2.1.

Local storage on the host is 51% used.

This is one of the VMs that won;t backup, however there are 6 identical, and only one of them backs up correctly.

Any ideas?

Danp

It sounds like you are running low on diskspace on the SR where these VMs are located. Have you checked the Unhealth VDIs section under Dashboard > Health tab to see if you have a coalesce issue?

RickyH

@Danp Hi Dan, there is only one Unhealthy VDI, but about 110 (500mb) Orphan VDIs....?

RickyH

Just too add something too. The 1 x VM that snapshots fine had management agent 9.4.1 on it - and the other 5 have 9.3.3. So I assumed this might have been the issue - and upgraded one of them to 9.4.1 as a test. Still same error.

Note that about 10 other VMs on the host snapshot and backup just fine.

RickyH

The error log for the snapshot failure:

vm.snapshot
{
  "id": "567459b9-362c-90e8-9b2b-9e61572243ff"
}
{
  "code": "SR_BACKEND_FAILURE_44",
  "params": [
    "",
    "There is insufficient space",
    ""
  ],
  "task": {
    "uuid": "f2c46023-ea35-7d61-1106-b799adec4648",
    "name_label": "Async.VM.snapshot",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20250829T10:31:09Z",
    "finished": "20250829T10:31:16Z",
    "status": "failure",
    "resident_on": "OpaqueRef:b71238e8-bce1-4a59-b9be-870e2de57558",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "SR_BACKEND_FAILURE_44",
      "",
      "There is insufficient space",
      ""
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [
      "OpaqueRef:09be2d2a-c450-42fc-8c3a-3c876274bb18"
    ],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 80))((process xapi)(filename list.ml)(line 110))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 122))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 130))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 171))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 209))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 220))((process xapi)(filename list.ml)(line 121))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 222))((process xapi)(filename ocaml/xapi/xapi_vm_clone.ml)(line 442))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/xapi_vm_snapshot.ml)(line 33))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
  },
  "message": "SR_BACKEND_FAILURE_44(, There is insufficient space, )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_44(, There is insufficient space, )
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202408190902/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202408190902/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202408190902/packages/xen-api/index.mjs:1041:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202408190902/packages/xen-api/index.mjs:1075:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202408190902/packages/xen-api/index.mjs:1065:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202408190902/packages/xen-api/index.mjs:1238:14)"
}

Danp

What is the type and size of the SR? How much space is free on it?

RickyH

The SR of the host is 51% full.

The VM:

Danp

The 1TB disk resides on a different SR, FME_Snap, which appears to be near capacity. This is why you can't take a snapshot or backup these VMs.

RickyH

@Danp Thanks Dan - that makes sense - however one of those 6 x VMs is backing up and snapshotting perfectly, and it has the extra disk too..
Why would this be?

RickyH

Is there a possibility of converting that SR to thin provisioning - would that help in this case?

RickyH

Or could I get a one off backup of those VMs by disconnecting that disk temporarily?

RickyH

Thanks everyone for their help. I believe I have a way forward now.