Backup error: "no writers left"

vmpr

Hi everyone, I've got the first time today an error with my backup:

5 vms were successfully backuped
2 didn't and failed with: Error: no writers left
I restarted those two vms and one finished successful but the other didn't, this is the log:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1621848907079",
  "jobId": "d3ada465-98aa-4f30-bf53-289d00249d17",
  "jobName": "delta_all",
  "message": "backup",
  "scheduleId": "5b2e2c96-2919-496f-9a4d-7db06fd28634",
  "start": 1621848907079,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "61673574-80c8-ae18-c379-562cc38a7eaf",
          "1d2b0597-69c3-9a07-34ea-f45a801affad"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "61673574-80c8-ae18-c379-562cc38a7eaf"
      },
      "id": "1621848910981",
      "message": "backup VM",
      "start": 1621848910981,
      "status": "failure",
      "tasks": [
        {
          "id": "1621848913457",
          "message": "snapshot",
          "start": 1621848913457,
          "status": "success",
          "end": 1621848918206,
          "result": "6053c653-1c21-12f6-6a4d-56419373bf9c"
        },
        {
          "data": {
            "id": "f34266e4-aaf7-4e70-88d1-ae1b811b7e55",
            "isFull": true,
            "type": "remote"
          },
          "id": "1621848918208",
          "message": "export",
          "start": 1621848918208,
          "status": "failure",
          "tasks": [
            {
              "id": "1621848918254",
              "message": "transfer",
              "start": 1621848918254,
              "status": "failure",
              "end": 1621851676334,
              "result": {
                "errno": -122,
                "code": "Unknown system error -122",
                "syscall": "write",
                "message": "Unknown system error -122: Unknown system error -122, write",
                "name": "Error",
                "stack": "Error: Unknown system error -122: Unknown system error -122, write"
              }
            }
          ],
          "end": 1621851676344,
          "result": {
            "errno": -122,
            "code": "Unknown system error -122",
            "syscall": "write",
            "message": "Unknown system error -122: Unknown system error -122, write",
            "name": "Error",
            "stack": "Error: Unknown system error -122: Unknown system error -122, write"
          }
        }
      ],
      "end": 1621851678754,
      "result": {
        "message": "no writers left",
        "name": "Error",
        "stack": "Error: no writers left\n    at VmBackup._callWriters (/opt/xo/xo-builds/xen-orchestra-202105071253/@xen-orchestra/backups/_VmBackup.js:118:13)\n    at async VmBackup._copyDelta (/opt/xo/xo-builds/xen-orchestra-202105071253/@xen-orchestra/backups/_VmBackup.js:190:5)\n    at async VmBackup.run (/opt/xo/xo-builds/xen-orchestra-202105071253/@xen-orchestra/backups/_VmBackup.js:371:9)"
      }
    },

any ideas what this error means, what could be the root cause? I've googled and searched in the forum for it, but couldn't find anything

cheers, Ringo

olivierlambert

@julien-f or @Darkbeldin might answer that

julien-f

@vmpr no writers left means all targets have failed, what's important is the reasons of these targets.
Here we can see Unknown system error -122: Unknown system error -122, write.

What type of remote are you using?

vmpr

@julien-f its a NFS remote

vmpr

@julien-f any ideas how to debug that?

julien-f

@vmpr Not much, this is a system issue which means that it's either due to your NFS server or to the connection between your XO and the NFS server.

First, try to unmount the share and restart xo-server, if it keeps happening, restart the whole machine on which XO is.

If that does not help, you should mount the share manually and do some tests directly on it.