Backup failing with "Error: HTTP connection has timed out"

[lawrencesystems]

I am running xo-server 5.84.1 xo-web 5.90.0 built from source and since the latest update I am sometimes getting a Error: HTTP connection has timed out. I never had this error come up prior to the upgrade and It's not consistent and running the same backup a few more times does not cause it to fail each time. Not sure if it might be the system getting a bit overloaded when running a while running a larger the backup, it's never occurred on small VM's or delta backups. The XO vm is running on a Xeon E5-2670 0 @ 2.60GHz with 16 cores assigned & 4GB of memory.

{
  "data": {
    "mode": "full",
    "reportWhen": "failure"
  },
  "id": "1638444000006",
  "jobId": "d1abe537-771e-4b62-8c3c-05d08c6854e8",
  "jobName": "Graylog",
  "message": "backup",
  "scheduleId": "6ea649f1-a45f-4665-af52-71ccad7edefe",
  "start": 1638444000006,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "62570b52-e56b-a79b-ca5f-88803038532f"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "62570b52-e56b-a79b-ca5f-88803038532f"
      },
      "id": "1638444000875",
      "message": "backup VM",
      "start": 1638444000875,
      "status": "failure",
      "tasks": [
        {
          "id": "1638444015767",
          "message": "snapshot",
          "start": 1638444015767,
          "status": "success",
          "end": 1638444017382,
          "result": "ada208e5-cb2f-5e4f-f6d1-19c64e6c8026"
        },
        {
          "data": {
            "id": "b0323f4d-1828-4ad1-b9bd-550f38ff6cfa",
            "type": "remote",
            "isFull": true
          },
          "id": "1638444017480",
          "message": "export",
          "start": 1638444017480,
          "status": "failure",
          "tasks": [
            {
              "id": "1638444017488",
              "message": "transfer",
              "start": 1638444017488,
              "status": "failure",
              "end": 1638450296391,
              "result": {
                "canceled": false,
                "method": "GET",
                "url": "https://192.168.3.92/export/?ref=OpaqueRef%3A0ca54512-7a8c-452b-af08-bb18c83a0138&use_compression=zstd&session_id=OpaqueRef%3A09c8ef17-0f39-464a-a2c2-e3d9c0a51499&task_id=OpaqueRef%3Ae16c33e8-8360-4848-8a29-20ae6027e103",
                "timeout": true,
                "message": "HTTP connection has timed out",
                "name": "Error",
                "stack": "Error: HTTP connection has timed out\n    at IncomingMessage.emitAbortedError (/etc/xo/xo-builds/xen-orchestra-202112010617/node_modules/http-request-plus/index.js:83:19)\n    at Object.onceWrapper (events.js:519:28)\n    at IncomingMessage.emit (events.js:400:28)\n    at IncomingMessage.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at IncomingMessage.emit (domain.js:475:12)\n    at TLSSocket.socketCloseListener (_http_client.js:432:11)\n    at TLSSocket.emit (events.js:412:35)\n    at TLSSocket.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at TLSSocket.emit (domain.js:475:12)\n    at net.js:686:12"
              }
            }
          ],
          "end": 1638450296392,
          "result": {
            "canceled": false,
            "method": "GET",
            "url": "https://192.168.3.92/export/?ref=OpaqueRef%3A0ca54512-7a8c-452b-af08-bb18c83a0138&use_compression=zstd&session_id=OpaqueRef%3A09c8ef17-0f39-464a-a2c2-e3d9c0a51499&task_id=OpaqueRef%3Ae16c33e8-8360-4848-8a29-20ae6027e103",
            "timeout": true,
            "message": "HTTP connection has timed out",
            "name": "Error",
            "stack": "Error: HTTP connection has timed out\n    at IncomingMessage.emitAbortedError (/etc/xo/xo-builds/xen-orchestra-202112010617/node_modules/http-request-plus/index.js:83:19)\n    at Object.onceWrapper (events.js:519:28)\n    at IncomingMessage.emit (events.js:400:28)\n    at IncomingMessage.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at IncomingMessage.emit (domain.js:475:12)\n    at TLSSocket.socketCloseListener (_http_client.js:432:11)\n    at TLSSocket.emit (events.js:412:35)\n    at TLSSocket.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at TLSSocket.emit (domain.js:475:12)\n    at net.js:686:12"
          }
        }
      ],
      "end": 1638450311244,
      "result": {
        "canceled": false,
        "method": "GET",
        "url": "https://192.168.3.92/export/?ref=OpaqueRef%3A0ca54512-7a8c-452b-af08-bb18c83a0138&use_compression=zstd&session_id=OpaqueRef%3A09c8ef17-0f39-464a-a2c2-e3d9c0a51499&task_id=OpaqueRef%3Ae16c33e8-8360-4848-8a29-20ae6027e103",
        "timeout": true,
        "message": "HTTP connection has timed out",
        "name": "Error",
        "stack": "Error: HTTP connection has timed out\n    at IncomingMessage.emitAbortedError (/etc/xo/xo-builds/xen-orchestra-202112010617/node_modules/http-request-plus/index.js:83:19)\n    at Object.onceWrapper (events.js:519:28)\n    at IncomingMessage.emit (events.js:400:28)\n    at IncomingMessage.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at IncomingMessage.emit (domain.js:475:12)\n    at TLSSocket.socketCloseListener (_http_client.js:432:11)\n    at TLSSocket.emit (events.js:412:35)\n    at TLSSocket.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n    at TLSSocket.emit (domain.js:475:12)\n    at net.js:686:12"
      }
    }
  ],
  "end": 1638450311245
}

[lawrencesystems]

Just noticed there was another update that makes a few changes to backup, I did not see the same error message that I am getting but I updated to the latest version and I will do some more testing to see if the error message comes back.
https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.md

[Danp]

Was just about to post suggesting that you update again. Hopefully the latest fixes address your issue.

[olivierlambert]

Also HTTP connection has timed out might be a real network issue, but first let's rule out other possibilities by using the latest commit on master

[iLix]

I am also seeing this on a large vm(almost 2tb), the sub 200GB VM’s backup fine. I can’t see any errors on the switches. I will add more details about my setup tomorrow.

[lawrencesystems]

Since the update I have run full backup that have taken over an hour to complete due to their size and many smaller delta backups and the error has not come up again. I assume whatever was causing the error was fixed in the xo-server 5.84.2 update.