Backup failed with message "HTTP request has been canceled"

erwand

Hi all, i've got several backup failed with this kind of message, when i try to launch the backup manually it work but with many VM in the backup it failed for some (9) and work for most (83)

any idea ?

thanks

{
      "data": {
        "type": "VM",
        "id": "9c0d7ef5-0bd2-dbd0-ea08-4d27d2a49a8f"
      },
      "id": "1623714947448",
      "message": "backup VM",
      "start": 1623714947448,
      "status": "failure",
      "tasks": [
        {
          "id": "1623714947719",
          "message": "snapshot",
          "start": 1623714947719,
          "status": "success",
          "end": 1623715166291,
          "result": "b0248f70-64d4-3ac6-d2a8-5ae92bc334b5"
        },
        {
          "data": {
            "id": "4433dda6-d07a-42ae-9732-f40156bf71e7",
            "isFull": false,
            "type": "remote"
          },
          "id": "1623715166292",
          "message": "export",
          "start": 1623715166292,
          "status": "failure",
          "tasks": [
            {
              "id": "1623715166378",
              "message": "transfer",
              "start": 1623715166378,
              "status": "failure",
              "end": 1623715466379,
              "result": {
                "canceled": true,
                "method": "GET",
                "url": "https://10.89.*.*/export_raw_vdi/?format=vhd&vdi=OpaqueRef%3A3b3fc0fe-9599-4eb0-8dee-88c6f41ed5b2&base=OpaqueRef%3A04eb8c1a-97f7-4737-986d-45d615e55a8e&session_id=OpaqueRef%3Abbfab2de-0aa5-42f8-816e-541675874a9c&task_id=OpaqueRef%3A772c8265-4b67-4daf-ab11-c43e39d8653e",
                "message": "HTTP request has been canceled",
                "name": "Error",
                "stack": "Error: HTTP request has been canceled\n    at IncomingMessage.emitAbortedError (/opt/xo/xo-builds/xen-orchestra-202106101449/node_modules/http-request-plus/index.js:75:19)\n    at Object.onceWrapper (events.js:482:28)\n    at IncomingMessage.emit (events.js:376:20)\n    at IncomingMessage.patchedEmit (/opt/xo/xo-builds/xen-orchestra-202106101449/@xen-orchestra/log/configure.js:93:17)\n    at IncomingMessage.emit (domain.js:470:12)\n    at TLSSocket.socketCloseListener (_http_client.js:432:11)\n    at TLSSocket.emit (events.js:388:22)\n    at TLSSocket.patchedEmit (/opt/xo/xo-builds/xen-orchestra-202106101449/@xen-orchestra/log/configure.js:93:17)\n    at TLSSocket.emit (domain.js:470:12)\n    at net.js:673:12"
              }
            }
          ],
          "end": 1623715466379,
          "result": {
            "canceled": true,
            "method": "GET",
            "url": "https://10.89.*.*/export_raw_vdi/?format=vhd&vdi=OpaqueRef%3A3b3fc0fe-9599-4eb0-8dee-88c6f41ed5b2&base=OpaqueRef%3A04eb8c1a-97f7-4737-986d-45d615e55a8e&session_id=OpaqueRef%3Abbfab2de-0aa5-42f8-816e-541675874a9c&task_id=OpaqueRef%3A772c8265-4b67-4daf-ab11-c43e39d8653e",
            "message": "HTTP request has been canceled",
            "name": "Error",
            "stack": "Error: HTTP request has been canceled\n    at IncomingMessage.emitAbortedError (/opt/xo/xo-builds/xen-orchestra-202106101449/node_modules/http-request-plus/index.js:75:19)\n    at Object.onceWrapper (events.js:482:28)\n    at IncomingMessage.emit (events.js:376:20)\n    at IncomingMessage.patchedEmit (/opt/xo/xo-builds/xen-orchestra-202106101449/@xen-orchestra/log/configure.js:93:17)\n    at IncomingMessage.emit (domain.js:470:12)\n    at TLSSocket.socketCloseListener (_http_client.js:432:11)\n    at TLSSocket.emit (events.js:388:22)\n    at TLSSocket.patchedEmit (/opt/xo/xo-builds/xen-orchestra-202106101449/@xen-orchestra/log/configure.js:93:17)\n    at TLSSocket.emit (domain.js:470:12)\n    at net.js:673:12"
          }
        }
      ],
      "end": 1623715866307,
      "result": {
        "message": "all targets have failed, step: writer.transfer()",
        "name": "Error",
        "stack": "Error: all targets have failed, step: writer.transfer()\n    at VmBackup._callWriters (/opt/xo/xo-builds/xen-orchestra-202106101449/@xen-orchestra/backups/_VmBackup.js:118:13)\n    at async VmBackup._copyDelta (/opt/xo/xo-builds/xen-orchestra-202106101449/@xen-orchestra/backups/_VmBackup.js:190:5)\n    at async VmBackup.run (/opt/xo/xo-builds/xen-orchestra-202106101449/@xen-orchestra/backups/_VmBackup.js:371:9)"
      }
    }

olivierlambert

Hmm weird, pinging @julien-f

Maybe too many requests on a XAPI?

julien-f

The message HTTP request has been canceled is not correct, it should say HTTP connection abruptly closed, it's an minor error which will be fixed soon.

It might indeed be triggered by too many connections from XO to the host, in that case, please make sure your XO is up-to-date with this commit: https://github.com/vatesfr/xen-orchestra/commit/51f013851d6d72275a16d37cd92b0ff88e7046d8

0

julien-f committed to vatesfr/xen-orchestra

feat(xen-api): limit concurrent calls to 20

Fixes xoa-support#3767

Can be changed via `callConcurrency` option.

erwand

@julien-f

thanks for your reply, where i can find the parameter "callconcurrency" you talk about in your post ?

olivierlambert

It will default to 20 if you don't set it up That should be enough to make initial tests

erwand

@olivierlambert I've update to latest XOA and same thing happen today ? Anything i can check to solve ?

olivierlambert

Can you be more specific? Are you using XOA or XO from the sources?

erwand

@olivierlambert Im using XO installed using the script provide by

olivierlambert

Please check you are on latest master commit. 3rd party scripts are not supported, but just tell us on which commit are you are also double check everything was rebuilt correctly.

erwand

@olivierlambert
I've double check and i'm on the laster master check out on 15/06/2021 and everything rebuild

olivierlambert

Can you tell us which commit are you exactly? (just to be sure).

We might want to test lower values then.

erwand

@olivierlambert I think: xen-orchestra-202106151431

olivierlambert

The commit id please 56e4847b6bb85da8ae2dc09e8e9fb7a0db36070a

You can have it by doing git rev-parse HEAD

erwand

@olivierlambert said in Backup failed with message "HTTP request has been canceled":

git rev-parse HEAD

51f013851d6d72275a16d37cd92b0ff88e7046d8

olivierlambert

@erwand said in Backup failed with message "HTTP request has been canceled":

51f013851d6d72275a16d37cd92b0ff88e7046d8

pinging @julien-f

julien-f

@erwand I don't know how this script works and if the libraries are installed from the repo itself or are coming from npmjs.org, because this change in xen-api has not been published.

You can check by doing something like this:

> cd packages/xo-server
> node -p 'require.resolve("xen-api")'
<root dir>/xen-orchestra/packages/xen-api/dist/index.js

Also, this error simply indicates that the connection was interrupted between XO and the host during the export of the VDI, this does not necessarily indicate an XO bug, please check your host's logs.

erwand

@julien-f said in Backup failed with message "HTTP request has been canceled":

node -p 'require.resolve("xen-api")'

Hi all,

I think the problem is fixed now. thanks for your help