Backup failing with "Error: HTTP connection has timed out"
-
I am running xo-server 5.84.1 xo-web 5.90.0 built from source and since the latest update I am sometimes getting a Error: HTTP connection has timed out. I never had this error come up prior to the upgrade and It's not consistent and running the same backup a few more times does not cause it to fail each time. Not sure if it might be the system getting a bit overloaded when running a while running a larger the backup, it's never occurred on small VM's or delta backups. The XO vm is running on a Xeon E5-2670 0 @ 2.60GHz with 16 cores assigned & 4GB of memory.
{ "data": { "mode": "full", "reportWhen": "failure" }, "id": "1638444000006", "jobId": "d1abe537-771e-4b62-8c3c-05d08c6854e8", "jobName": "Graylog", "message": "backup", "scheduleId": "6ea649f1-a45f-4665-af52-71ccad7edefe", "start": 1638444000006, "status": "failure", "infos": [ { "data": { "vms": [ "62570b52-e56b-a79b-ca5f-88803038532f" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "62570b52-e56b-a79b-ca5f-88803038532f" }, "id": "1638444000875", "message": "backup VM", "start": 1638444000875, "status": "failure", "tasks": [ { "id": "1638444015767", "message": "snapshot", "start": 1638444015767, "status": "success", "end": 1638444017382, "result": "ada208e5-cb2f-5e4f-f6d1-19c64e6c8026" }, { "data": { "id": "b0323f4d-1828-4ad1-b9bd-550f38ff6cfa", "type": "remote", "isFull": true }, "id": "1638444017480", "message": "export", "start": 1638444017480, "status": "failure", "tasks": [ { "id": "1638444017488", "message": "transfer", "start": 1638444017488, "status": "failure", "end": 1638450296391, "result": { "canceled": false, "method": "GET", "url": "https://192.168.3.92/export/?ref=OpaqueRef%3A0ca54512-7a8c-452b-af08-bb18c83a0138&use_compression=zstd&session_id=OpaqueRef%3A09c8ef17-0f39-464a-a2c2-e3d9c0a51499&task_id=OpaqueRef%3Ae16c33e8-8360-4848-8a29-20ae6027e103", "timeout": true, "message": "HTTP connection has timed out", "name": "Error", "stack": "Error: HTTP connection has timed out\n at IncomingMessage.emitAbortedError (/etc/xo/xo-builds/xen-orchestra-202112010617/node_modules/http-request-plus/index.js:83:19)\n at Object.onceWrapper (events.js:519:28)\n at IncomingMessage.emit (events.js:400:28)\n at IncomingMessage.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n at IncomingMessage.emit (domain.js:475:12)\n at TLSSocket.socketCloseListener (_http_client.js:432:11)\n at TLSSocket.emit (events.js:412:35)\n at TLSSocket.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n at TLSSocket.emit (domain.js:475:12)\n at net.js:686:12" } } ], "end": 1638450296392, "result": { "canceled": false, "method": "GET", "url": "https://192.168.3.92/export/?ref=OpaqueRef%3A0ca54512-7a8c-452b-af08-bb18c83a0138&use_compression=zstd&session_id=OpaqueRef%3A09c8ef17-0f39-464a-a2c2-e3d9c0a51499&task_id=OpaqueRef%3Ae16c33e8-8360-4848-8a29-20ae6027e103", "timeout": true, "message": "HTTP connection has timed out", "name": "Error", "stack": "Error: HTTP connection has timed out\n at IncomingMessage.emitAbortedError (/etc/xo/xo-builds/xen-orchestra-202112010617/node_modules/http-request-plus/index.js:83:19)\n at Object.onceWrapper (events.js:519:28)\n at IncomingMessage.emit (events.js:400:28)\n at IncomingMessage.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n at IncomingMessage.emit (domain.js:475:12)\n at TLSSocket.socketCloseListener (_http_client.js:432:11)\n at TLSSocket.emit (events.js:412:35)\n at TLSSocket.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n at TLSSocket.emit (domain.js:475:12)\n at net.js:686:12" } } ], "end": 1638450311244, "result": { "canceled": false, "method": "GET", "url": "https://192.168.3.92/export/?ref=OpaqueRef%3A0ca54512-7a8c-452b-af08-bb18c83a0138&use_compression=zstd&session_id=OpaqueRef%3A09c8ef17-0f39-464a-a2c2-e3d9c0a51499&task_id=OpaqueRef%3Ae16c33e8-8360-4848-8a29-20ae6027e103", "timeout": true, "message": "HTTP connection has timed out", "name": "Error", "stack": "Error: HTTP connection has timed out\n at IncomingMessage.emitAbortedError (/etc/xo/xo-builds/xen-orchestra-202112010617/node_modules/http-request-plus/index.js:83:19)\n at Object.onceWrapper (events.js:519:28)\n at IncomingMessage.emit (events.js:400:28)\n at IncomingMessage.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n at IncomingMessage.emit (domain.js:475:12)\n at TLSSocket.socketCloseListener (_http_client.js:432:11)\n at TLSSocket.emit (events.js:412:35)\n at TLSSocket.patchedEmit (/etc/xo/xo-builds/xen-orchestra-202112010617/@xen-orchestra/log/configure.js:118:17)\n at TLSSocket.emit (domain.js:475:12)\n at net.js:686:12" } } ], "end": 1638450311245 } -
Just noticed there was another update that makes a few changes to backup, I did not see the same error message that I am getting but I updated to the latest version and I will do some more testing to see if the error message comes back.
https://github.com/vatesfr/xen-orchestra/blob/master/CHANGELOG.md -
Was just about to post suggesting that you update again. Hopefully the latest fixes address your issue.
-
Also
HTTP connection has timed outmight be a real network issue, but first let's rule out other possibilities by using the latest commit on master
-
I am also seeing this on a large vm(almost 2tb), the sub 200GB VM’s backup fine. I can’t see any errors on the switches. I will add more details about my setup tomorrow.
-
Since the update I have run full backup that have taken over an hour to complete due to their size and many smaller delta backups and the error has not come up again. I assume whatever was causing the error was fixed in the xo-server 5.84.2 update.
-
N northportio referenced this topic on
