http time out error on backup

acebmxer

I have opened a support ticket and tunnel is open - Ticket#7749796

here is a section of the error log. I have rebooted XOA and rebooted the Proxy but yet the job still remains hung. Maybe i didn't reboot them in the correct order? As stated support ticket and tunnel are open.

   {
         "id": "1766901730734",
         "message": "snapshot",
         "start": 1766901730734,
         "status": "success",
         "end": 1766901734483,
         "result": "a995b25e-ab77-a217-fccd-7e8a1d06df57"
       },
       {
         "data": {
           "id": "3022703c-5029-4ac3-bda1-3a76c6634a39",
           "isFull": true,
           "type": "remote"
         },
         "id": "1766901734494",
         "message": "export",
         "start": 1766901734494,
         "status": "interrupted",
         "tasks": [
           {
             "id": "1766901737246",
             "message": "transfer",
             "start": 1766901737246,
             "status": "interrupted"
           }
         ]
       }
     ],
     "infos": [
       {
         "message": "Transfer data using NBD"
       }
     ]
   }
 ],
 "end": 1766901871318,
 "result": {
   "url": "https://10.120.20.119/api/v1",
   "message": "HTTP connection has timed out",
   "name": "Error",
   "stack": "Error: HTTP connection has timed out\n    at ClientRequest.<anonymous> (/usr/local/lib/node_modules/xo-server/node_modules/http-request-plus/index.js:61:25)\n    at ClientRequest.emit (node:events:518:28)\n    at ClientRequest.patchedEmit [as emit] (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:52:17)\n    at TLSSocket.emitRequestTimeout (node:_http_client:849:9)\n    at Object.onceWrapper (node:events:632:28)\n    at TLSSocket.emit (node:events:530:35)\n    at TLSSocket.patchedEmit [as emit] (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:52:17)\n    at TLSSocket.Socket._onTimeout (node:net:595:8)\n    at listOnTimeout (node:internal/timers:581:17)\n    at processTimers (node:internal/timers:519:7)"

schedule.runSequence
{
  "schedules": [
    "3a677031-4eb9-427a-99fc-22afa411b745",
    "cadef7b9-2541-4850-9bfd-871d9602c6bb",
    "599d9321-7081-4840-9617-d31762d4764c",
    "828b5e60-4302-455c-94f1-479a8001f955"
  ]
}
{
  "url": "https://10.120.20.119/api/v1",
  "message": "HTTP connection has timed out",
  "name": "Error",
  "stack": "Error: HTTP connection has timed out
    at ClientRequest.<anonymous> (/usr/local/lib/node_modules/xo-server/node_modules/http-request-plus/index.js:61:25)
    at ClientRequest.emit (node:events:518:28)
    at ClientRequest.patchedEmit [as emit] (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:52:17)
    at TLSSocket.emitRequestTimeout (node:_http_client:849:9)
    at Object.onceWrapper (node:events:632:28)
    at TLSSocket.emit (node:events:530:35)
    at TLSSocket.patchedEmit [as emit] (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:52:17)
    at TLSSocket.Socket._onTimeout (node:net:595:8)
    at listOnTimeout (node:internal/timers:581:17)
    at processTimers (node:internal/timers:519:7)"

backupNg.runJob
{
  "id": "bb0adfc7-93b9-45c9-a90f-7aa327f95a05",
  "schedule": "3a677031-4eb9-427a-99fc-22afa411b745",
  "vms": [
    "4c2fb974-f2d5-b0fb-7385-1683e0275ef2",
    "025f71ff-ecb2-9d0c-7692-f7005d22c659",
    "78aecc3d-4817-a11c-04b4-9d0f49f8736e",
    "9f36a8c5-c988-8546-b0f9-30ac9d0a5015",
    "db89c6f0-3330-7180-b5e4-7f2f36a241e6",
    "e5c656bb-8bb2-461e-a801-402ae150d022"
  ]
}
{
  "code": -32000,
  "data": {
    "jobId": "bb0adfc7-93b9-45c9-a90f-7aa327f95a05",
    "stack": "Error: job is already running
    at file:///usr/local/lib/node_modules/@xen-orchestra/proxy/app/mixins/backups.mjs:96:25
    at file:///usr/local/lib/node_modules/@xen-orchestra/proxy/app/mixins/backups.mjs:114:20
    at processTicksAndRejections (node:internal/process/task_queues:95:5)"
  },

acebmxer

Restarting tool stack on master host of remote pool cleared the stuck job and backup was able run failed vms.

Support ticket still open fyi.

myles3

This kind of HTTP timeout during backups usually isn’t a hard failure with XCP-NG itself. In many cases it comes down to how Xen Orchestra is communicating with the host while exporting the VM, especially for larger backups.

A few things that are commonly worth checking:

Versions – make sure both XCP-NG and Xen Orchestra are up to date. Some timeout-related issues have been improved over time.
Network stability – if the backup traffic runs over a slower or slightly unstable link, long exports can trigger HTTP timeouts. This shows up more often with big VMs.
Backup type – full backups are more prone to this than delta/incremental backups, particularly if the VM disks are large.
Compression settings – sometimes compression can cause long pauses during export. Disabling it or switching methods has helped others avoid timeouts.
Logs – checking XO logs and /var/log/xensource.log on the host around the failure usually gives a clearer hint about what’s actually stalling.

If your backup traffic is routed through extra layers (firewalls, VPNs, or even things like cheap shared proxies for testing or monitoring), it’s also worth temporarily bypassing those to rule out added latency or connection drops.

If you can share your XO/XCP-NG versions, backup type, and roughly how large the VMs are, it’ll be easier for others here to pinpoint the cause.

acebmxer

@myles3

Hello Myles3. This post is over 20 days old. The issue was since been resolved by support. I dont remember the exact solution off the top of my head with out looking it up.