Problem: XO fails to backup a specific VM
xo-server: 5.87.0

I've been trying to backup a specific VM using delta backups and I've been met with HTTP timeouts only when the backup is about to finish. The backup page shows as failed but the backup task is still running:

Here's a little info about my setup (things I've tried to fix/isolate the issue after):

VM setup:

• OS: WIndows Server 2016 Standard (guest tools installed)
• Disks: 2
  
  

XCP-NG Hosts:
Version: 8.2.1

• 2 Hosts (2x Dell R640 with Xeon SIlver 4108 & 64GB RAM)
• 1 Resource Pool
• Shared NFS SSD Storage connected to hosts at 10G on separate LAN

Things tried:

• I've tried backing up to a remote over 1G NFS & SMB
• I've created a separate backup task for this specific vm
• I've set a timeout of 72 hours
• I've tried changing management interfaces from 1G network to a separate 10G network
• I've tried backing up to another remote over 10G NFS & SMB
• I've updated XO (built from sources) & the OS as that seemed to resolve itself after updating in this thread by @lawrencesystems but didn't resolve my issue
• I've tried the quick deploy XO vm

The above yields two other vms backing up successfully in every case while the specific VM in question failing to back up.

Here is the log:

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1646328525726",
  "jobId": "933a38ad-48d8-48ed-bbec-e037639c9d13",
  "jobName": "Sage",
  "message": "backup",
  "scheduleId": "c7da8664-c4a3-41f3-82a6-c065da75b3db",
  "start": 1646328525726,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "13e70ae1-3223-7b94-dc59-c31f7416af82"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "13e70ae1-3223-7b94-dc59-c31f7416af82"
      },
      "id": "1646328526912",
      "message": "backup VM",
      "start": 1646328526912,
      "status": "failure",
      "tasks": [
        {
          "id": "1646328527437",
          "message": "snapshot",
          "start": 1646328527437,
          "status": "success",
          "end": 1646328533606,
          "result": "0dd66066-38bd-247b-88fc-613c99c7fe91"
        },
        {
          "data": {
            "id": "14021778-5160-457b-b949-a38c6e84ccc8",
            "isFull": true,
            "type": "remote"
          },
          "id": "1646328533607",
          "message": "export",
          "start": 1646328533607,
          "status": "failure",
          "tasks": [
            {
              "id": "1646328533647",
              "message": "transfer",
              "start": 1646328533647,
              "status": "failure",
              "end": 1646331954922,
              "result": {
                "canceled": false,
                "method": "GET",
                "url": "https://10.131.200.3/export_raw_vdi/?format=vhd&vdi=OpaqueRef%3A0ccbbba4-1449-4184-be31-409b9266ba62&session_id=OpaqueRef%3A1f4f5498-b540-40b9-8ada-809a5937b90c&task_id=OpaqueRef%3Ab4256fe3-de2f-498e-9539-aed9a2f56159",
                "timeout": true,
                "message": "HTTP connection has timed out",
                "name": "Error",
                "stack": "Error: HTTP connection has timed out\n    at IncomingMessage.emitAbortedError (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/node_modules/http-request-plus/index.js:83:19)\n    at Object.onceWrapper (node:events:509:28)\n    at IncomingMessage.emit (node:events:390:28)\n    at IncomingMessage.patchedEmit (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:118:17)\n    at IncomingMessage.emit (node:domain:475:12)\n    at IncomingMessage._destroy (node:_http_incoming:179:10)\n    at _destroy (node:internal/streams/destroy:102:25)\n    at IncomingMessage.destroy (node:internal/streams/destroy:64:5)\n    at TLSSocket.socketCloseListener (node:_http_client:407:11)\n    at TLSSocket.emit (node:events:402:35)"
              }
            }
          ],
          "end": 1646331954922,
          "result": {
            "canceled": false,
            "method": "GET",
            "url": "https://10.131.200.3/export_raw_vdi/?format=vhd&vdi=OpaqueRef%3A0ccbbba4-1449-4184-be31-409b9266ba62&session_id=OpaqueRef%3A1f4f5498-b540-40b9-8ada-809a5937b90c&task_id=OpaqueRef%3Ab4256fe3-de2f-498e-9539-aed9a2f56159",
            "timeout": true,
            "message": "HTTP connection has timed out",
            "name": "Error",
            "stack": "Error: HTTP connection has timed out\n    at IncomingMessage.emitAbortedError (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/node_modules/http-request-plus/index.js:83:19)\n    at Object.onceWrapper (node:events:509:28)\n    at IncomingMessage.emit (node:events:390:28)\n    at IncomingMessage.patchedEmit (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:118:17)\n    at IncomingMessage.emit (node:domain:475:12)\n    at IncomingMessage._destroy (node:_http_incoming:179:10)\n    at _destroy (node:internal/streams/destroy:102:25)\n    at IncomingMessage.destroy (node:internal/streams/destroy:64:5)\n    at TLSSocket.socketCloseListener (node:_http_client:407:11)\n    at TLSSocket.emit (node:events:402:35)"
          }
        }
      ],
      "end": 1646332004582,
      "result": {
        "canceled": false,
        "method": "GET",
        "url": "https://10.131.200.3/export_raw_vdi/?format=vhd&vdi=OpaqueRef%3A0ccbbba4-1449-4184-be31-409b9266ba62&session_id=OpaqueRef%3A1f4f5498-b540-40b9-8ada-809a5937b90c&task_id=OpaqueRef%3Ab4256fe3-de2f-498e-9539-aed9a2f56159",
        "timeout": true,
        "message": "HTTP connection has timed out",
        "name": "Error",
        "stack": "Error: HTTP connection has timed out\n    at IncomingMessage.emitAbortedError (/usr/local/lib/node_modules/xo-server/node_modules/xen-api/node_modules/http-request-plus/index.js:83:19)\n    at Object.onceWrapper (node:events:509:28)\n    at IncomingMessage.emit (node:events:390:28)\n    at IncomingMessage.patchedEmit (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/log/configure.js:118:17)\n    at IncomingMessage.emit (node:domain:475:12)\n    at IncomingMessage._destroy (node:_http_incoming:179:10)\n    at _destroy (node:internal/streams/destroy:102:25)\n    at IncomingMessage.destroy (node:internal/streams/destroy:64:5)\n    at TLSSocket.socketCloseListener (node:_http_client:407:11)\n    at TLSSocket.emit (node:events:402:35)"
      }
    }
  ],
  "end": 1646332004583
}

My apologies for the lengthy post but I've tried just about every avenue I can think of. I'm hoping you may see something that I'm not. Thanks.