XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Error: invalid HTTP header in response body

    Scheduled Pinned Locked Moved Backup
    19 Posts 5 Posters 546 Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      peo @manilx
      last edited by

      @manilx Does your backup job succeed if you run it manually after a failure (as I have figured out is happening for me) ?
      I did some more tests and found out that the setting "Purge snapshot data when using CBT" under advanced backup settings might be the cause of the problem. The only thing is that with this setting disabled, every machine being backed up will keep a space-wasting snapshot in the same storage repository as the snapshotted disk.

      M 1 Reply Last reply Reply Quote 0
      • M Offline
        manilx @peo
        last edited by

        @peo I don't have this setting set. The errors appear inconsistently.

        P 1 Reply Last reply Reply Quote 0
        • P Offline
          peo @manilx
          last edited by

          @manilx as you describe it, it might be a problem with the remote NFS storage, but still, the it would be nice if that error message could be a bit clearer.

          M 1 Reply Last reply Reply Quote 0
          • M Offline
            manilx @peo
            last edited by

            @peo Nope. Remote has no issue whatsoever.
            The issue is with the backup code because this is what changed. Worked before for many months without issues.
            And it's still working at the office on stable XOA.

            1 Reply Last reply Reply Quote 0
            • M Offline
              markxc
              last edited by

              dear all i am facing the same issue

              Error: invalid HTTP header in response body

              {
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1749107336350",
  "jobId": "89511625-67f3-46be-98fe-8d7a7584386f",
  "jobName": "All-2-50G-Delta",
  "message": "backup",
  "scheduleId": "405be84c-8b49-408a-871e-b21c211ee326",
  "start": 1749107336350,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "20e7d42f-bf69-186d-ec65-82e08230e557"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "20e7d42f-bf69-186d-ec65-82e08230e557",
        "name_label": "PfSense_Ip_92_61_60_23_EXT_172_16_16_99_INT_2022_03_03_07_24_57"
      },
      "id": "1749107339106",
      "message": "backup VM",
      "start": 1749107339106,
      "status": "failure",
      "tasks": [
        {
          "id": "1749107339113",
          "message": "clean-vm",
          "start": 1749107339113,
          "status": "success",
          "end": 1749107339118,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1749107339665",
          "message": "snapshot",
          "start": 1749107339665,
          "status": "success",
          "end": 1749107341927,
          "result": "02828209-8cc6-8694-8fdd-af5aa04d5b90"
        },
        {
          "data": {
            "id": "455eee8b-49ca-4552-9431-ebb417fbd8f8",
            "isFull": true,
            "type": "remote"
          },
          "id": "1749107341927:0",
          "message": "export",
          "start": 1749107341927,
          "status": "success",
          "tasks": [
            {
              "id": "1749107347491",
              "message": "clean-vm",
              "start": 1749107347491,
              "status": "success",
              "end": 1749107347494,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1749107347496
        }
      ],
      "end": 1749107347496,
      "result": {
        "message": "invalid HTTP header in response body",
        "name": "Error",
        "stack": "Error: invalid HTTP header in response body\n    at checkVdiExport (file:///opt/xo/xo-builds/xen-orchestra-202503211145/@xen-orchestra/xapi/vdi.mjs:40:19)\n    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)\n    at async Xapi.exportContent (file:///opt/xo/xo-builds/xen-orchestra-202503211145/@xen-orchestra/xapi/vdi.mjs:326:7)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202503211145/@xen-orchestra/backups/_incrementalVm.mjs:56:34\n    at async Promise.all (index 1)\n    at async cancelableMap (file:///opt/xo/xo-builds/xen-orchestra-202503211145/@xen-orchestra/backups/_cancelableMap.mjs:11:12)\n    at async exportIncrementalVm (file:///opt/xo/xo-builds/xen-orchestra-202503211145/@xen-orchestra/backups/_incrementalVm.mjs:25:3)\n    at async IncrementalXapiVmBackupRunner._copy (file:///opt/xo/xo-builds/xen-orchestra-202503211145/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:44:25)\n    at async IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202503211145/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:396:9)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202503211145/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
      }
    }
  ],
  "end": 1749107347497
}
              1 Reply Last reply Reply Quote 0
              • P Offline
                peo @olivierlambert
                last edited by

                @olivierlambert Have you heard of any progress in this case ? I found out that I got rid of the error when I disabled "Purge snapshot data when using CBT" on all my backups, so the backups are no longer failing, except from the Docker VM I have mentioned in another case "Success or failure" (or similar), which indicates failure, but restores successfully (which is weird).

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  I'm not aware of similar issues for our XOA users 😕

                  @florent does it ring any bell?

                  M 1 Reply Last reply Reply Quote 0
                  • M Offline
                    markxc @olivierlambert
                    last edited by

                    @olivierlambert

                    today same issue again

                    Snapshot
                    Start: 2025-06-12 07:16
                    End: 2025-06-12 07:16
                    50G-Backup-Delta
                    Start: 2025-06-12 07:16
                    End: 2025-06-12 07:16
                    Duration: a few seconds
                    Start: 2025-06-12 07:16
                    End: 2025-06-12 07:16
                    Duration: a few seconds
                    Error: invalid HTTP header in response body
                    Type: full

                    vm is a PFSense

                    with template Other install media

                    Management agent 6.2.0-76888 detected
                    Hardware virtualization with paravirtualization drivers enabled (PVHVM)

                    florentF 1 Reply Last reply Reply Quote 0
                    • florentF Offline
                      florent Vates 🪐 XO Team @markxc
                      last edited by

                      @markxc if you are on master you should have something on the xo logs , starting with invalid HTTP header in response body

                      Can you check and tell us what is the error message attached ?

                      M 1 Reply Last reply Reply Quote 0
                      • M Offline
                        markxc @florent
                        last edited by

                        @florent

                        un 12 07:16:23 xenorchestra xo-server[1679463]: 2025-06-12T05:16:23.729Z xo:xapi:vdi WARN invalid HTTP header in response body {
Jun 12 07:16:23 xenorchestra xo-server[1679463]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Jun 12 07:16:23 xenorchestra xo-server[1679463]:     'content-length: 266\r\n' +
Jun 12 07:16:23 xenorchestra xo-server[1679463]:     'content-type:text/html\r\n' +
Jun 12 07:16:23 xenorchestra xo-server[1679463]:     'connection:close\r\n' +
Jun 12 07:16:23 xenorchestra xo-server[1679463]:     'cache-control:no-cache, no-store\r\n' +
Jun 12 07:16:23 xenorchestra xo-server[1679463]:     '\r\n' +
Jun 12 07:16:23 xenorchestra xo-server[1679463]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a whi>Jun 12 07:16:23 xenorchestra xo-server[1679463]: }
Jun 12 07:16:28 xenorchestra xo-server[1679463]: 2025-06-12T05:16:28.522Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
Jun 12 07:16:28 xenorchestra xo-server[1679463]:   path: '/xo-vm-backups/04d4ce05-81fd-e2b6-ef0d-4e9b91f3ffb1/20250610T051506Z.json',
Jun 12 07:16:28 xenorchestra xo-server[1679463]:   actual: 1116023296,
Jun 12 07:16:28 xenorchestra xo-server[1679463]:   expected: 1116023808
Jun 12 07:16:28 xenorchestra xo-server[1679463]: }
Jun 12 07:16:29 xenorchestra xo-server[1675344]: 2025-06-12T05:16:29.954Z xo:backups:mergeWorker INFO merge in progress {
Jun 12 07:16:29 xenorchestra xo-server[1675344]:   done: 59198,
Jun 12 07:16:29 xenorchestra xo-server[1675344]:   parent: '/xo-vm-backups/eda7fcd9-484f-7f19-b5ae-0cfd94ca2207/vdis/89511625-67f3-46be-98fe-8d7a7584386>Jun 12 07:16:29 xenorchestra xo-server[1675344]:   progress: 83,
Jun 12 07:16:29 xenorchestra xo-server[1675344]:   total: 71442
Jun 12 07:16:29 xenorchestra xo-server[1675344]: }
Jun 12 07:16:39 xenorchestra xo-server[1675344]: 2025-06-12T05:16:39.954Z xo:backups:mergeWorker INFO merge in progress {
Jun 12 07:16:39 xenorchestra xo-server[1675344]:   done: 59467,
Jun 12 07:16:39 xenorchestra xo-server[1675344]:   parent: '/xo-vm-backups/eda7fcd9-484f-7f19-b5ae-0cfd94ca2207/vdis/89511625-67f3-46be-98fe-8d7a7584386
un 12 07:16:39 xenorchestra xo-server[1675344]: 2025-06-12T05:16:39.954Z xo:backups:mergeWorker INFO merge in progress {
Jun 12 07:16:39 xenorchestra xo-server[1675344]:   done: 59467,
Jun 12 07:16:39 xenorchestra xo-server[1675344]:   parent: '/xo-vm-backups/eda7fcd9-484f-7f19-b5ae-0cfd94ca2207/vdis/89511625-67f3-46be-98fe-8d7a7584386>
Jun 12 07:16:39 xenorchestra xo-server[1675344]:   progress: 83,
Jun 12 07:16:39 xenorchestra xo-server[1675344]:   total: 71442
Jun 12 07:16:39 xenorchestra xo-server[1675344]: }
Jun 12 07:16:46 xenorchestra xo-server[1679463]: 2025-06-12T05:16:46.783Z @xen-orchestra/xapi/disks/Xapi WARN openNbdCBT Error: CBT is disabled
Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at XapiVhdCbtSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/xapi/d>
Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async #openNbdCbt (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/xapi/disks>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/di>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/backups/_incrementalVm.>
Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async Promise.all (index 1)
Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async cancelableMap (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/backups/>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async exportIncrementalVm (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/ba>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async IncrementalXapiVmBackupRunner._copy (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xe>
Jun 12 07:16:46 xenorchestra xo-server[1679463]:   code: 'CBT_DISABLED'
Jun 12 07:16:46 xenorchestra xo-server[1679463]: }
Jun 12 07:16:46 xenorchestra xo-server[1679463]: 2025-06-12T05:16:46.863Z @xen-orchestra/xapi/disks/Xapi WARN openNbdCBT Error: CBT is disabled
Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at XapiVhdCbtSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/xapi/d>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async #openNbdCbt (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/xapi/disks>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/di>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/backups/_incrementalVm.>
Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async Promise.all (index 2)
Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async cancelableMap (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/backups/>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async exportIncrementalVm (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xen-orchestra/ba>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async IncrementalXapiVmBackupRunner._copy (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@>Jun 12 07:16:46 xenorchestra xo-server[1679463]:     at async IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202506050918/@xe>
Jun 12 07:16:46 xenorchestra xo-server[1679463]:   code: 'CBT_DISABLED'
Jun 12 07:16:46 xenorchestra xo-server[1679463]: }
                        florentF 1 Reply Last reply Reply Quote 0
                        • florentF Offline
                          florent Vates 🪐 XO Team @markxc
                          last edited by

                          @markxc thank you that is the part I wanted

                          I think someone on the xapi side should look into it

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post