Confirm commit 1a7b5 backups completed successfully.

Last night 2 of my vms failed to complete a delta backup. As the tasks could not be cancled in any way i rebooted XO (built from sources) the task still show "runing" so i restarted the tool stack on host 1 and the tasks cleared. I attempted to restart failed backups and again the backup just hangs. It create the snapshot but never transfer data. The Remote is the same location as the nfs storage the vms are running from. So i know the Storage is good.

A few more rebooted of XO and tool stack. I rebooted both host and each time backups get stuck. If i try to start a new backup (same job) all vms hang. I tried to run a full delta backup and same. I tried to update XO but I am on current master build as of today (6b263) I tried to do a force update and still backup never completes.

I built a new VM for XO and installed from sources and still fail.

Here is one of the logs from the backups...

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "hideSuccessfulItems": true
  },
  "id": "1750503695411",
  "jobId": "95ac8089-69f3-404e-b902-21d0e878eec2",
  "jobName": "Backup Job 1",
  "message": "backup",
  "scheduleId": "76989b41-8bcf-4438-833a-84ae80125367",
  "start": 1750503695411,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "b25a5709-f1f8-e942-f0cc-f443eb9b9cf3",
          "3446772a-4110-7a2c-db35-286c73af4ab4",
          "bce2b7f4-d602-5cdf-b275-da9554be61d3",
          "e0a3093a-52fd-f8dc-1c39-075eeb9d0314",
          "afbef202-af84-7e64-100a-e8a4c40d5130"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "b25a5709-f1f8-e942-f0cc-f443eb9b9cf3",
        "name_label": "SeedBox"
      },
      "id": "1750503696510",
      "message": "backup VM",
      "start": 1750503696510,
      "status": "interrupted",
      "tasks": [
        {
          "id": "1750503696519",
          "message": "clean-vm",
          "start": 1750503696519,
          "status": "success",
          "end": 1750503696822,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1750503697911",
          "message": "snapshot",
          "start": 1750503697911,
          "status": "success",
          "end": 1750503699564,
          "result": "6e2edbe9-d4bd-fd23-28b9-db4b03219e96"
        },
        {
          "data": {
            "id": "1575a1d8-3f87-4160-94fc-b9695c3684ac",
            "isFull": false,
            "type": "remote"
          },
          "id": "1750503699564:0",
          "message": "export",
          "start": 1750503699564,
          "status": "success",
          "tasks": [
            {
              "id": "1750503701979",
              "message": "clean-vm",
              "start": 1750503701979,
              "status": "success",
              "end": 1750503702141,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1750503702142
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "invalid HTTP header in response body"
          },
          "message": "Retry the VM backup due to an error"
        }
      ]
    },
    {
      "data": {
        "type": "VM",
        "id": "3446772a-4110-7a2c-db35-286c73af4ab4",
        "name_label": "XO"
      },
      "id": "1750503696512",
      "message": "backup VM",
      "start": 1750503696512,
      "status": "interrupted",
      "tasks": [
        {
          "id": "1750503696518",
          "message": "clean-vm",
          "start": 1750503696518,
          "status": "success",
          "end": 1750503696693,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1750503712472",
          "message": "snapshot",
          "start": 1750503712472,
          "status": "success",
          "end": 1750503713915,
          "result": "a1bdef52-142c-5996-6a49-169ef390aa2e"
        },
        {
          "data": {
            "id": "1575a1d8-3f87-4160-94fc-b9695c3684ac",
            "isFull": false,
            "type": "remote"
          },
          "id": "1750503713915:0",
          "message": "export",
          "start": 1750503713915,
          "status": "success",
          "tasks": [
            {
              "id": "1750503716280",
              "message": "clean-vm",
              "start": 1750503716280,
              "status": "success",
              "end": 1750503716383,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1750503716385
        }
      ],
      "warnings": [
        {
          "data": {
            "attempt": 1,
            "error": "invalid HTTP header in response body"
          },
          "message": "Retry the VM backup due to an error"
        }
      ]
    },
    {
      "data": {
        "type": "VM",
        "id": "bce2b7f4-d602-5cdf-b275-da9554be61d3",
        "name_label": "iVentoy"
      },
      "id": "1750503702145",
      "message": "backup VM",
      "start": 1750503702145,
      "status": "interrupted",
      "tasks": [
        {
          "id": "1750503702148",
          "message": "clean-vm",
          "start": 1750503702148,
          "status": "success",
          "end": 1750503702233,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1750503702532",
          "message": "snapshot",
          "start": 1750503702532,
          "status": "success",
          "end": 1750503704850,
          "result": "05c5365e-3bc5-4640-9b29-0684ffe6d601"
        },
        {
          "data": {
            "id": "1575a1d8-3f87-4160-94fc-b9695c3684ac",
            "isFull": false,
            "type": "remote"
          },
          "id": "1750503704850:0",
          "message": "export",
          "start": 1750503704850,
          "status": "interrupted",
          "tasks": [
            {
              "id": "1750503706813",
              "message": "transfer",
              "start": 1750503706813,
              "status": "interrupted"
            }
          ]
        }
      ],
      "infos": [
        {
          "message": "Transfer data using NBD"
        }
      ]
    },
    {
      "data": {
        "type": "VM",
        "id": "e0a3093a-52fd-f8dc-1c39-075eeb9d0314",
        "name_label": "Docker of Things"
      },
      "id": "1750503716389",
      "message": "backup VM",
      "start": 1750503716389,
      "status": "interrupted",
      "tasks": [
        {
          "id": "1750503716395",
          "message": "clean-vm",
          "start": 1750503716395,
          "status": "success",
          "warnings": [
            {
              "data": {
                "path": "/xo-vm-backups/e0a3093a-52fd-f8dc-1c39-075eeb9d0314/20250604T160135Z.json",
                "actual": 6064872448,
                "expected": 6064872960
              },
              "message": "cleanVm: incorrect backup size in metadata"
            }
          ],
          "end": 1750503716886,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1750503717182",
          "message": "snapshot",
          "start": 1750503717182,
          "status": "success",
          "end": 1750503719640,
          "result": "9effb56d-68e6-8015-6bd5-64fa65acbada"
        },
        {
          "data": {
            "id": "1575a1d8-3f87-4160-94fc-b9695c3684ac",
            "isFull": false,
            "type": "remote"
          },
          "id": "1750503719640:0",
          "message": "export",
          "start": 1750503719640,
          "status": "interrupted",
          "tasks": [
            {
              "id": "1750503721601",
              "message": "transfer",
              "start": 1750503721601,
              "status": "interrupted"
            }
          ]
        }
      ],
      "infos": [
        {
          "message": "Transfer data using NBD"
        }
      ]
    }
  ],
  "end": 1750504870213,
  "result": {
    "message": "worker exited with code null and signal SIGTERM",
    "name": "Error",
    "stack": "Error: worker exited with code null and signal SIGTERM\n    at ChildProcess.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202506202218/@xen-orchestra/backups/runBackupWorker.mjs:24:48)\n    at ChildProcess.emit (node:events:518:28)\n    at ChildProcess.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202506202218/@xen-orchestra/log/configure.js:52:17)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:293:12)\n    at Process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}

Thank you for the update. Last night before I saw your replys that is exactly what i did. I purged the backup's for the two VMs from BackBlaze. The backup job completed successfully while copying over the previous backup images.

@Danp said in XOA license issue:

@acebmxer It looks like you've resolved the first issue by binding the trial license. You haven't yet purchased support, so you won't be able to bind licenses to your XCP-ng hosts. This has no effect on the functionality of the hosts.

Ok thank you for the update that this is expected behavior. We do plan to pay for support just need to take care of a few things on our side before we do so.

@DustinB

That pic was taken from the host tab. But yes I removed from the settings tab and host 2 still shows under pool 1 and all looks ok.

Thank you.

@olivierlambert

Got it working. With some help in another forum a user helped. Added the following line in the advanced tap for the proxy host.

client_max_body_size 0;

I thought i have uploaded ISOs previously with no issues before but i may have been mistaken and have uploaded other ways for other reasons.

@cichy

You need to select a vm template.

Update - After shutting down the VM and removing the previous snapshot and creating new one. I left VM powered off after creating new snapshot. VM is migrating now.

Screenshots of VM settings and files...

I dont see a the file that it is looking for...

I am trying to complete my vmware migration and I have 2 more vm's to migrat. All have been successful until now...

This is the error message i get when trying to migrate the VM.

503 Service Unavailable https://192.168.1.125/folder/ArcDC01/ArcDC01-000001-sesparse.vmdk?dcPath=ha-datacenter&dsName=SSD_DataStore

Suggestions? I have since started the VM back up on vmware and the VM is working.

The only thing i did differently was take a snapshot from a shutdown state before removing the vmware tools. This server is our Primary Windows Active Directory.

@Danp said in XOA license issue:

@acebmxer It looks like you've resolved the first issue by binding the trial license. You haven't yet purchased support, so you won't be able to bind licenses to your XCP-ng hosts. This has no effect on the functionality of the hosts.

Ok thank you for the update that this is expected behavior. We do plan to pay for support just need to take care of a few things on our side before we do so.

@Greg_E

I know this is an old thread but just incase others come accross it. @Greg_E did you resolve this?

Did you install the UEFI certs into the new install - https://docs.xcp-ng.org/guides/guest-UEFI-Secure-Boot/

I had some issues and had to do them manually see here - https://xcp-ng.org/forum/topic/11129/xcp-ng-8.3-uefi-certs-wont-install

@kubuntu-newbie

Yeah i didnt do anything with disabling the vgpu. Just a standard Win 11 or ubuntu VM and attached the nivdia gpu.

I was able to get nvidia rtx 4070 passed through to a windows VM and Ubuntu server. Need to make sure you passing all the devices associated with the GPU, specifically the Audio controller on the GPU.

These were on AMD systems.

Just an FYI I currently only have 1 xcp-ng host as I am still in the vmware migration. I have 2 more vms to migrate which is scheduled for tomorrow. If no issues I will be able to being over the 2nd host to xcp-ng.

Thread created to move the conversation