@danp
I was having a similar issue with OVA imports (using Chrome), tried a different browser (Firefox) and was able to import the OVA.

Went back to the reverse proxy and tried in Firefox, I currently have two VMs importing directly from VMware

I had an issue with the backing up of a linux VM that had a [nobak] disk associated with it. When attempting the Health Check, I would receive the following error

waitObjectState: timeout reached before OpaqueRef:7c60ab35-dba3-4c4a-887a-0f350a68c3b9 in expected state

Watching the state of the VM startup, I could see the OS would not load. I initially thought this was an issue with XO, but earlier I realised it was the host OS hanging.

This was due to the OS not being able to mount the disk from the health check, as it was not backed up due to [nobak].

Modifying the /etc/fstab in the host OS to include nofail resolved the health check failure. Eg.

UUID=a2c230fd-13e0-4453-9d56-1ab123bfad9e /Cache xfs     defaults,nofail        0       0

Hopefully this will help anybody experiencing similar issues.

@olivierlambert many thanks for the very quick response, it is indeed XO from sources. I think the burst explanation makes sense.

Also, thanks for you and your teams continued development and support with both XCP-NG and XO.

@rtjdamen I understand now

@rtjdamen Thanks for the super fast response!

Just removed the existing snapshots and the task is proceeding.

Did you mean cbt is enabled as opposed to cbt is invalid?

I was looking to do some updates on our TrueNAS Scale device providing an NFS share to my XCP-ng hosts (8.2.1), we have CBT enabled for backups.

However when I try to move the Xen Orchestra VDI from TrueNAS to local storage I receive the following error:

{
  "id": "0m261vorl",
  "properties": {
    "method": "vdi.migrate",
    "params": {
      "id": "f91f81f2-308d-4de9-879e-c1fa84a37d27",
      "sr_id": "49822b62-3367-7e7c-76ee-1cfc91a262e9"
    },
    "name": "API call: vdi.migrate",
    "userId": "7b63bade-51f3-4916-9174-f969da17774a",
    "type": "api.call"
  },
  "start": 1728731129889,
  "status": "failure",
  "updatedAt": 1728731132752,
  "end": 1728731132752,
  "result": {
    "code": "VDI_CBT_ENABLED",
    "params": [
      "OpaqueRef:4f16cd0e-fbaf-48c3-aae4-092b9906b9e4"
    ],
    "task": {
      "uuid": "7ce61fba-d6d3-12cb-2585-79d5b69d3857",
      "name_label": "Async.VDI.pool_migrate",
      "name_description": "",
      "allowed_operations": [],
      "current_operations": {},
      "created": "20241012T11:05:31Z",
      "finished": "20241012T11:05:32Z",
      "status": "failure",
      "resident_on": "OpaqueRef:fe0440a3-4a31-44d6-8317-a0e64d0ee01e",
      "progress": 1,
      "type": "<none/>",
      "result": "",
      "error_info": [
        "VDI_CBT_ENABLED",
        "OpaqueRef:4f16cd0e-fbaf-48c3-aae4-092b9906b9e4"
      ],
      "other_config": {},
      "subtask_of": "OpaqueRef:NULL",
      "subtasks": [],
      "backtrace": "(((process xapi)(filename ocaml/xapi-client/client.ml)(line 7))((process xapi)(filename ocaml/xapi-client/client.ml)(line 19))((process xapi)(filename ocaml/xapi-client/client.ml)(line 12359))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 134))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
    },
    "message": "VDI_CBT_ENABLED(OpaqueRef:4f16cd0e-fbaf-48c3-aae4-092b9906b9e4)",
    "name": "XapiError",
    "stack": "XapiError: VDI_CBT_ENABLED(OpaqueRef:4f16cd0e-fbaf-48c3-aae4-092b9906b9e4)\n    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202410111017/packages/xen-api/_XapiError.mjs:16:12)\n    at default (file:///opt/xo/xo-builds/xen-orchestra-202410111017/packages/xen-api/_getTaskResult.mjs:13:29)\n    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202410111017/packages/xen-api/index.mjs:1041:24)\n    at file:///opt/xo/xo-builds/xen-orchestra-202410111017/packages/xen-api/index.mjs:1075:14\n    at Array.forEach (<anonymous>)\n    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202410111017/packages/xen-api/index.mjs:1065:12)\n    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202410111017/packages/xen-api/index.mjs:1238:14)"
  }
}

I can see a task disabling CBT on the disk and looking at the UI it shows as CBT disabled.

I experience the same issue attempting to migrate other VDI's too.

The problem still existed on latest as of earlier today.

I removed the VM from the original (Smart Mode) backup job and cleaned up any VDI's and detached backups. As a test I created a separate backup of the VM in question pointing to the same target, this backup was successful.

Hopefully I've worked around the issue, I will try adding the VM back into a Smart Mode job in the coming days.

With the latest commit on Master (72592), here is the resultant complete backup log

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1724152335898",
  "jobId": "5e0a3dca-5c7f-4459-9b57-6af74a01d812",
  "jobName": "Production every other day",
  "message": "backup",
  "scheduleId": "1f6ca79c-0e6e-460e-bad5-5e984ec28ef5",
  "start": 1724152335898,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "45df57b9-e2d9-ed2f-e467-345bd6f10296"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "45df57b9-e2d9-ed2f-e467-345bd6f10296",
        "name_label": "sv-uts"
      },
      "id": "1724152337383",
      "message": "backup VM",
      "start": 1724152337383,
      "status": "failure",
      "tasks": [
        {
          "id": "1724152337409",
          "message": "clean-vm",
          "start": 1724152337409,
          "status": "success",
          "end": 1724152339005,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1724152339972",
          "message": "snapshot",
          "start": 1724152339972,
          "status": "success",
          "end": 1724152344659,
          "result": "3d4f7f45-753f-6f58-464a-53dc1d63c054"
        },
        {
          "data": {
            "id": "179129c4-10c2-40f8-ba64-534ff4dc7da4",
            "isFull": true,
            "type": "remote"
          },
          "id": "1724152344660",
          "message": "export",
          "start": 1724152344660,
          "status": "success",
          "tasks": [
            {
              "id": "1724152350897",
              "message": "transfer",
              "start": 1724152350897,
              "status": "success",
              "end": 1724158256392,
              "result": {
                "size": 80103683584
              }
            },
            {
              "id": "1724158257369",
              "message": "clean-vm",
              "start": 1724158257369,
              "status": "success",
              "end": 1724158259162,
              "result": {
                "merge": false
              }
            }
          ],
          "end": 1724158259163
        }
      ],
      "end": 1724158259163,
      "result": {
        "generatedMessage": false,
        "code": "ERR_ASSERTION",
        "actual": false,
        "expected": true,
        "operator": "strictEqual",
        "message": "VM must be a snapshot",
        "name": "AssertionError",
        "stack": "AssertionError [ERR_ASSERTION]: VM must be a snapshot\n    at Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202408201207/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:245:20)\n    at Function.from (<anonymous>)\n    at asyncMap (/opt/xo/xo-builds/xen-orchestra-202408201207/@xen-orchestra/async-map/index.js:23:28)\n    at Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202408201207/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:233:13)\n    at Function.from (<anonymous>)\n    at asyncMap (/opt/xo/xo-builds/xen-orchestra-202408201207/@xen-orchestra/async-map/index.js:23:28)\n    at IncrementalXapiVmBackupRunner._removeUnusedSnapshots (file:///opt/xo/xo-builds/xen-orchestra-202408201207/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:219:11)\n    at IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202408201207/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:384:18)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202408201207/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
      }
    }
  ],
  "end": 1724158259164
}

@olivierlambert Just checked, we are one commit behind on Master.

I will look to update now.

We are using XO from sources (latest) against a pool with two XCP-NG hosts XCP-ng 8.2.1 (latest)

We have a VM that looks to be failing its delta backup to an offsite S3 target, we receive the error 'VM must be a snapshot'. Checked both backup and main pool health, no warnings are showing for snapshots or VDIs

We did try removing the existing snapshot and migrating the VM between hosts, but the error reappears.

Logfile -

{
  "data": {
    "mode": "delta",
    "reportWhen": "failure"
  },
  "id": "1724150430187",
  "jobId": "5e0a3dca-5c7f-4459-9b57-6af74a01d812",
  "jobName": "Production every other day",
  "message": "backup",
  "scheduleId": "1f6ca79c-0e6e-460e-bad5-5e984ec28ef5",
  "start": 1724150430187,
  "status": "failure",
  "infos": [
    {
      "data": {
        "vms": [
          "45df57b9-e2d9-ed2f-e467-345bd6f10296"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "45df57b9-e2d9-ed2f-e467-345bd6f10296",
        "name_label": "server"
      },
      "id": "1724150431672",
      "message": "backup VM",
      "start": 1724150431672,
      "status": "failure",
      "tasks": [
        {
          "id": "1724150431700",
          "message": "clean-vm",
          "start": 1724150431700,
          "status": "success",
          "end": 1724150433318,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1724150434097",
          "message": "clean-vm",
          "start": 1724150434097,
          "status": "success",
          "end": 1724150435469,
          "result": {
            "merge": false
          }
        }
      ],
      "end": 1724150435471,
      "result": {
        "generatedMessage": false,
        "code": "ERR_ASSERTION",
        "actual": false,
        "expected": true,
        "operator": "strictEqual",
        "message": "VM must be a snapshot",
        "name": "AssertionError",
        "stack": "AssertionError [ERR_ASSERTION]: VM must be a snapshot\n    at Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202408191446/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:245:20)\n    at Function.from (<anonymous>)\n    at asyncMap (/opt/xo/xo-builds/xen-orchestra-202408191446/@xen-orchestra/async-map/index.js:23:28)\n    at Array.<anonymous> (file:///opt/xo/xo-builds/xen-orchestra-202408191446/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:233:13)\n    at Function.from (<anonymous>)\n    at asyncMap (/opt/xo/xo-builds/xen-orchestra-202408191446/@xen-orchestra/async-map/index.js:23:28)\n    at IncrementalXapiVmBackupRunner._removeUnusedSnapshots (file:///opt/xo/xo-builds/xen-orchestra-202408191446/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:219:11)\n    at IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202408191446/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:354:16)\n    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n    at async file:///opt/xo/xo-builds/xen-orchestra-202408191446/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
      }
    }
  ],
  "end": 1724150435472
}

Issue Encountered:

A backup of a Windows VM failed a health check overnight, with the error message:
waitObjectState: timeout reached before OpaqueRef.

Manual Check:

Performed a manual backup and health check, which resulted in the same error.

Diagnosis:

Connected to the VM using Xen Orchestra and monitored the console. Discovered that a Windows update was being applied during the VM boot process, which was causing the VM to not boot correctly from the perspective of Xen Orchestra (XO).

Resolution:

By applying the outstanding Windows update and rebooting the VM, the issue was resolved, and the backup and health check were successful afterward.

The problem was essentially due to the VM being in an update state during the health check, which prevented XO from properly detecting and managing the VM's state. Applying the update and rebooting ensured the VM was in a proper state for backup and health check operations.

@danp
I was having a similar issue with OVA imports (using Chrome), tried a different browser (Firefox) and was able to import the OVA.

Went back to the reverse proxy and tried in Firefox, I currently have two VMs importing directly from VMware

@Danp No error messages and nothing in either Settings - Logs or via the CLI (journalctl)

It could be due to the age of the ESXi host or the fact I am using a reverse SSL proxy to cope with the legacy ciphers.

I'll look to export the VMs as OVA's and import that way.