Update: I was able to move the VM to a different host that is also xcp-ng 8.2. When I tried moving the VM back to where it came from now I'm getting: NO_HOSTS_AVAILABLE()

vm.migrate
{
  "mapVdisSrs": {
    "f4d88025-9e26-44a1-8028-4d5d9c11c994

": "18850841-5eb8-6b28-73ba-cae1c40af157"
  },
  "mapVifsNetworks": {
    "b56126a8-85ab-80de-9db8-4864ec5618e5

": "99b40169-8dab-6e9d-db96-23277b405b4a

"
  },
  "migrationNetwork": "99b40169-8dab-6e9d-db96-23277b405b4a

",
  "sr": "18850841-5eb8-6b28-73ba-cae1c40af157",
  "targetHost": "69c0c257-a547-4d49-8270-b49eafe35c2c",
  "vm": "78938c32-eb52-491f-1fbf-3ad5d2e374f0"
}
{
  "code": "NO_HOSTS_AVAILABLE",
  "params": [],
  "task": {
    "uuid": "4eb56a28-ffa7-cee6-31e1-87209af8de75

",
    "name_label": "Async.VM.migrate_send",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20250131T20:51:54Z",
    "finished": "20250131T20:51:54Z",
    "status": "failure",
    "resident_on": "OpaqueRef:72cbec6b-d489-49ca-a8a8-e623e3c5f4e4",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "NO_HOSTS_AVAILABLE"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_placement.ml)(line 104))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1199))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1356))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1191))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2279))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
  },
  "message": "NO_HOSTS_AVAILABLE()",
  "name": "XapiError",
  "stack": "XapiError: NO_HOSTS_AVAILABLE()
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/index.mjs:1068:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/index.mjs:1102:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/index.mjs:1092:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/index.mjs:1265:14)"
}

I rebooted XO and tried moving the VM to the xcp-ng 8.3 machine again. It looks like the task started but now it's hung at "Connection to VM console (on xcp-m83) 0%" and I can't "Cancel" or "Destroy" the task

I'm trying to migrate a powered off VM from one host to another and when I try to migrate I get the error below:

The source host is xcp-ng 8.2 and the destination host is xcp-ng 8.3

vm.migrate
{
  "vm": "78938c32-eb52-491f-1fbf-3ad5d2e374f0",
  "mapVifsNetworks": {
    "b56126a8-85ab-80de-9db8-4864ec5618e5

": "99b40169-8dab-6e9d-db96-23277b405b4a

"
  },
  "migrationNetwork": "99b40169-8dab-6e9d-db96-23277b405b4a

",
  "sr": "18850841-5eb8-6b28-73ba-cae1c40af157",
  "targetHost": "69c0c257-a547-4d49-8270-b49eafe35c2c"
}
{
  "code": "HANDLE_INVALID",
  "params": [
    "SR",
    "OpaqueRef:e8fd26cd-3470-4ef8-9738-8c056e034394

"
  ],
  "task": {
    "uuid": "b0365bf6-8275-0194-fa8d-463cd966463c

",
    "name_label": "Async.VM.migrate_send",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20250131T19:17:57Z",
    "finished": "20250131T19:17:58Z",
    "status": "failure",
    "resident_on": "OpaqueRef:e3f0dcf1-42a7-4ed2-afed-1d387e305657",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "HANDLE_INVALID",
      "SR",
      "OpaqueRef:e8fd26cd-3470-4ef8-9738-8c056e034394

"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_migrate.ml)(line 1573))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 131))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 1228))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 35))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2298))((process xapi)(filename lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/xapi/rbac.ml)(line 205))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 95)))"
  },
  "message": "HANDLE_INVALID(SR, OpaqueRef:e8fd26cd-3470-4ef8-9738-8c056e034394

)",
  "name": "XapiError",
  "stack": "XapiError: HANDLE_INVALID(SR, OpaqueRef:e8fd26cd-3470-4ef8-9738-8c056e034394

)
    at Function.wrap (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/index.mjs:1068:24)
    at file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/index.mjs:1102:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/index.mjs:1092:12)
    at Xapi._watchEvents (file:///opt/xo/xo-builds/xen-orchestra-202501311655/packages/xen-api/index.mjs:1265:14)"
}

I tried to move the VM between hosts using XCP-ng Center and also got an error:

Migrating VM 'IPM-ova on 'xcp-m90t'' to 'xcp-m83 in 'xcp-m83''
Object has been deleted.SR:OpaqueRef:e8fd26cd-3470-4ef8-9738-8c056e034394
Time: 00:00:01	xcp-m83	Jan 31, 2025 2:35 PM	Dismiss

I'm able to move other VMs around just fine.

Is XCP-ng 8.3 based on CentOS 8.5 or something else? I'm having problems getting NUT to work and I'd like to try to get it working on the "parent" OS first.

@apz I see. Thanks.

I'm getting a lot of this message when examining output of dmesg. Would anyone know what this means?

I’m trying out XCP-ng 8.3. I moved a VM from a host running XCP-ng 8.2.1 to XCP-ng 8.3, that went just fine. Now I powered off the VM and I’m trying to move the VM back to the 8.2.1 host but I’m getting:

VM_HOST_INCOMPATIBLE_VERSION_MIGRATE

Is going back no supported?

@olivierlambert I just noticed this after I tried to export a large VM. The export did not complete, nothing was downloaded. Not sure if that's coincidence.

From source. Master, commit 636c8. As far as I know I don't have any scripts going.

I just noticed that about every second there is a new task popping up in the task pane and the number of items keeps increasing. Please see screen shot. Is this normal? This happens even right after rebooting xo. Is this just normal operation or is something stuck?

I have a host with 36 physical cores. On this system there will be two VMs running. Sometimes they will run one at a time, other times they will both be running simultaneously. Should I assign all 36 cores to both VMs so that each VM has access to all the cores when running on its own or will that cause horrible inefficiencies when both VMs are running and it’s better to assign 18 cores to each VM? The load will scale to as many cores as are available. Both VMs are running Windows. If I assign 18 cores to the VM and the VM is running on its own will this mean half of the CPU will be idle?

What about memory, can I over provision memory (assign the total amount of RAM to each VM) or is that no longer supported.

Any insight is appreciated.

I deleted backups of VMs I no longer need (Backups > Restore > Delete backups). No problems so far, but when I want to look at the backup data store, I noticed that there are still directory structures left behind that are associated with those deleted backups (though they are empty). It would be nice if those folders were also deleted so that there is less folders to worry about when creating off site backups.

@olivierlambert The issue started according when I attempted to do a restore of a backup. The restore failed and the rogue tasks started to happen. I waited a few days until a new Xen commit was available, I updated Xen and tried the restore again. This time the restore was successful but the tasks kept coming. I waited some more, updated Xen (today) and the failing tasks are continuing. I noticed that there are some patches available for the hosts. I applied those, rebooted the hosts but unfortunately the tasks are still coming.

@Danp Hello again. I just updated Xen to commit 42dc7. The problem is still occurring

@Danp said in Backup Restore is Stuck:

@Kajetan321 I've seen this exact behavior on a client's site, but it isn't occurring on my local install. This will hopefully be fixed soon.

Are your VM restores working correctly now?

I just tried to do the same restore that caused the initial problem. The restore worked correctly. So it looks like my restores are working.

@Danp I haven't tried to do a restore again. I was hoping to fix this problem before trying again.

@Danp

I restarted the tool stacks, rebooted all hosts and performed a clean installation of Xen. I'm still getting the "XO user authentication" entry about every second. There must be something up on one of the hosts. Any ideas on how to fix? I'm now running the latest Xen (commit 0794a).

In tasks, when I click on "open raw log" a tab opens with the text "Not Found"

@DustinB OK, reset tool stack on both hosts, that helped, the restore task is now gone. I'm still getting the "XO user authentication" entry about every second though even after reverting to my restore point again. Maybe reboot everything?

@Danp By revert I'm assuming revert to an earlier snapshot. Reverting to the last snapshot got me to commit 6056a. Is that what you meant?

The weird thing is, the restore task is still there and still not progressing.

Hello,

I created a test VM (Pop!_OS) and created a backup job for it. It has been backing up daily without any visible issues. To test the restore process, I deleted the VM and then went into restore to get my VM back from backups. The restore process appears to be stuck. When looking in tasks I get:

Please note the "XO user authentication" entry that I'm getting about every second.

Hovering over the Cancel and Destroy icons turns my courser turns to a prohibition icon (red circle with a red diagonal dash going through it).

Under the restore tab I'm seeing this:

How do I cancel this task and what to do about the failed restore?

I'm using Xen from source, I'm on Xen Orchestra, commit e8c6f.

Thank you.