Unable to Migrate VM's to newly added host in pool

jmannik

I am trying to live migrate vm's to a freshly added host to the pool.
I am getting thee below error:

vm.migrate
{
  "vm": "5270ed06-85dc-1050-dc26-789a39a6ca0a",
  "migrationNetwork": "b012ab5c-bbe8-ccca-bbe7-596468bb04cf",
  "targetHost": "b2857df4-26dd-4e93-9ab0-60cc1f753bd1"
}
{
  "code": "VM_LACKS_FEATURE",
  "params": [
    "OpaqueRef:10dfa2af-92c1-194a-a787-8ffa0c44adee"
  ],
  "task": {
    "uuid": "cd8fa000-92d5-3b9e-578a-c3359f5c3814",
    "name_label": "Async.VM.migrate_send",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20250531T11:53:50Z",
    "finished": "20250531T11:53:50Z",
    "status": "failure",
    "resident_on": "OpaqueRef:aa4ba7f4-b2d8-6068-e502-a8bc50a06177",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "VM_LACKS_FEATURE",
      "OpaqueRef:10dfa2af-92c1-194a-a787-8ffa0c44adee"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_vm_lifecycle.ml)(line 743))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1652))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/helpers.ml)(line 1706))((process xapi)(filename ocaml/xapi/xapi_vm_helpers.ml)(line 1651))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 2612))((process xapi)(filename ocaml/xapi/rbac.ml)(line 188))((process xapi)(filename ocaml/xapi/rbac.ml)(line 197))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 77)))"
  },
  "message": "VM_LACKS_FEATURE(OpaqueRef:10dfa2af-92c1-194a-a787-8ffa0c44adee)",
  "name": "XapiError",
  "stack": "XapiError: VM_LACKS_FEATURE(OpaqueRef:10dfa2af-92c1-194a-a787-8ffa0c44adee)
    at Function.wrap (file:///opt/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
    at default (file:///opt/xen-orchestra/packages/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1072:24)
    at file:///opt/xen-orchestra/packages/xen-api/index.mjs:1106:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1096:12)
    at Xapi._watchEvents (file:///opt/xen-orchestra/packages/xen-api/index.mjs:1269:14)"
}

The VM is a Windows based VM, the latest tools were downloaded and installed on the same day as this post.

It is moving from a Ryzen 5900X host to another Ryzen 5900X host.

It is using shared storage (NFS Share).

I really wish the error message would specify what feature is lacking as it would help me troubleshoot this much easier.

olivierlambert

Hi,

This means the VM isn't cooperating. The most likely thing is an issue regarding the tools, not correctly installed or working.

jmannik

@olivierlambert
I would have thought that too... except it does it for every VM

At this stage I am just going to rebuild the pool and migrate the VM's and other hosts across to it instead.

jmannik

So just to update on this whole thing, after building a new pool and migrating everything over it is all sorted and working as expected now.
I don't know what is different because I could not find any difference between the old pool and the new pool.

Something that was incredibly frustrating and that I found rather backwards to setup and get working correctly was the interface order.
Each host has 1x1gb and 4x10g network interfaces, two of the three hosts had the interfaces ordered as expected with the 1g interface being eth0 (network cards Chelsio T540-CR cards, 1g eth is onboard realtek)
One host decided to have the 1g interface on eth1 for no apparent reason.
The process of renaming and reordering the interfaces was the most convoluted frustrating experience with configuring basic networking ive experienced in a long time.
I do not understand why it is done like it is? Why does it take so much work to do this? It seems like a very basic thing that should be easy. The process I had to follow to fix this was as follows:

SSH into the host
Re-assign the management interface to eth4
Reboot the host
SSH back into the host via eth4
Shut down the interfaces eth0 and eth1
Run the commands interface-rename --rename eth0=1g interface mac eth1=first interface on 10g car
Run the same command except --update instead of --rename (if I did not do both commands in that order it would NOT work)
Re-enable the two interfaces and reboot again.
Removed and recreate the pif's for the above two interfaces.

I spent an entire weekend to work out this process

I don't understand why this had to be this hard to do, nor do I understand why the interfaces were the wrong order on 1 host of the 3. Identical motherboards, and configuration. The only difference is that one of the two correct interface order boxes has a 3900xt instead of a 5900x cpu.

olivierlambert

Ping @stormi and/or @bleader

jmannik

Just to be clear, everything is working as I want it right now... so I don't want to poke it out of fear of it breaking again

olivierlambert

Yes but your feedback is important, I want to be sure it's seen by our XCP-ng team

jmannik

@olivierlambert Appreciated, hopefully my frustration doesn't come across as rude and only as frustrated.

The biggest question mark for me is why the 1gb interface decided to be eth1 despite all other interfaces being on the one network card?
Ideally (in my opinion) re-assigning interface order in the future should be just like other config options where you click on the interface assignment in Xen Orchestra, type in the new name of the interface and it is done. It would probably mean having to rename eth0 to eth5, renaming eth1 to eth0 and then renaming eth5 to eth1... but what I would expect.

Another thing that would make things easier/better is the ability to reassign networks to different pifs via a similar thing to the above.
In an ideal world I don't want to have to drop to a cli to do the configuration changes unless im scripting a large number of things or doing something I would consider out of the ordinary

olivierlambert

No no, don't worry, it shouldn't be that frustrating. That's why I want to check what do we have in the pipes in the future to make it easier.