Live Migrate between two stand alone pools fails

DustinB

So I'm trying to live migrate a few workloads off of a host that I need to patch up to 8.2.1, If I could I would just shutdown the VM's and start them on a different host (waiting for my team on that answer).

Anyways, I just tried one VM and it failed, here is the logs.

vm.migrate
{
  "vm": "dfecebe0-6ac1-b511-85ee-b43aa0223578",
  "mapVifsNetworks": {
    "9084387e-b812-4a15-7c00-fb5a9b89d409": "89899d15-f176-8027-b9c6-758c1c6cb86e"
  },
  "migrationNetwork": "89899d15-f176-8027-b9c6-758c1c6cb86e",
  "sr": "eaa4aed3-d67d-fadf-a024-adf0bf48a2e3",
  "targetHost": "b14123b2-3c21-47cb-aae2-3d2e79010dac"
}
{
  "code": 21,
  "data": {
    "objectId": "dfecebe0-6ac1-b511-85ee-b43aa0223578",
    "code": "VM_BAD_POWER_STATE"
  },
  "message": "operation failed",
  "name": "XoError",
  "stack": "XoError: operation failed
    at operationFailed (/opt/xen-orchestra/packages/xo-common/src/api-errors.js:21:32)
    at file:///opt/xen-orchestra/packages/xo-server/src/api/vm.mjs:482:15
    at Object.migrate (file:///opt/xen-orchestra/packages/xo-server/src/api/vm.mjs:469:3)
    at Api.callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:304:20)"
}

If I'm live migrating, why would the VM be in a bad power state?

The host that this VM runs on is running XCP-ng 8.0.0 (GPLv2), the host I'm migrating too is running XCP-ng 8.2.1 (GPLv2)

DustinB

Here's more logs

{
  "data": {
    "callId": "n87tgit8xk",
    "method": "vm.migrate",
    "params": {
      "vm": "dfecebe0-6ac1-b511-85ee-b43aa0223578",
      "mapVifsNetworks": {
        "9084387e-b812-4a15-7c00-fb5a9b89d409": "89899d15-f176-8027-b9c6-758c1c6cb86e"
      },
      "migrationNetwork": "89899d15-f176-8027-b9c6-758c1c6cb86e",
      "sr": "eaa4aed3-d67d-fadf-a024-adf0bf48a2e3",
      "targetHost": "b14123b2-3c21-47cb-aae2-3d2e79010dac"
    },
    "timestamp": 1704979629389,
    "duration": 941,
    "error": {
      "code": 21,
      "data": {
        "objectId": "dfecebe0-6ac1-b511-85ee-b43aa0223578",
        "code": "VM_BAD_POWER_STATE"
      },
      "message": "operation failed",
      "name": "XoError",
      "stack": "XoError: operation failed\n    at operationFailed (/opt/xen-orchestra/packages/xo-common/src/api-errors.js:21:32)\n    at file:///opt/xen-orchestra/packages/xo-server/src/api/vm.mjs:482:15\n    at Object.migrate (file:///opt/xen-orchestra/packages/xo-server/src/api/vm.mjs:469:3)\n    at Api.callApiMethod (file:///opt/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:304:20)"
    }
  },
  "event": "apiCall",
  "previousId": "$5$$7dae8a7421651a967f0ecf611df904e41bef48e4e9dec77cc720081aeda5407c",
  "subject": {
    "userId": "7835589b-088f-43e2-8065-02a1d33273c6",
    "userIp": "::ffff:10.81.234.2",
    "userName": "dustin"
  },
  "time": 1704979629389,
  "id": "$5$$0545dd339ec16c2ff9b1cb9efa2c9c9637f845e6b9b2475d31713d3d8d16534b"
}

Danp

Hey @DustinB

Have you checked the xcp-ng logs to see if they provide any additional details that are useful? Also, you may want to try a warm migration to see if that makes a difference.

Danp

Also, show us the output of the following command --

xe vm-list uuid=dfecebe0-6ac1-b511-85ee-b43aa0223578

DustinB

@Danp xe

vm-list uuid=dfecebe0-6ac1-b511-85ee-b43aa0223578
uuid ( RO)           : dfecebe0-6ac1-b511-85ee-b43aa0223578
     name-label ( RW): Lab - Redhat8
    power-state ( RO): running

DustinB

@Danp Which logs in particular would you think might have some information? Nothing stands out, but the logs are pretty busy.

I've not tried a warm migration

DustinB

@DustinB said in Live Migrate between two stand alone pools fails:

@Danp Which logs in particular would you think might have some information? Nothing stands out, but the logs are pretty busy.

I've not tried a warm migration

I also don't have a warm migration option, since these hosts are all standalone pools.

Danp

I would check var/log/xensource.log first. If the error occurs immediately, then you should be able to tail the file and watch for the error to occur.

Danp

@DustinB said in Live Migrate between two stand alone pools fails:

I also don't have a warm migration option

Did you look under the VM's Advanced tab in XO?

DustinB

@Danp said in Live Migrate between two stand alone pools fails:

I would check var/log/xensource.log first. If the error occurs immediately, then you should be able to tail the file and watch for the error to occur.

Yea it's not instant, it just kind of times out.

All VM's are using an shared NFS storage repo as well. I might just tell the team that in order to complete my work I need a window that I can power everything down and start it on a different host.

DustinB

@Danp said in Live Migrate between two stand alone pools fails:

@DustinB said in Live Migrate between two stand alone pools fails:

I also don't have a warm migration option

Did you look under the VM's Advanced tab in XO?

Yeah

Danp

It should be there unless your XO is way out of date.

DustinB

@Danp said in Live Migrate between two stand alone pools fails:

It should be there unless your XO is way out of date.

That is very possible, there are currently 3 different XO's running, for reason I won't go into.

I'm trying to patch the host all to the same level and consolidate everything under the latest XO, and ideally create a pool out of all 3 hosts. I'm not sure that I'll be able to pool them all if there are VM's running on any given host.

DustinB

This post is deleted!

Danp

You could always try searching for the error in the logs with something like the following --

cat /var/log/xensource.log | grep "VM_BAD_POWER_STATE"

DustinB

@Danp

cat /var/log/xensource.log | grep "VM_BAD_POWER_STATE"
Jan 11 05:42:59 xen-vm xapi: [error|xen-vm|26628549 |Async.VM.migrate_send R:b60d2c889e8c|xapi] Caught Server_error(VM_BAD_POWER_STATE, [ OpaqueRef:92efd4b1-f1c4-4760-91e9-8c7f744cdeea; halted, suspended; running ]) while destroying VM uuid=4ea00c91-c306-c551-87a3-ba0c0fd8bf61 on destination host
Jan 11 08:28:03 xen-vm xapi: [error|xen-vm|26719641 |Async.VM.migrate_send R:cc817c26ddbd|backtrace] VM.assert_can_migrate D:f6152f70a65f failed with exception Server_error(VM_BAD_POWER_STATE, [ OpaqueRef:5008bd88-786a-4bde-a428-b137113a94d6; Halted; Running ])
Jan 11 08:28:03 xen-vm xapi: [error|xen-vm|26719641 |Async.VM.migrate_send R:cc817c26ddbd|backtrace] Raised Server_error(VM_BAD_POWER_STATE, [ OpaqueRef:5008bd88-786a-4bde-a428-b137113a94d6; Halted; Running ])
Jan 11 08:28:03 xen-vm xapi: [error|xen-vm|26719641 ||backtrace] Async.VM.migrate_send R:cc817c26ddbd failed with exception Server_error(VM_BAD_POWER_STATE, [ OpaqueRef:5008bd88-786a-4bde-a428-b137113a94d6; Halted; Running ])
Jan 11 08:28:03 xen-vm xapi: [error|xen-vm|26719641 ||backtrace] Raised Server_error(VM_BAD_POWER_STATE, [ OpaqueRef:5008bd88-786a-4bde-a428-b137113a94d6; Halted; Running ])

DustinB

And searching for the UUID I'm migrating.

cat /var/log/xensource.log | grep "dfecebe0-6ac1-b511-85ee-b43aa0223578"
Jan 11 05:42:02 xen-vm xenopsd-xc: [debug|xen-vm|39 |Async.VM.migrate_send R:b60d2c889e8c|xenops] EVENT on other VM: dfecebe0-6ac1-b511-85ee-b43aa0223578
Jan 11 08:28:02 xen-vm xapi: [debug|xen-vm|26719641 |Async.VM.migrate_send R:cc817c26ddbd|audit] VM.migrate_send: VM = 'dfecebe0-6ac1-b511-85ee-b43aa0223578 (Lab - Redhat8)'
Jan 11 08:28:02 xen-vm xapi: [debug|xen-vm|26719641 |VM.assert_can_migrate D:f6152f70a65f|audit] VM.assert_can_migrate: VM = 'dfecebe0-6ac1-b511-85ee-b43aa0223578 (Lab - Redhat8)'
Jan 11 08:28:02 xen-vm xapi: [debug|xen-vm|587 |xapi events D:4a4835d5324c|xenops] Event on VM dfecebe0-6ac1-b511-85ee-b43aa0223578; resident_here = true
Jan 11 08:28:02 xen-vm xapi: [ info|xen-vm|26719650 ||export] VM.export_metadata: VM = 'dfecebe0-6ac1-b511-85ee-b43aa0223578' ('Lab - Redhat8'); with_snapshot_metadata = 'true'; include_vhd_parents = 'false'; preserve_power_state = 'true
Jan 11 08:28:02 xen-vm xapi: [debug|xen-vm|587 |xapi events D:4a4835d5324c|xenops] Event on VM dfecebe0-6ac1-b511-85ee-b43aa0223578; resident_here = true
Jan 11 08:28:03 xen-vm xapi: [debug|xen-vm|587 |xapi events D:4a4835d5324c|xenops] Event on VM dfecebe0-6ac1-b511-85ee-b43aa0223578; resident_here = true
Jan 11 09:36:07 xen-vm xapi: [debug|xen-vm|26725037 UNIX /var/lib/xcp/xapi||cli] xe vm-list uuid=dfecebe0-6ac1-b511-85ee-b43aa0223578 username=root password=(omitted)
Jan 11 09:36:14 xen-vm xapi: [debug|xen-vm|26725079 UNIX /var/lib/xcp/xapi||cli] xe vm-list uuid=dfecebe0-6ac1-b511-85ee-b43aa0223578 username=root password=(omitted)

Danp

Server_error(VM_BAD_POWER_STATE, [ OpaqueRef:5008bd88-786a-4bde-a428-b137113a94d6; Halted; Running ]

Based on the description here, it appears to be detecting a halted VM, but the VM is expected to be running.

Are you able to location a VM with the UUID of 5008bd88-786a-4bde-a428-b137113a94d6 on either server?

P.S. You could also try this from one of your other XO VMs or even spin up a new one to see if that solves the problem.

DustinB

@Danp said in Live Migrate between two stand alone pools fails:

Server_error(VM_BAD_POWER_STATE, [ OpaqueRef:5008bd88-786a-4bde-a428-b137113a94d6; Halted; Running ]

Based on the description here, it appears to be detecting a halted VM, but the VM is expected to be running.

Are you able to location a VM with the UUID of 5008bd88-786a-4bde-a428-b137113a94d6 on either server?

P.S. You could also try this from one of your other XO VMs or even spin up a new one to see if that solves the problem.

Good find, the UUID 5008bd88-786a-4bde-a428-b137113a94d6 is somehow attached to a different VM.

But when I look at the VM itself, it has a different UUID (within XO).

DustinB

Okay now I'm just confused and this must be an XO version bug, I added the 3rd host (that I was wanting to migrate these VMs too) and it shows the VM there.

So maybe this is just a display bug.