Migration woes - SR_BACKEND_FAILURE_78

garethw

Hi All,

I'm in the process of migrating some VMs from one server to another. After a couple of initial hicups I have moved 3 of the 4 VMs that I need leaving one large (circa 2tb) one to move.

I was initially having problems because the migration was taking more than 24 hours, so have added "pending_task_timeout = 172800" to a file in /etc/xapi.conf.d and restarted the toolstack.

I am now getting a SR_BACKEND_FAILURE_78 error. Google hasn't helped much (other than a post on the LTS forums from a couple of years ago that I tried to help answer!) but I wondered if anyone could give me some more details on what that specific failure means. Full log is below.

Thanks in advance for any assistance.

vm.migrate
{
  "vm": "bb7c8076-728d-4a09-38f1-b227c3d16c2e",
  "mapVdisSrs": {
    "0c793c82-7768-4490-9c42-48b2abd8c819": "baf6a596-d7bd-c9fe-1a16-e1296938194a",
    "582c36f0-a599-4546-9011-a545674dc55e": "ee1d59e1-3036-3764-a152-0b99812b81e7"
  },
  "mapVifsNetworks": {
    "a8ce03de-1675-f19a-a94b-1b50822bd3ff": "d121db64-65ab-ce28-ea15-4e80fd60be5d"
  },
  "migrationNetwork": "aa838565-8ec5-c814-4b17-c8c9d715d54a",
  "sr": "ee1d59e1-3036-3764-a152-0b99812b81e7",
  "targetHost": "b9ede076-9a6c-42f4-b08b-50d84952f8fc"
}
{
  "code": 21,
  "data": {
    "objectId": "bb7c8076-728d-4a09-38f1-b227c3d16c2e",
    "code": "SR_BACKEND_FAILURE_78"
  },
  "message": "operation failed",
  "name": "XoError",
  "stack": "XoError: operation failed
    at operationFailed (/usr/local/lib/node_modules/xo-server/node_modules/xo-common/src/api-errors.js:21:32)
    at file:///usr/local/lib/node_modules/xo-server/src/api/vm.mjs:482:15
    at Object.migrate (file:///usr/local/lib/node_modules/xo-server/src/api/vm.mjs:469:3)
    at Api.callApiMethod (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/api.mjs:304:20)"
}

garethw

Stand by...

I realised that the failure happened within seconds of me starting the migration so did some more digging.

It might be because the previous failed migrations left the partially migrated disks on the SR of the destination and I had run out of disk space as a result... I've cleared the failed disks and started again.

Why do you always notice these things just after you post to a forum...

olivierlambert

Because writing things down helps to realize some stuff It's fine, keep us posted!

garethw

@olivierlambert it's now 22 hours in with an estimated 13 hours to go... Fingers crossed.

garethw

Sad times...

Failed again, rhis probably needs a different post as it's a totally different error but if anyone spots this and has any ideas I'd appreciate the pointers;

vm.migrate
{
  "vm": "bb7c8076-728d-4a09-38f1-b227c3d16c2e",
  "mapVdisSrs": {
    "0c793c82-7768-4490-9c42-48b2abd8c819": "baf6a596-d7bd-c9fe-1a16-e1296938194a",
    "582c36f0-a599-4546-9011-a545674dc55e": "ee1d59e1-3036-3764-a152-0b99812b81e7"
  },
  "mapVifsNetworks": {
    "a8ce03de-1675-f19a-a94b-1b50822bd3ff": "d121db64-65ab-ce28-ea15-4e80fd60be5d"
  },
  "migrationNetwork": "aa838565-8ec5-c814-4b17-c8c9d715d54a",
  "sr": "ee1d59e1-3036-3764-a152-0b99812b81e7",
  "targetHost": "b9ede076-9a6c-42f4-b08b-50d84952f8fc"
}
{
  "code": 21,
  "data": {
    "objectId": "bb7c8076-728d-4a09-38f1-b227c3d16c2e",
    "code": "INTERNAL_ERROR"
  },
  "message": "operation failed",
  "name": "XoError",
  "stack": "XoError: operation failed
    at operationFailed (/usr/local/lib/node_modules/xo-server/node_modules/xo-common/src/api-errors.js:21:32)
    at file:///usr/local/lib/node_modules/xo-server/src/api/vm.mjs:482:15
    at runMicrotasks (<anonymous>)
    at runNextTicks (internal/process/task_queues.js:60:5)
    at processImmediate (internal/timers.js:435:9)
    at process.topLevelDomainCallback (domain.js:147:15)
    at process.callbackTrampoline (internal/async_hooks.js:132:14)
    at Object.migrate (file:///usr/local/lib/node_modules/xo-server/src/api/vm.mjs:469:3)
    at Api.callApiMethod (file:///usr/local/lib/node_modules/xo-server/src/xo-mixins/api.mjs:304:20)"
}

olivierlambert

At least, it's not related to XO

Something went wrong during the migration, could be many things. The logs on source and destination hosts might provide details on what went wrong.

Danp

Maybe you should consider some alternative methods to achieve your migration goal, such as using one of XO's backup methods to replicate the VM on the target server.

olivierlambert

Warm migration is probably the easiest one

garethw

@olivierlambert said in Migration woes - SR_BACKEND_FAILURE_78:

logs on source and destination hosts

Are we talking, looking through /var/log or something that is in the XO interface somewhere? Going home, hosts, hostname, logs shows "Nothing so far" for both hosts.

@Danp said in Migration woes - SR_BACKEND_FAILURE_78:

consider some alternative methods to achieve your migration goal

@olivierlambert said in Migration woes - SR_BACKEND_FAILURE_78:

Warm migration is probably the easiest one

Migrate whilst the VM is running? I was trying a migrate whilst shutdown as I assumed that would be easier.

Trying one more time.

olivierlambert

No, I specifically said warm, not live or hot migration It's an XO feature allowing to migrate in a kind of "safer" way.

garethw

@olivierlambert Ok, thanks. I'll look up warm migration and go from there.

garethw

I'm struggling to find the warm migration option (looking in advanced) is it on the features that's part of Starter/Enterprise/Premium? I have to admit that this XO is only on the Free account.

olivierlambert

You need a fully up to date XO (check that first). If you are on XOA Free, you might need a trial (send me a private chat message with your email registered to this XOA, so I can unlock you the free trial). Alternatively, you can also use XO from the sources