XO CE migrate fails for VM - "Internal Error"

[vmpr]

Re: XO CE migrate fails for WinServer 2019 VM - "Internal Error"

Hi everyone, I have a similar problem like the earlier thread. My xcp-ng 8.0 pool has two members and the VMs are on local storage. so every time I update ich have to migrate all the VMS from one server to the other. with XOA it should be no problem with some clicks, but it doesn't work

when I move a VM via xoa I can see how it moves data from one host to the other and then I get this error:

vm.migrate
{
  "mapVdisSrs": {
    "5c9b25da-b5a1-4050-b8ed-4b261cbb0339": "c34046c7-658d-66c2-fdec-292d29a0681c",
    "be0a7dfd-6950-4fef-9965-342a05a06093": "c34046c7-658d-66c2-fdec-292d29a0681c"
  },
  "migrationNetwork": "0a6c51c6-93d9-d9f9-2a76-e8c2330c4448",
  "targetHost": "cc689219-af8a-44e0-b096-23da85534502",
  "vm": "acef4db1-2087-58c2-f5ee-6d48beb34fb1"
}
{
  "code": 21,
  "data": {
    "objectId": "acef4db1-2087-58c2-f5ee-6d48beb34fb1",
    "code": "INTERNAL_ERROR"
  },
  "message": "operation failed",
  "name": "XoError",
  "stack": "XoError: operation failed
    at factory (/home/toor/xen-orchestra/packages/xo-common/src/api-errors.js:21:32)
    at /home/toor/xen-orchestra/packages/xo-server/src/api/vm.js:520:15
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at Object.migrate (/home/toor/xen-orchestra/packages/xo-server/src/api/vm.js:509:3)
    at Api.callApiMethod (/home/toor/xen-orchestra/packages/xo-server/src/xo-mixins/api.js:291:20)"
}

in XCP-ng Center it shows this error for that move:

"Failed","Migrating VM 'it-dd-prx'
Internal error: Storage_error ([S(Vdi_does_not_exist);S(57b54124-67fe-4a86-9e5c-f2a1e1bb49c1)])
Time: 00:04:14","amy","Aug 3, 2020 5:36 AM"

If I do the same move via XCP-ng Center, it works without problems - but you can't move more than three VMs at the same time, so that's not really an option.

When I do the migrate via CLI command it also works without any problems:
[07:31 sheldon ~]# xe vm-migrate vm=it-dd-prx host=amy remote-master=10.13.38.200 remote-username=root remote-password=*************

I guess there is something wrong with the API call from XOA to migrate the VMs from local storage to local storage - if you need more informations please let me know!

thanks, Ringo

[olivierlambert]

Do you have any CD plug to this VM?

[vmpr]

thy for the reply Olivier - just checked, no cd inserted or DVD drive connected to VM - its the same VM every time for all different migrate methods

[olivierlambert]

Do you have snapshots for this VM?

[vmpr]

yes it has three normal snapshots

[olivierlambert]

I have the feeling there's a connection between those snapshots and the error.

Could you deploy an XOA, open a support tunnel so we can take a look?

[vmpr]

I could do that sure, is there a way to use the xoa ce for that support tunnel?
I dug deeper into the problem and tested a little bit:

• removed all the snapshots from the above-mentioned VM and wow - no you can migrate it via XOA from local storage to local storage
• I removed all snapshots from my other VMS, besides the one from the nightly xoa backup job
  • didn't work -> every VM logged the same error
• I removed all snapshots from all VMs
  • now it works to migrate all the VMs via XOA but that is not really a solution, because xoa backup runs every night and i don't want to remove all snapshots from the VMs every time I move them to the other host

It also doesn't explain why the move works with snapshots via XCP-ng Center and CLI

any thoughts from your side?
thanks, guys! Ringo

[gn_ro]

@vmpr I think the error is because you run out of space when moving. Try to use a larger SR or move one VM at a time, or serialize the operation if you don't have the space for.

P.S.: Every move creates snapshots for every disk. When moving from CLI or XCP-ng Center you move one VM at a time and needs less space. I think XOA starts to move all VM's at once (@olivierlambert correct me if I'm wrong), so the space required is more and soon exhausted. You can check this by starting a move from XOA and checking SR utilization (actual use, total and allocated space).

[vmpr]

thanks @gn_ro - but Nah that's not the problem, its a 6TB storage and 1TB ist used ... 5TB space should be enough. I am also moving three vms at the same time via xcp-ng centre and cli, as XOA does ... that's the limit of how many VMS you can move simultaneous from local storage to local storage.

[olivierlambert]

Yep, if it was an XCP-ng issue, it would also fails with xe command.

[vmpr]

@olivierlambert any idea what could be the problem?

[tony]

I had this issue before, I couldn't migrate VM that was running from NFS storage to another host and store it in NFS storage as well, I had to migrate it to the new host's local storage first then move VDI to NFS storage later.

However I tried again with XCP-ng center and migrating VM stored on NFS to another host's NFS storage works fine.

[olivierlambert]

@vmpr Please create an issue on Xen Orchestra GitHub repo, this way XO dev will see it