XO CE migrate fails for VM - "Internal Error"

vmpr

Re: XO CE migrate fails for WinServer 2019 VM - "Internal Error"

Hi everyone, I have a similar problem like the earlier thread. My xcp-ng 8.0 pool has two members and the VMs are on local storage. so every time I update ich have to migrate all the VMS from one server to the other. with XOA it should be no problem with some clicks, but it doesn't work

when I move a VM via xoa I can see how it moves data from one host to the other and then I get this error:

vm.migrate
{
  "mapVdisSrs": {
    "5c9b25da-b5a1-4050-b8ed-4b261cbb0339": "c34046c7-658d-66c2-fdec-292d29a0681c",
    "be0a7dfd-6950-4fef-9965-342a05a06093": "c34046c7-658d-66c2-fdec-292d29a0681c"
  },
  "migrationNetwork": "0a6c51c6-93d9-d9f9-2a76-e8c2330c4448",
  "targetHost": "cc689219-af8a-44e0-b096-23da85534502",
  "vm": "acef4db1-2087-58c2-f5ee-6d48beb34fb1"
}
{
  "code": 21,
  "data": {
    "objectId": "acef4db1-2087-58c2-f5ee-6d48beb34fb1",
    "code": "INTERNAL_ERROR"
  },
  "message": "operation failed",
  "name": "XoError",
  "stack": "XoError: operation failed
    at factory (/home/toor/xen-orchestra/packages/xo-common/src/api-errors.js:21:32)
    at /home/toor/xen-orchestra/packages/xo-server/src/api/vm.js:520:15
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at Object.migrate (/home/toor/xen-orchestra/packages/xo-server/src/api/vm.js:509:3)
    at Api.callApiMethod (/home/toor/xen-orchestra/packages/xo-server/src/xo-mixins/api.js:291:20)"
}

in XCP-ng Center it shows this error for that move:

"Failed","Migrating VM 'it-dd-prx'
Internal error: Storage_error ([S(Vdi_does_not_exist);S(57b54124-67fe-4a86-9e5c-f2a1e1bb49c1)])
Time: 00:04:14","amy","Aug 3, 2020 5:36 AM"

If I do the same move via XCP-ng Center, it works without problems - but you can't move more than three VMs at the same time, so that's not really an option.

When I do the migrate via CLI command it also works without any problems:
[07:31 sheldon ~]# xe vm-migrate vm=it-dd-prx host=amy remote-master=10.13.38.200 remote-username=root remote-password=*************

I guess there is something wrong with the API call from XOA to migrate the VMs from local storage to local storage - if you need more informations please let me know!

thanks, Ringo

olivierlambert

Do you have any CD plug to this VM?

vmpr

thy for the reply Olivier - just checked, no cd inserted or DVD drive connected to VM - its the same VM every time for all different migrate methods

olivierlambert

Do you have snapshots for this VM?

vmpr

yes it has three normal snapshots

olivierlambert

I have the feeling there's a connection between those snapshots and the error.

Could you deploy an XOA, open a support tunnel so we can take a look?

vmpr

I could do that sure, is there a way to use the xoa ce for that support tunnel?
I dug deeper into the problem and tested a little bit:

removed all the snapshots from the above-mentioned VM and wow - no you can migrate it via XOA from local storage to local storage
I removed all snapshots from my other VMS, besides the one from the nightly xoa backup job
- didn't work -> every VM logged the same error
I removed all snapshots from all VMs
- now it works to migrate all the VMs via XOA but that is not really a solution, because xoa backup runs every night and i don't want to remove all snapshots from the VMs every time I move them to the other host

It also doesn't explain why the move works with snapshots via XCP-ng Center and CLI

any thoughts from your side?
thanks, guys! Ringo

gn_ro

@vmpr I think the error is because you run out of space when moving. Try to use a larger SR or move one VM at a time, or serialize the operation if you don't have the space for.

P.S.: Every move creates snapshots for every disk. When moving from CLI or XCP-ng Center you move one VM at a time and needs less space. I think XOA starts to move all VM's at once (@olivierlambert correct me if I'm wrong), so the space required is more and soon exhausted. You can check this by starting a move from XOA and checking SR utilization (actual use, total and allocated space).

vmpr

thanks @gn_ro - but Nah that's not the problem, its a 6TB storage and 1TB ist used ... 5TB space should be enough. I am also moving three vms at the same time via xcp-ng centre and cli, as XOA does ... that's the limit of how many VMS you can move simultaneous from local storage to local storage.

olivierlambert

Yep, if it was an XCP-ng issue, it would also fails with xe command.

vmpr

@olivierlambert any idea what could be the problem?

tony

I had this issue before, I couldn't migrate VM that was running from NFS storage to another host and store it in NFS storage as well, I had to migrate it to the new host's local storage first then move VDI to NFS storage later.

However I tried again with XCP-ng center and migrating VM stored on NFS to another host's NFS storage works fine.

olivierlambert

@vmpr Please create an issue on Xen Orchestra GitHub repo, this way XO dev will see it