XO CE migrate fails for VM - "Internal Error"
-
Re: XO CE migrate fails for WinServer 2019 VM - "Internal Error"
Hi everyone, I have a similar problem like the earlier thread. My xcp-ng 8.0 pool has two members and the VMs are on local storage. so every time I update ich have to migrate all the VMS from one server to the other. with XOA it should be no problem with some clicks, but it doesn't work
when I move a VM via xoa I can see how it moves data from one host to the other and then I get this error:
vm.migrate { "mapVdisSrs": { "5c9b25da-b5a1-4050-b8ed-4b261cbb0339": "c34046c7-658d-66c2-fdec-292d29a0681c", "be0a7dfd-6950-4fef-9965-342a05a06093": "c34046c7-658d-66c2-fdec-292d29a0681c" }, "migrationNetwork": "0a6c51c6-93d9-d9f9-2a76-e8c2330c4448", "targetHost": "cc689219-af8a-44e0-b096-23da85534502", "vm": "acef4db1-2087-58c2-f5ee-6d48beb34fb1" } { "code": 21, "data": { "objectId": "acef4db1-2087-58c2-f5ee-6d48beb34fb1", "code": "INTERNAL_ERROR" }, "message": "operation failed", "name": "XoError", "stack": "XoError: operation failed at factory (/home/toor/xen-orchestra/packages/xo-common/src/api-errors.js:21:32) at /home/toor/xen-orchestra/packages/xo-server/src/api/vm.js:520:15 at runMicrotasks (<anonymous>) at processTicksAndRejections (internal/process/task_queues.js:97:5) at Object.migrate (/home/toor/xen-orchestra/packages/xo-server/src/api/vm.js:509:3) at Api.callApiMethod (/home/toor/xen-orchestra/packages/xo-server/src/xo-mixins/api.js:291:20)" }
in XCP-ng Center it shows this error for that move:
"Failed","Migrating VM 'it-dd-prx' Internal error: Storage_error ([S(Vdi_does_not_exist);S(57b54124-67fe-4a86-9e5c-f2a1e1bb49c1)]) Time: 00:04:14","amy","Aug 3, 2020 5:36 AM"
If I do the same move via XCP-ng Center, it works without problems - but you can't move more than three VMs at the same time, so that's not really an option.
When I do the migrate via CLI command it also works without any problems:
[07:31 sheldon ~]# xe vm-migrate vm=it-dd-prx host=amy remote-master=10.13.38.200 remote-username=root remote-password=*************
I guess there is something wrong with the API call from XOA to migrate the VMs from local storage to local storage - if you need more informations please let me know!
thanks, Ringo
-
Do you have any CD plug to this VM?
-
thy for the reply Olivier - just checked, no cd inserted or DVD drive connected to VM - its the same VM every time for all different migrate methods
-
Do you have snapshots for this VM?
-
yes it has three normal snapshots
-
I have the feeling there's a connection between those snapshots and the error.
Could you deploy an XOA, open a support tunnel so we can take a look?
-
I could do that sure, is there a way to use the xoa ce for that support tunnel?
I dug deeper into the problem and tested a little bit:- removed all the snapshots from the above-mentioned VM and wow - no you can migrate it via XOA from local storage to local storage
- I removed all snapshots from my other VMS, besides the one from the nightly xoa backup job
- didn't work -> every VM logged the same error
- I removed all snapshots from all VMs
- now it works to migrate all the VMs via XOA but that is not really a solution, because xoa backup runs every night and i don't want to remove all snapshots from the VMs every time I move them to the other host
It also doesn't explain why the move works with snapshots via XCP-ng Center and CLI
any thoughts from your side?
thanks, guys! Ringo -
@vmpr I think the error is because you run out of space when moving. Try to use a larger SR or move one VM at a time, or serialize the operation if you don't have the space for.
P.S.: Every move creates snapshots for every disk. When moving from CLI or XCP-ng Center you move one VM at a time and needs less space. I think XOA starts to move all VM's at once (@olivierlambert correct me if I'm wrong), so the space required is more and soon exhausted. You can check this by starting a move from XOA and checking SR utilization (actual use, total and allocated space).
-
thanks @gn_ro - but Nah that's not the problem, its a 6TB storage and 1TB ist used ... 5TB space should be enough. I am also moving three vms at the same time via xcp-ng centre and cli, as XOA does ... that's the limit of how many VMS you can move simultaneous from local storage to local storage.
-
Yep, if it was an XCP-ng issue, it would also fails with
xe
command. -
@olivierlambert any idea what could be the problem?
-
I had this issue before, I couldn't migrate VM that was running from NFS storage to another host and store it in NFS storage as well, I had to migrate it to the new host's local storage first then move VDI to NFS storage later.
However I tried again with XCP-ng center and migrating VM stored on NFS to another host's NFS storage works fine.
-
@vmpr Please create an issue on Xen Orchestra GitHub repo, this way XO dev will see it