Import from ESXi 6 double importing vmdk file?
-
My biggest VM has five 2TB drives attached to it. When I try to import the powered off VM, it sets up drive copy tasks that are like:
[xva-disp-import][ESXI]CDrive-flat.vmdk
[xva-disp-import][ESXI]DDrive-flat.vmdk
[xva-disp-import][ESXI]EDrive-flat.vmdk
[xva-disp-import][ESXI]CDrive-flat.vmdk <- boot drive listed twice
[xva-disp-import][ESXI]FDrive-flat.vmdk
[xva-disp-import][ESXI]GDrive-flat.vmdkConsidering that it says it's going to take two days to finish importing all of these, I'm wondering if it's actually importing that file twice. If it's simultaneously trying to read the file two times then it's going to slow both transfers and make it take even longer than copying it twice sequentially.
Unfortunately, I'm limited to 1GB LAN on the existing server. That's probably a lot of the reason that the import is going so slowly.
I also wonder if there is a way I could do the import without making simultaneous copies of the different drives but instead copy the sequentially. I'm guessing it's oversaturating the 1GB ethernet anyway so I see no benefit to copying all five files at once.
In case it matters, two of the five drives are on a mirrored array but the other three are on individual 2TB drives. They aren't raw access in ESXi but are configured as virtual disks on those physical drives. This is my backup VM and those drives didn't need redundancy since they are a backup target for personal files.
Running XO 5.93 from source. Still in the testing phase to migrate to XCP-ng and this is the biggest VM I've tried to migrate, the other ones should be easier but this one is the least business-critical so it is a good one to try early.
-
Pinging @florent for an opinion. That would be even better to discuss your infrastructure in detail with a dedicated/private ticket
-
Solved one mystery and now I have another, which could be related.
First of all, the duplicate file name is because my VM in ESXi has two files with the same name but on different data stores. I didn't mean to do that. Makes me wonder if that caused issues during the import, if it was trying to copy one of them over the other, and if that's what caused the import to fail. The task list says the import failed. It seems like it copied all the vmdk files over. The raw log for that task contains one
ERR_STREAM_PREMATURE_CLOSEerror and six"already finalized or destroyed"errors. They all happened during theCold import of disks scsi0:xstep. -
Thanks for the feedback, let's see if @florent has some tricks in his sleeve
-
@CodeMercenary the name shouldn't create problem since we use the disk refs
the ERR_STREAM_PREMATURE_CLOSE is probably because the import is limited to 2TB minus 8MB of data reserved for metadata , and one disk of your may be exactly 2TBThe import till 5.91 ( decemebre) was doiing 2 passes, but now it should work in one pass
-
@florent Thank you for the feedback. My last import attempt was with 5.93. I had been hopeful that the changes in 5.91 would make the migration work because my prior attempt was pre-5.91.
The drives mounted to that VM are 60.00GB, 1500.00GB, 1819.88GB, 1799.98GB, 1819.88GB and 1023.98GB. That's what it shows for drives sizes in Windows Disk Management. In vCenter the largest are listed as 1.78TB.
Ideally this should be the first VM I migrate. I'm still working on figuring out if XCP-ng will work for us and this migration failure, plus the eclipse in Texas has stalled the testing by a couple weeks now.
Maybe I could detach those drives in ESXi, migrate the VM, change the mac and IP then copy the data between the old and new VM with both of them running. That sounds painful and slow.
Or, probably better, I could create a new temporary VM on XCP-ng with several drive attached then sync those drives through a network share, then detach the drives and migrate the other VM, attach the drives to the migrated VM and remove that temporary VM.
The drives attached to that VM don't have 2TB of data on them, though they are quite full. Perhaps I could shrink the volume in Windows then shrink the vmdk file in ESXi. Not sure if ESXi allows that but I'll have to do some research to find out.
Also, it would be cool if a future version of XO would determine that the vmdk is too big to import so it would fail right away rather than failing after saturating my network bandwidth for 24 to 48 hours. Fortunately in my network that's not a huge deal for the users but it has meant that I've spent 4 days trying to import something that will always fail, since I've tried this import a couple times now.
-
@CodeMercenary That are some nice volume to transfer
Agent/file based migration is on the table, but there are a lot of hurdles to stay platform independant. Depending on the plarform you can use a combiation of rsync or robocopy /vss to transfer the file while both the VM are running, "freeze" the source VM, make a last sync and provide the services from the destination (xcp-ng) VM
the right size check is updated and the fix will reach master in a few days : https://github.com/vatesfr/xen-orchestra/pull/7554/commits/6e899287c6a1a26cb95099cbecc5d080c91c76f0
Please note that what is limited is the capactiy of the drive, not the data present on it -
@florent Yeah, it's a lot of data, thankfully my other VMs are not nearly as large. I'm still not sure why it failed when none of the virtual drives are 2TB. The largest ones are configured with a 1.82TB max so even the capacity of the drive is less than the max.
I'm moving ahead with a file level sync attempt to see if that works.
To be clear, this post was as much or more about helping you figure out what's wrong so other people don't have the same issue, than it is about making this import work for my VM. With the flood you are getting from VMware refugees, I figure I'm not the only person with large drives to import. In other words, if there's something I can do to help you figure out why it fails then I'm willing to help.
-
@CodeMercenary thanks
these little correction improve the process for everyone -
I figured, for the purpose of completion, I'd document what I did to solve this problem.
- Create a new VM on the target XCP-ng host
- Attach 6 virtual drives to the VM in the sizes of the VM I need to migrate
- Install Windows Server 2022 in the target VM
- Format the additional 5 drives
- Share the 5 drives over the network
- From source VM on ESXi
- Map network drives to each of the 5 drives in the target VM
- Copy files and folders to each drive
- Originally tried using RichCopyUI which uses Robocopy but it kept crashing
- Finally used FreeFileSync over the course of several days and nights to do an initial copy of the data
- Performed one final mirror of the data from the source to the target
- Shut down the source VM
- In vCenter, detach all but the boot drive from the source VM
- Migrate the VM from ESXi to XCP-ng using XO
- Attach drive files from target VM to migrated VM
- Boot migrated VM, it had no network access and the CD wasn't working but a couple reboots fixed it.
- Only 2 of the 5 extra drives appeared. Two attached automatically but 3 weren't even visible in Disk Manager.
- Install XCP-ng guest tools and reboot
- Extra drives now working inside the VM
- Had to fight a little with network settings and my DHCP server to get the same IP address assigned
- Did some testing, had to tweak a few things in some software but everything seems to be working normally again without too much work
- Shut down the old host running ESXi - one step closer to severing ties with VMware and also made my server room quieter and cooler
- Create a new VM on the target XCP-ng host
