Hmmmmm I need to look back through some of my old posts and notes but I had a similar issue with a VM with large number of disks. The disks themselves being big wasn't the issue, it was just having a huge number of them that eventually cased a problem. Ended up migrating by moving a single VHD and rebuilding things (the specific system didn't really "need" all the VHDs so it was an easy fix for me, wouldn't be the case here haha).
If you are trying to reduce the number of hosts though, why would migrating the VCSA help? If you're powering off a host then wouldn't you not need the VCSA to be running anyway? Or do you have more than 1 ESXi instance? Might be easiest to just migrate it to another ESXi host if that's the goal, since you won't need the VCSA once you fully move to XCP-ng (assuming you are doing so).