@Jonathon this is really nice to have shared, as we are looking to migrate from the RKE cluster we've deployed on bare-metal Xen to XCP-ng VMs to setup an RKE2 cluster to migrate to.

Will review this and probably have a bunch of questions!

@Forza There are various topics touching on how SMAPIv1 is a bottleneck here, eg: https://xcp-ng.org/forum/topic/9389/backup-migration-performance/8 - that's probably not the best example as it is also more on backups than migration!

@pdonias nice - so I gather we can delete from the 'server' settings the secondary host that we tried adding there, but is showing an error.

From what you say, I gather the 'pool connections' is for master servers of pools so there should just be one in the settings for each pool?

@ZaphodB I have the same experience some 2.5yrs later - I don't know why there isn't some more clarity in XO as to what and when you should add a machine as a "Server" under settings. When I installed XCP-NG on my second 'host' it joined the pool but wasn't added in XO as a "Server".

@Meth0d thanks for sharing your experience. Fortunately I was able to just hit "Convert to HVM" after my imported PV failed to reboot after upgrade from 20.04 to 22.04 - was all set to load a livecd etc too!

@stevewest15 that is a shame! Thanks for sharing!

Were you able to work it out? I have a 32GB RAM domU that keeps failing migration between nodes unfortunately!

@john-c I am seeing an option for Migration compression in XO, under Xen settings on the Advanced tab for a Pool of 8.2.1 servers. Haven't tried it though.

@olivierlambert thanks for investing in this. Great news!

@olivierlambert this is a concern for me with moving to 8.3 - having a bunch of VMs just migrated from Xen with PV. I understand 8.3 doesn’t support PV and potentially some older operating systems. Will there be a page of supported OS / Distribution versions or have I just not looked hard enough for one that’s already there?

@olivierlambert thanks, my comment wasn’t meant as a criticism. I appreciate all you share so freely in both software and in the forums! Just was wondering how another user got on.

I have benefitted immensely thanks to Xen on bare metal over the years and while some people push towards KVM, I found the import process to XCP-NG very smooth once I got python2 built on the Debian12 dom0s that have served so well.

rdiff-backup was a really solid backup method for us using LVM snapshots and iscsi, so it’s just taking some time to get used to the GUI and the new CLI - and that the CLI can help resolve things stuck in the GUI. Your forums and personal posts are gold.

Automated boot testing after a backup is fantastic.

Thanks for everything!

@CodeMercenary Great question! Similar situation here. Sorry you didn't get a response.

What did you end up doing?

@mauzilla we've been running iscsi based backups with our previous 'bare metal' Debian Xen system, thanks to rdiff-backup. It worked really well for our purposes and I am still a bit uncomfortable with XO / XCP-ng backing up over a 1Gbps link. Hopefully a 10Gbps link will resolve that.

Have you considered using ZFS replication with TrueNAS as part of your backup strategy?

@mauzilla sounds like we are doing similar things! I gather you're now running TrueNAS Scale as we are? We decided to go with RC2 of Electric Eel since there's been some big architecture changes between 24.04 and 24.10.

We are just running it through testing and have run into performance issues for backups to an NFS target and I was interested to read the huge gain in speed you got from moving to iSCSI.

I was thinking of seeing if we can backup to an iSCSI target while keeping many of the VMs on NFS. Have you tried this and had any success?

@mauzilla It's been a few years, but this seems like a common issue. How have you worked around it?

@olivierlambert Thanks! Just read this and after experimenting with dynamic memory I see what you mean!

@nikade thanks for the ideas of where to look!

In my case we're testing and just have a 1Gb link between these hosts, which is what I was putting it down to.

This particular VM is a freshly migrated PV from Debian Xen with:

Memory limits (min/max)
Static: 16 MiB/16 GiB
Dynamic: 8 GiB/16 GiB

Could that Dynamic setting be the problem because as I recall it reduces the VM to 8 on migrate, so when doing the migrate perhaps 8 isn't enough for the VM?

I will try changing it to 16/16 and see if that has any noticeable impact. Thanks!

We're seeing this issue when trying to migrate a Debian VM with 16GB of RAM.

It is a worker node in a Kubernetes cluster so it is likely that the RAM changes a fair bit. It is not uncommon for the migration to fail due to the freeze hitting a 30 second time limit.

A Windows 10 Pro VM with 16GB of RAM migrates fine, because not much is changing in the RAM I expect.

Following along for recommendations! Our hosts sound very similar to @arc1 except our network speed is slower, which is one thing we are working on.

I have been migrating VMs from Xen on Debian 11/12 using the python2 script this past week and it worked best if I had ensured the VMs were booting on Debian using pygrub as the bootloader rather than the specified kernel from the Dom0.

Then all I had to do after running the script was go into Xen Orchestra and change the "PV args" for the VM under "Advanced" to "root=/dev/xvda ro" (instead of "root=/dev/xvda2 ro" from the imported config), edit the "Network" config to match the MAC addresses used by the VM before, and mount the tools disk.

The VM will then boot and I can edit the /etc/fstab file to use xvda and xvdb for the boot disk and swap respectively (instead of xvda2 and xvda1 that Debian had been using).

I edited my VM's /etc/network/interfaces (because we have standardised on having the internal network as the first ethernet port rather than the second), then mounted the tools and ran 'sudo /mnt/Linux/install.sh' before rebooting. It is a pretty quick process once you've done a dozen or so!!

I tried updating the script to use python3 but as I am not a python developer I decided it was safer to build python2.7 from sources to run on Debian 12, since unfortunately python 2 is no longer available from packages on Debian 12.

Thanks for sharing your experience here, which I thought I would just add to as it may help others migrating to xcp-ng over coming months/years. All I can say is, "Do it!". The sooner the better!

Once I have all the VM migrated in PV mode, I will then work on moving to HVM.

I also worked on trying to update the script to python3 last week and hadn't realised you had this work here so was starting from scratch.

I am not a python developer so when I got to errors in the section that is actually streaming the files I decided to stop and just finish 'making' python2.7 on the servers that I need to get the VMs off and that has worked for me.

It doesn't seem like it should be a huge amount of work left to upgrade the script to python3 but probably not something I have the skills or time for right now.

Thanks for sharing your efforts here. What have you ended up doing @josef1909