Hello Vates and XCP-ng community,

I have updated my Xen Orchestra from sources VM and want to report that after updating my mirror backup job does not work anymore. I updated to commit 875ceb7cab9735efc29c345603a4280cb6072fbc (feat: release XO 6.3.1).

Prior to updating XO it has been working fine for weeks. I did not change anything about it's config.

I get "Error: Cannot read properties of undefined (reading 'id')" for each any every VM that the mirror job tries to mirror.

It is a "mirror incremental" backup job.

Is this a known issue?
Did anyone else encounter this?

If a XO developer needs logs, I can provide them in a private message as I am not sure if they contain sensitive information in this case.

My XO VM is running on Debian 13 and Node 22 by the way.

Thanks and best regards

@Pilow I am using Xen Orchestra from sources. I think the commits that improved RAM / heap usage are not released for XOA. But I suspect a XOA release coming soon as they already pushed "technical release" commit which is mostly an indicator that a XOA release will follow soon.

I can confirm that recent changes in Xen Orchestra improved the situation a lot.
I am still monitoring and will keep testing before coming to a conclusion but so far the backups did not crash due to Node heap size being exhausted .

I can confirm that when using Citrix/Xenserver guest utilities version 8.4 (https://github.com/xenserver/xe-guest-utilities/releases/tag/v8.4.0) memory ballooning / DMC is working fine.

After live migration the RAM of the linux guest is expanded to dynamic_max again.

So this issue was in fact caused by Rust based xen-guest-agent.

For now I'll keep using Citrix/Xenserver guest utilities on my Linux guests until the feature is implemented in Vates rust-based guest utilities.

Best regards

@planedrop Pretty sure the official and recommended way is to use the V2V (VMWare to Vates) tool that is baked into Xen Orchestra.

I used that tool back when I migrated from our old VMWare Cluster to the new XCP-ng environment. It worked without any issues.

Best approach IMO would be to disconnect the unneeded virtual disks on the source VM on VMWare side and then use V2V tool to migrate over the VM.
I did it this way for a VM with disk bigger than 2TB aswell. After migrating the VM I created an LVM volume on the XCP-ng VM with the required size and rsync-ed the remaining data from VMDK > 2TB to target XCP-ng LVM volume.

Documentation: https://docs.xen-orchestra.com/v2v-migration-guide

@dinhngtu @teddyastie Do you have a time line / rough estimated release date regarding Rust-based xen-guest-agent version 1.0?
Currently evaluating if I need to switch my Linux guests back to Citrix guest tools in the meantime.

Looking forward to a reply.

Thank you in advance.

@olivierlambert Can you provide insights? Is nested virtualization as in running Hyper-V inside a XCP-ng VM expected to work?

@User-cxs I have one last idea:

• shutdown your VM
• disable nested virtualization for your VM via Xen Orchestra
• SSH login on your XCP-ng host (pool master)
• execute "xe vm-param-set uuid=<UUID> platform:exp-nested-hvm=true" (UUID should be the UUID of your VM)
• start your VM and try again

By following these steps you will use Xen parameter "exp-nested-hvm" instead of "nested-virt" on your VM. This may or may not make a difference.

If that does not change anything I am out of ideas.
Best regards

@User-cxs PV drivers / Xen Tools installed? Was the system rebooted cleanly after enabling nested virtualization/Hyper-V?

@User-cxs Is there an error message when clicking on / hovering over the yellow warning symbol next to "Nested virtualization" on your Xen Orchestra screenshot?

You need that feature for Docker desktop virtualization to work in your VM.

Also: do you have PV drivers / guest utilities installed on your VM?

@benapetr Hmmm I am also using XCP-ng 8.3 (all patches installed) and latest XO from sources... On my end XO6 is working fine. Maybe it is caused by something specific to your setup/configuration... Maybe start with a fresh XO configuration and check if it then works?

//EDIT: what is your underlying OS version and Node JS version used for building / running XO?

//EDIT2: Also are you running XO behind a reverse proxy? If so try disabling it as I saw issues like these being caused by reverse proxy setups. These kind of issues can be investigated by using dev tools of your browser and checking network / console tab while loading XO.

@benapetr Is your XO fully up to date? I remember that there was a bug that caused this spinning wheel / infinite loading on XO6 that got fixed already. Maybe the issue you faced is related to that. I do not know about compatibility with 8.2 but can imagine that it is not supported as it is EOL so probably no compatibility guaranteed by Vates.

@abudef said:

@MajorP93 Sure I did

Hmm you previously mentioned different command so I was not sure if you really followed documentation correctly.

@abudef said:

Nested virtualization doesn’t work very well in Xen. For example, when I set up a small test playground, I had XCP-ng 8.3 as the primary host and another XCP-ng running on it as a nested host. When I then booted Debian 12 on the nested host, it caused the entire nested host to crash and reboot. On the other hand, Windows VMs on the nested host run quite well.

Later, when I was preparing a test lab, I installed ESXi 8.0U3e on a Dell server and then deployed four virtualized XCP-ng 8.3 hosts on top of it. A number of Linux and Windows VMs run on them without any issues.

Did you follow the official documentation for nested virtualization?

https://docs.xcp-ng.org/guides/xcpng-in-a-vm/#nested-xcp-ng-using-xcp-ng

Most importantly setting via command line on pool master:

xe vm-param-set uuid=<UUID> platform:exp-nested-hvm=true

and

xe vm-param-set uuid=<UUID> platform:nic_type="e1000"

@dom0 As already previously mentioned XCP-ng Center / XenCenter are not officially supported and a third-party product.
It is generally advised to use Xen Orchestra for all administration / management tasks.

If it is a requirement for you to use a thick client (such as XCP-ng Center) you might want to try XenAdminQt: https://github.com/benapetr/XenAdminQt

It is also not officially supported but a very new project that gets updated frequently. Maybe that one works better for you.

@dinhngtu Looking forward to that! I'll stick with the Rust guest utilities for the time being—hope to see that new release soon!

Can also confirm that I was able to apply this round of patches using rolling update method without any issues or slowdowns on a pool of 5 hosts.

@dinhngtu Also if it is related to the Rust-based guest agent: do you guys plan to fix the issue and release a new version soon or is it advised to revert to Xenserver guest agent for now?

@dinhngtu Yes, I am using the rust based guest agent (https://gitlab.com/xen-project/xen-guest-agent) on all of my Linux guests.

Good job tracking the issue down to that!

Is a suspend/resume performed automatically during live migration?

//EDIT: I switched from Citrix/Xenserver guest agent to the Vates rust one somewhere in January so maybe my assumption of this issue being related to January round of patches was wrong.
Or maybe I did not see this issue since the ballooning down to dynamic min was broken before said round of patches according to changelog.

@Pilow Yeah, true.
Also during my CBT-enabled-backups tests live migrations did trigger the fallback to a full from time to time aswell (VM residing on a different host during backup run compared to last backup run).
But I did those tests ~6 months ago so maybe some fixes have been applied in that regard.

You are right, in a thick provision SR / block based scenario taking a snapshot would result in the same size of base VHD being allocated again for the purpose of snapshotting... not really practical.

I really hope that CBT receives some more love as we plan to move our storage to a vSAN cluster and intend to use iSCSI instead of NFS by that time so using CBT would also be our best bet then...
CBT also reduces the load on the SR (as in I/O) as it removes the need to constantly coalesce disks during backup job re-creation / deletion of snapshots.

@acebmxer Interesting during my tests CBT was not really stable as in backups falling back to full quite often. I recall reading somewhere in the documentation that CBT has some quirks so it seems to be a known issue.