I updated my test environment and performed a few tests:

• migrating VMs back and forth between VHD based NFS SR and QCOW2 based iSCSI SR --> VMs got converted between VHD and QCOW2 just fine, live migration worked
• creation of rather big QCOW2 based VMs (2.5+ TB)
• NBD-enabled delta backups of a mixed set of VMs (small, big, QCOW2, VHD)

All tests worked fine so far. Only thing that I noticed: When converting VHD-based VMs to QCOW2 format I was not able to storage migrate more than 2 VMs at a time. XO said something about "not enough memory". That might be related to my dom0 in test environment only having 4GB of RAM. Maybe not related to VHD to QCOW2 migration path. I never saw this error in my live environment where all node's dom0 have 8GB RAM.

Update candidate looks good so far from my point of view.

@florent Thanks!
I did not have time (yet) to look into the heap export due to weekend being in between.
I will update to current master and provide you with the heap export in the following days.
Best regards

@florent Yes I am using XO from sources:

Currently on the branch that @bastien-nollet asked me to test.

Yeah sure I can export a memory heap. Can you give me instructions for that?
Also: currently I have one backup job running. Is it better to take the memory heap snapshot while backup is running or afterwards?

Best regards

Hello @MathieuRA , @florent

Moving the discussion to this thread as it seems to be tackling the same issue: https://xcp-ng.org/forum/topic/11892/xoa-memory-usage/28?_=1776334896974

I reported my findings so far over there

Best regards

Hello @bastien-nollet @florent

moving the discussion from this thread (link), to this one as it seems to be tackling the same issue.

This is the RAM usage of my Xen Orchestra instance before applying "fix(rest-api): fix memory-leak on subscription"

Between 04/07 and 04/10 the system was not rebooted. The RAM usage increased step by step as shown in the screenshot caused by backup runs. Each backup run increased RAM usage and it did not go back to where it was before the backup run.

This is the RAM usage after applying said fix via branch "mra-fix-rest-memory-leak".

Between 04/14 and 04/16 the VM was not restarted.
Backup jobs are scheduled to run every day.
RAM usage still increased a bit by each backup run but it seems to be less.
Looks like an improvement (which is great!) but maybe the issue is not fully fixed yet.

I can not fully tell as I need to keep monitoring this a bit more.

I will keep having an eye on this and report back.

Thanks for working on this and best regards.

I updated to branch "mra-fix-rest-memory-leak".
I will look at backup job results tomorrow and report back.

@MathieuRA Sounds promising! Once my backup job finished I will update my XO instance and start testing. Thanks for the heads up and for working on this.
Best regards

@simonp Hello! Sorry for the delay. I had some time to test the branch that you mentioned.

I can say that the backup job took way less time to finish after you applied those fixes.
In my case it came down from 2 days to 10 hours.

I wanted to compare all of the backup runs of the last weeks and give you some real data but unfortunately my Xen Orchestra seems to not show the backup runs that took place before updating to your branch in backup history.

I still feel like backups take a little bit longer compared to before the merge-refactor but it is only a small difference.

Anyways. It is a big improvement and the jobs finish in reasonable time again. Thank you!

@simonp I'll have to wait for the currently running backup job to finish.
I will update my XO instance afterwards and re-test.
I can probably give you feedback in this regard tomorrow or the day after at latest.
Thanks and best regards

@simonp Awesome. Thanks for working on this.

@Pilow I noticed the same thing.
After upgrading my XO instance my backups jobs started to take waaaay longer to finish.
I also noticed that the clean-vm / merging seems to slow down the backup jobs quite heavily.

@simonp
Maybe it would be a good idea to test these kind of things before releasing. Maybe by adding time thresholds to the unit tests as in "if step a or b takes longer than x seconds it is considered failed"?

@Andrew Yeah but XOA is still using it hence my interest in aligning my XO-CE instance with XOA as close as possible.

@Pilow Yeah maybe but I hoped that XOA is somewhat aligned with XO from sources users. At least in a way that XO-CE users have the option to use the same Node version that Vates declares as most stable to use.
With the introduction of disk-cli this is not the case anymore.

//EDIT: at least when using the common scripts or docker containers maintained by community for running XO-CE. Those scripts / Dockerfiles could be edited to exclude disk-cli. Then it would still be possible to run XO on Node 20 I guess.

@Pilow I was also suprised to see that disk-cli depends on node 22 or later.
Before disk-cli got introduced I was able to build XO on node 20 just fine.
Had to bump my node version on XO-CE in order to be able to keep building XO.

I was also suprised to see that XOA still uses node 20 even though XO from sources can not be built anymore using that particular node version.

Hello Vates and XCP-ng community,

I have updated my Xen Orchestra from sources VM and want to report that after updating my mirror backup job does not work anymore. I updated to commit 875ceb7cab9735efc29c345603a4280cb6072fbc (feat: release XO 6.3.1).

Prior to updating XO it has been working fine for weeks. I did not change anything about it's config.

I get "Error: Cannot read properties of undefined (reading 'id')" for each any every VM that the mirror job tries to mirror.

It is a "mirror incremental" backup job.

Is this a known issue?
Did anyone else encounter this?

If a XO developer needs logs, I can provide them in a private message as I am not sure if they contain sensitive information in this case.

My XO VM is running on Debian 13 and Node 22 by the way.

Thanks and best regards

@Pilow I am using Xen Orchestra from sources. I think the commits that improved RAM / heap usage are not released for XOA. But I suspect a XOA release coming soon as they already pushed "technical release" commit which is mostly an indicator that a XOA release will follow soon.

I can confirm that recent changes in Xen Orchestra improved the situation a lot.
I am still monitoring and will keep testing before coming to a conclusion but so far the backups did not crash due to Node heap size being exhausted .

I can confirm that when using Citrix/Xenserver guest utilities version 8.4 (https://github.com/xenserver/xe-guest-utilities/releases/tag/v8.4.0) memory ballooning / DMC is working fine.

After live migration the RAM of the linux guest is expanded to dynamic_max again.

So this issue was in fact caused by Rust based xen-guest-agent.

For now I'll keep using Citrix/Xenserver guest utilities on my Linux guests until the feature is implemented in Vates rust-based guest utilities.

Best regards

@planedrop Pretty sure the official and recommended way is to use the V2V (VMWare to Vates) tool that is baked into Xen Orchestra.

I used that tool back when I migrated from our old VMWare Cluster to the new XCP-ng environment. It worked without any issues.

Best approach IMO would be to disconnect the unneeded virtual disks on the source VM on VMWare side and then use V2V tool to migrate over the VM.
I did it this way for a VM with disk bigger than 2TB aswell. After migrating the VM I created an LVM volume on the XCP-ng VM with the required size and rsync-ed the remaining data from VMDK > 2TB to target XCP-ng LVM volume.

Documentation: https://docs.xen-orchestra.com/v2v-migration-guide

@dinhngtu @teddyastie Do you have a time line / rough estimated release date regarding Rust-based xen-guest-agent version 1.0?
Currently evaluating if I need to switch my Linux guests back to Citrix guest tools in the meantime.

Looking forward to a reply.

Thank you in advance.