Offline
On tomorrow 's release we will:
following this we will add core xo metrics to the openmetrics ( prometheus) export to allow users and support to dig through real world data when needed
Cancelling a backup run is also on our roadmap ,as soon as we have done reworking the tasks to use xo-task on the full chain, and not an hybrid of various iteration of the tasks objects
better tracking of the backup run is also on our roadmap , this is also linked to the task changes plus are some issues with the size on the remote before being able to do this
In any case, both are on our backlog
@Pilow said in Veeam backup with XCP NG:
@olivierlambert I get you.
Same in veeam and synthetic fulls, it's only between the server/proxy and the repository.
but we get a progressing percentage while the job sits while doing the synthetic full.would be cool to "see" the progress of merging.
why ? bad habits from previous technologies used I guess.
Giving better observability is something we are working actively on , this is tightly coupled with being able to stop a task, and I hope we will be able to show the progress soon
@manilx yes, that's it
we should change the "full backup interval" to "base/complete backup interval", to clarify what is a full . And we will do it.
we are reworking the import code , along the new backup code ( https://xcp-ng.org/forum/topic/10664/our-future-backup-code-test-it ) to unify the differetn disk transformation path
For now:
@andrewreid Yes, I can't wait to share it with everybody.
@AlexD2006 thanks for signaling this, we just merged a fix, that fix it on our labs
can you test it on your side ?
https://github.com/vatesfr/xen-orchestra/pull/9202
@AlexD2006 this looks like the right issue, we will look into it immediately ( yesterday was a public holiday in France)
@Andrew then again, with such a precise report, the fix is easier
the fix should join master soon
found it , there was not one, but 2 bugs
albeit small they were putting the xo backup in a incorrect state
the fix is here, and will be merged as soon as possible and then released as a patch for xoa users
https://github.com/vatesfr/xen-orchestra/pull/8780
really thank you everyone for the information
we did a lot of ground work on the file restore front , and have plan to continue working on it.
What is testable and should work :
Note that, by design, the file restore is slow : each time the xoa want to read 4KB of data or even 1B we have to load a full backup block (2MB) decrypt, and inflate it ,extract the 4KB , ... . Expect 1/10 to 1/30 of your remote performance
But at least it should not crash anymore, with zip or tar restore. The worse we tested is a windows folder ( 26GB , 10 of thousands of files)
For now the fixes are in https://github.com/vatesfr/xen-orchestra/pull/9776 ( branch fix_flr ) and are expected to reach master during next week
@tsukraw multiple NBD connection will open multiple reading connection, but the writing one is always one stream per disk in incremental replication
with full replication, it's one stream for read and one for write
@acebmxer back to work
thank you for yor patience and help on this. I feel that it's not the same issue , with abrupt increase
W will try our best to also fix this one
@laszlobortel since we are generating the stream sequentially, we though that there wasn't any real world gain. Also, if my understanding is correct using one thread means we will have only one process( + xo + nbdkit-vddk) processes , and we were trying to limit the risk of zombie process locking the disks , which was far too common with the previous code
we'll do a quick test to ensure it does not beak anything and propose either a better default of a configuration option
note that this will be a concurrency per disk, so VM with a lot of disk will put more pressure on the source
also do you have a reverse proxy/ http proxy in front of xo ? it can block the bigger upload by default
@nasheayahu qcow2 disk import has been merged recently ( https://github.com/vatesfr/xen-orchestra/pull/9817 ) , but there is a caveat
the clusters in the disk must be in order to work . This is generally the case for a disk exported by any system, but not the case for a disk used in production that you want to import
where does this disk comes from ? if you have access to your SR from the outside, you can also put the qcow2 file directly , by renaming it as a valid uuid ( I think uuidgen can do this ) and putting it directly in the SR/uuid repository
@poddingue this is something that was hidden with the previous system ( same disk chains, but not shown as snapshot )
@julienxovates are you ok to not check the vm tagged as replication from this chech ?
the last rewrite of the stream processing ( spring 2025 ) focused on stability and memory footprint, and , on a standard cpu, it tops at around 300MB/s per backup job. Your benchmarks are very interesting, and they confirm most of it.
this limit was not really an issue since, in most case the xapi was limiting around 100MB/s per disk , but it will be more a more visible limit
Note that master have some fixes on the memory usage (not related to backups)
That's why we have started an internal workforce focused on performance, with all the teams from the kernel to the backups, including storage, network and xapi.
If I can brag a little :
i9 , nvme disk , backup to a nvme disk in passthrough, xoa and vm are on the same host, so it's quite far from real world data, but it shows where the limit is
@tsukraw I am taking the ticket and will keep you informed as soon as possible
