@jr-m4 you can export the heap memory of the nodeJS process by doing kill -SIGUSER2 <xoserverpid>

onte that this will increase a lot the memory consumed by the xo process even when its done exporting the memory . This will help us know what xo is doing at the moment

@acebmxer we are far more explicit on the real usage of NBD, paing the way to mark a failed NBD connexion as an error in a few month

also working on a diagnostic tool to be more efficient on the checks

thank you all for your time

@probain the first checkc to do are network one :

is there a default backup network on he pool ? if yes the xo will only look at this network

check if :

• the xo can access network(s) with nbd enabled
• the host have an ip on those networks
• are you on ipv4 or v6 ?
• xo can access the ip of the host on the nbd newtork

fix is here https://github.com/vatesfr/xen-orchestra/pull/10346 , it works on @flakpyro proxies

the root cause is a timing condition between compression buffering data silently and timeout watching any stall connection

@acebmxer can you give me the support tunnel number by chat ? patch is ready

@acebmxer Hi am testing an hypothesis where removing an old library introduced a implicit compression and buffering between the proxy and the how.

Can I provide a branch to test, or can I deploy a fix through a support tunnel ?

@JB this really looks like a network issue

are the host/xoa remotes ?

@JB this looks like an issue while reading the export part. Did you configure a default backup network in the pool ?
is there any firewall/sdn controller that can filter out some reading ?

note that we have removed several point that where hiding some errors this month.

@Andrew @probain
XO was hiding some error message and fell back silently ( or fail later) , this onth updates gives us more information

are you using NBD ( it is mandatory for qcow2 disks) ?
is NBD enable on at least one network per pool ?
can XO access this network ?
is there a default backup network on the pool ? with NBD ? accessible by XO ?

any more logs on journalctl ( I am expecting some nbdinfos )

both fixes are now on master AND released as a patch

thank you @pedro_udifar

@pedro_udifar that is strange that we have exactly the same error, even though we should have added 28 bytes

I pushed a little debug that will log additional lines in journalctl

'will output stream ', 'stream is encrypted' and 'stream is updated '

Can you update your branch, rebuild, rerun the transfer and post the log ( with the attached values ) ?

regards

@pedro_udifar object lock works quite well with xo
it is automatically detected if the s3 user has enough privilege to see if object lock is enabled

the main point is that it won't create "cache.json.gz" files since theses files must be updated, so the backup listing is a little slower

note that XO retention should be at least 1 more than object lock . At worst you will have error , saying that XO coudn't delete an old backup

The fix is on a separated branch : fix_s3_nonencrypted_to_encrypted_xva ( PR : https://github.com/vatesfr/xen-orchestra/pull/10061 )

@pedro_udifar we have found a clue , if the destination is encrypted and the source is not

can you test this branch ? https://github.com/vatesfr/xen-orchestra/pull/10061
if you are using a xoa, we can deploy the fix if you open a support tunnel

now that I am in front of the source code , it is already getting the real size, so it should compute the right size

are the source or destination encrypted ?

@poddingue said:

Thanks for the detailed write-up, Pedro.
I'm not a backup expert, far from it. but you might be right that https://github.com/vatesfr/xen-orchestra/pull/9396 only fixed the size estimation for direct full backups, not the mirror path.
Your error is the same as the maximum size allowed one, just twelve bytes over 209715200000. Before this turns into a GitHub issue, it would help to know whether it reproduces on a fresh mirror job and whether bumping minPartSize actually cleared it, so we can be sure it's the mirror code and not the B2 remote settings.
The object storage docs list Backblaze B2 as supported (https://docs.xen-orchestra.com/xo5/object-storage-support#supported-storage-providers) but don't say much about part-size tuning.
If it's awkward to test in isolation, a mention to @Team-XO-Backend is probably the quickest route, since they own the backup job code.
The XVA checksum warnings in your second screenshot look like a separate non-blocking clean VM directory step rather than the cause.
I hope that points somewhere useful!

nice catch
and the mirror path is easier to fix . Thaks @pedro_udifar and @poddingue we will fix it asap

will it be possible to test a potential fix @pedro_udifar , since it's often specific to a provider ( and need huge VM to work )?

the new code is now in master

@Andrew working on it with @julienxovates

at least including them by default , without following them, in the archive seems doable . The change on the XO5 UI may be trickier, and we are not far to rewrite it for XO6

@Andrew good news : the lvm id/name collision is the root cause of a lot of issues of the the file restore.

I think restoring the symbolic links will be hard to secure ( a symbolic link can point anywhere in the os and the disk are mounted in the xoa as root) and quite tricky to ensure we are not bombing ourself

I think we can add them to the zip/tar as (brokenà symbolic link, but we need to look into the impact on the client side file browser

@Andrew thank

I pushed some fixes , can you retry the VM . With this config in the toml ? if it fails if will gives us more logs in journalctl

[logs]
# Display all logs matching this filter, regardless of their level
filter = 'xo:backups:RemoteAdapter'

@ravenet not directly, it's not mountable into linux

But if ( and this is a big if) we are able to make FLR more resilent, I have some next step that would expose the disks so you can mount it directly in your preferred OS or XCP , probably through webdav