@andrewreid Yes, I can't wait to share it with everybody.

@ronivay that's a great information. I can reproduce the comportment with xenserver 7.1 CU 2
Thank you

@cg We're still in the early phase, but it seems possible to connect the tape loader to a host, and to let the host expose it. Passthrough will means that Xo's host is directly connected, which is not always the case

Backup to tape is quite different than backuping on disk, S3 or even glacier, I am confident that it will be possible, and we will communicate as soon as we have a working copy.

@Forza
We're currently creating a lab that will allow us to make benchmark more easily. The latest version of the merging operation only do a block renaming + deletion of unused block (blocks are 2MB , not configurable for now), no data copy, thus no data transfer between the remote and XO.
It is faster on performant backend, but we still have some fine tuning to do to ensure we don't overload slower backend. In the meantime we put some very conservatives settings.

backup tiering and backup immutability are also in the work. I don't have a precise ETA, but it's in a few weeks to a few month at most.

theses features will pave the way for backup to tape

@cg hi, I am working on the backups , and I can add some precisions

• compression is already on (first on S3, but now for all backups remotes) with the new vhd storage system, using brotli compression by default ( gzip possible , but it's slower and less efficient) . Compression level can be configured
• deduplicaton is in our backlog
• backups can be processed in parallel, and with the new vhd storage vhd blocks are uploaded in parallel ( default 8, can be configured)
• delta backup merging is done in parallel ( 2 by default, can be configured)
• we 're working on using network block device server to be able to read the data in parallel from the host, with the goal of speeding up backups and offloading DOM0

@lukask I am in the process of checking for the unused vhd ( it's generally not a problem ) and @julien-f is working on the cache problem

Thank you

@lawrencesystems We added the healthcheck for full backup (here : https://github.com/vatesfr/xen-orchestra/pull/6401 )
Also we removed our js smb implementation ( https://github.com/vatesfr/xen-orchestra/commit/b50e95802c6a876d4e6ffd0c62c209a187b1bfb2 )

to try to circumvent the strange samba comportment

fbeauchamp created this issue in vatesfr/xen-orchestra

closed fix(backups): add healthcheck to full backup #6401

0 julien-f committed to vatesfr/xen-orchestra

feat(fs): remove JS based SMB handler

It's not well tested nor maintained.

Hello,

We could reproduce it with a job with only continuous replication. the PR with the fix is here : https://github.com/vatesfr/xen-orchestra/pull/6437/files

Can you confirm it fixes the problem ?

Regards
Florent

fbeauchamp created this issue in vatesfr/xen-orchestra

closed fix(xo-server): fix running replication job #6437

@Tristis-Oris hi,I see two problem :

• an older merge failed for all theses VM , and some block ae missing ( 2/49 for the fisrt one, 2/52 for the second one) . These backup are irrecuperable and should be deleted

• there is a desynchro between the cached metadata and the file system, you can delete them by running
  find /mnt/mih-5300-2/xo-vm-backups/ -name "cache.json.gz" -type f -delete

@julien-f I didn't find a debug option, but we can put a custom logger. I did quick test with a direct console/log.info and the volume of data logged is huge. I'm trying to make a more reasonable logger

@lukask I am in the process of checking for the unused vhd ( it's generally not a problem ) and @julien-f is working on the cache problem

Thank you

hi @lukask ,

When a backup is missing, can you go into your NAS and delete the cache.json.gz files ? They will be regenerated when checking the restore tab again

Regards

Flo

@Danp yes that is an automatic healthcheck process that try to ensure that the vhd chain are healthy

before this summer it was done without information in the UX, we choose to expose this to help detecting probelm before they're out of control

@nuentes in your xo-serve/dist folder, can you try

./db-cli.mjs repl
job.get('886244d4-6ec3-4d76-90b1-aafb7ce2eaef')
job.remove('886244d4-6ec3-4d76-90b1-aafb7ce2eaef`')

job.get() #here the problematic job shoulw not appears

then restart xo-server

Also, can you check if there is any linked schedule
./db-cli.mjs ls schedule

@Tackyone after rereading your posts I added more detail on the legacy mode

is it more clear ?

@Tackyone in block mode : merging in block mode is a multistep rename , where we try to rename as little block as possible while limiting the time with a broken vhd
we move the 2MB blocks of the newer VHD to the older one (overwriting some blocks in the process), then rename the alias ( a sort symbolic link ) to have the link that pointed to the newer VHD so that it point to the older vhd folder. Older alias and the dataless folder of the newer vhd are purged

in file mode : we modify the content of the older vhd file with 2MB parts from the newer vhd file. Then we rename. Rsync should be able to transfer only part of the file, but depending on the number of changes and the "chunk" sized use by rsync, it can transfer most of the file. We probably could implement a rsync friendly mode, where new and modified parts are always written at the end of the file, but it will lead to growing vhd and create its own set of problem.

One of our next step will be to implement a transfert/ replication inside XO, that will benefit from knowing the file structures and planification.

@tanjix nope it was with the changes we made to plug encryption on the data pipeline
but it's not the cause of the backup faling

do you have some logs on the xenserver side ?

can you monitor available memory on XO during a backup ?

@tanjix I found the cause of the error messages ENOTDIR (a side effect of encryption )

but it won't explain why the backup fails ( worker exited with code null ) I'm still looking for a cause

@tamas said in libfuse.so: error adding symbols: File in wrong format:

fast-xml-parser

fast-xml-parser is an older deps (last change is at least a few months old). Do you know which is the last working version on your system ?

hi @Can-eh-dian11
Is there a way to give me access to the VHD ? given its size, I would prefer a remote access to the server

regards