Compressed backups

olivierlambert

FYI, you should create a remote with "Store backup as multiple data blocks instead of a whole VHD file" to enjoy better backup perf with compression (IIRC, we use brotli). Every info we can compress will be compressed.

frank-s

@olivierlambert Thanks. I'll do that.

CJ

@olivierlambert Are you referring to the compression ratio when you mention performance? I turned it on for my remote and it tanked my transfer speeds compared to the standard sequential backup.

Since I'm using zfs for my remote, is there a way to turn off compression for the delta backups?

olivierlambert

Question for @florent

florent

@CJ said in Compressed backups:

@olivierlambert Are you referring to the compression ratio when you mention performance? I turned it on for my remote and it tanked my transfer speeds compared to the standard sequential backup.

Since I'm using zfs for my remote, is there a way to turn off compression for the delta backups?

you can turn compression off by setting vhd compression to none instead of brotly in your configuration file ,

[backups]
# Mode to use for newly created backup directories
vhdDirectoryCompression = 'brotli

writing vhd blocks is often slower than sequential, and compression should not have a lot of impact , but the merging is faster. So the best solution depends on the limiting factor on your infrastructure
(also this storage mode have been made for s3 which do not like modifying the files )

abudef

@olivierlambert said in Compressed backups:

Compression is enabled by default for incremental backups.

Hi, if I configure Delta Backup and run a backup for the first time, a full backup of the affected virtual machines is performed. This initial full backup is also compressed?

olivierlambert

IIRC, everything is compressed using Brotli by default, including the full.

abudef

If I understand correctly, for compression to work for Incremental (Delta) backup, remote storage must have the option "Store backup as multiple data blocks instead of a whole VHD file" enabled, otherwise vhd files are stored without compression. Is that right?

olivierlambert

I think we compress everything on flight, but I'm not sure, @florent could tell

abudef

At least the first backup, i.e. the full backup (and maybe every key full backup milestone later) does not look like it is compressed. If the "Store backup as multiple data blocks instead of a whole VHD file" option is off, a .vhd of 116 GiB is stored, if it is on, the whole data block structure is only 67 GiB.

abudef

In addition, I have verified that if the option "Store backup as multiple data blocks instead of a whole VHD file" is enabled, the restore from the data blocks structure takes about 2-3 longer than from the .vhd if this option is disabled (I tested the health check restore).

abudef

(The idea behind is to have both a compressed delta backup (the backup repository itself is backed up to another location via vpn, so the size of the backup matters) and the fastest possible restore option.)

frank-s

@abudef I just had a backup failure. This triggered a full (key) backup on the next run which transferred 165GiB. When I rdpd into the vm I could see that the vm contained 345GiB of data including the system drive. I conclude, therefore that compression is working for full key backups at a ration of around 2:1.
I hope this helps.

abudef

@frank-s Do you have the "data block" option enabled for remote or not?

frank-s

@abudef Yes, data blocks is enabled on the remote.

abudef

@frank-s This confirms my observations and tests that for compression to be active, data blocks must be enabled for remote.

abudef

@florent Can you please provide some summary on backup setup versus compression versus remotes configuration?

abudef

Hello @florent may I remind you to make this question clear, when and under what circumstances is compression used and when is it not? Thanks a lot.

florent

@abudef
incremental backups on block storage are compressed with brotli in fastest mode. But this is enough to have a massive gain , since we compress all the zeroes.

incremental backup ( *.vhd files ) on non block storage aren't compressed

Full backup (as in xva file ) aren't compressed by xo, but can be compressed by xcp ng ( using gzip or zstd )

for now, this is not configurable, we are waiting for the future remote form to handle this.

abudef

@florent Clear now, thanks!