Compressed backups

DustinB

Hi, Does delta (incremental) backup compress backups on the backup repository? Is there a way to enable this to reduce the amount of data to transfer?
Thanks.

Delta, by definition is compressed to changes only, there has been some discussion on the forum about how effective this is when comparing Linux and Windows guests and how much the delta's are different.

Additionally, you can enable compression within the advanced settings under a given backup, see below for an example.

frank-s

@DustinB
I couldn't see the Compression field under "delta backups" but now I see it for full backups. I guess then compression is not available for delta backup because it would screw up file merge when retention has been reached. Is this right?

olivierlambert

Compression is enabled by default for incremental backups.

frank-s

@olivierlambert Thank you. Does this apply to the key backup as well as the increrments?

olivierlambert

FYI, you should create a remote with "Store backup as multiple data blocks instead of a whole VHD file" to enjoy better backup perf with compression (IIRC, we use brotli). Every info we can compress will be compressed.

frank-s

@olivierlambert Thanks. I'll do that.

CJ

@olivierlambert Are you referring to the compression ratio when you mention performance? I turned it on for my remote and it tanked my transfer speeds compared to the standard sequential backup.

Since I'm using zfs for my remote, is there a way to turn off compression for the delta backups?

olivierlambert

Question for @florent

florent

@CJ said in Compressed backups:

@olivierlambert Are you referring to the compression ratio when you mention performance? I turned it on for my remote and it tanked my transfer speeds compared to the standard sequential backup.

Since I'm using zfs for my remote, is there a way to turn off compression for the delta backups?

you can turn compression off by setting vhd compression to none instead of brotly in your configuration file ,

[backups]
# Mode to use for newly created backup directories
vhdDirectoryCompression = 'brotli

writing vhd blocks is often slower than sequential, and compression should not have a lot of impact , but the merging is faster. So the best solution depends on the limiting factor on your infrastructure
(also this storage mode have been made for s3 which do not like modifying the files )

abudef

@olivierlambert said in Compressed backups:

Compression is enabled by default for incremental backups.

Hi, if I configure Delta Backup and run a backup for the first time, a full backup of the affected virtual machines is performed. This initial full backup is also compressed?

olivierlambert

IIRC, everything is compressed using Brotli by default, including the full.

abudef

If I understand correctly, for compression to work for Incremental (Delta) backup, remote storage must have the option "Store backup as multiple data blocks instead of a whole VHD file" enabled, otherwise vhd files are stored without compression. Is that right?

olivierlambert

I think we compress everything on flight, but I'm not sure, @florent could tell

abudef

At least the first backup, i.e. the full backup (and maybe every key full backup milestone later) does not look like it is compressed. If the "Store backup as multiple data blocks instead of a whole VHD file" option is off, a .vhd of 116 GiB is stored, if it is on, the whole data block structure is only 67 GiB.

abudef

In addition, I have verified that if the option "Store backup as multiple data blocks instead of a whole VHD file" is enabled, the restore from the data blocks structure takes about 2-3 longer than from the .vhd if this option is disabled (I tested the health check restore).

abudef

(The idea behind is to have both a compressed delta backup (the backup repository itself is backed up to another location via vpn, so the size of the backup matters) and the fastest possible restore option.)

frank-s

@abudef I just had a backup failure. This triggered a full (key) backup on the next run which transferred 165GiB. When I rdpd into the vm I could see that the vm contained 345GiB of data including the system drive. I conclude, therefore that compression is working for full key backups at a ration of around 2:1.
I hope this helps.

abudef

@frank-s Do you have the "data block" option enabled for remote or not?

frank-s

@abudef Yes, data blocks is enabled on the remote.

abudef

@frank-s This confirms my observations and tests that for compression to be active, data blocks must be enabled for remote.