Backup size larger than it should be

kamil-v4

Hello,

I'm currently facing an issue while backing up a VM to a separate host using NFS. The problem lies in the backup size, which is showing as 190 GB, even though the VM contains only almost 20 GB of data.

Here are some additional details:

Initially, the VM had 190 GB of data, but I moved it elsewhere, reducing the data size to 20 GB.
After reducing the data size, I deleted the previous backup, ran the backup job again, and ended up with a 190 GB backup.
The VM is stored on a thin-provisioned SR, and its VDI size is 200 GB.
The root partition of the VM uses the XFS filesystem with LVM.
The backup job is configured as a Delta Backup.
The target server utilizes the EXT4 filesystem with LVM.
Interestingly, I have another VM with a similar setup (same filesystem and backup job), but it behaves as expected. This VM has a VDI size of 50 GB, contains 8 GB of data, and the backup size is 8 GB.

I'm puzzled as to why this is happening and would appreciate any insights.

Thanks in advance.

tjkreidl

@kamil-v4 Perhaps the space has not yet been recovered? How full is that SR? It should be under around 90% full for space recovery to be able to take place. Also, have you tried manually running an sr-scan which should trigger a coalesce process?

kamil-v4

@tjkreidl Thank you for your reply.
The local SR is only using 21% of the storage, so it has plenty of room to breathe. I have tried running sr-scan from the CLI and by using "rescan all disks", but with no luck.
What I have noticed though is that there is a base copy of the VM that appears to consume space on the local SR:

I have found the following article: https://xcp-ng.org/forum/topic/6719/deleting-snapshots-and-base-copies/11 but when I go to the Advanced tab of the local SR, it doesn't show any VDIs to be coalesced.
Reclaim free space option doesn't work for me as I'm getting the following error (possibly because the SR is used with HDDs not SSDs):

Trim on [8b69e781-0ece-595b-aa5b-832c79bfbec8] not supported

tjkreidl

@kamil-v4 If you have another SR with space available, one option would be to move the VM to it and then back again, which should in principle reclaim the space. On the SR that you move thee VM to, it should show up coalesced. Sure you don't have any hidden snapshots or such that might be contributing to the bloated space?

kamil-v4

@tjkreidl I can't see any hidden snapshots, only a single snapshot created by the delta backup.
I will try moving the VM to another host, thanks for the advice.

tjkreidl

@kamil-v4 Not to another host, to another SR! The VM has to be moved to a different storage device. The host is irrelevant if within the same pool.

kamil-v4

@tjkreidl I had no other storage repository connected to that host, so I had to attempt to move it to another host, which has its own storage repository. Unfortunately, the VM/Backup size is still nearly 190GB.

florent

@kamil-v4 hi kami,

the backup size is the size of the data block that have been used at least once, even if you remove the data, the blocks stays allocated, thus are saved when doing a key backup.

I am not sure of what would be the best strategy to deflate this VHD on XCP-ng side, as I am working more on the backup side of the XCP-ng.XO stack

tjkreidl

@florent I am perplexed because thin provisioning should allow for coalescing. I would still try moving the VM to another SR and back again would again be my suggestion.

florent

@tjkreidl thin mode doesn't allocate space that is not needed. But once allocated, they stays there.
I am curious if migration from on Sr to another works. If not I will look into this

On the backup side, you can use a block based repository, each block are individually compressed. It should reduce a lot , especially if the blocks are full of zeros

olivierlambert

Yes, don't mix coalesce and trim. SMAPIv1 doesn't support trim, so only regenerating the VHD (eg by moving it or copy it) will get rid of those empty blocks.

tjkreidl

@olivierlambert Ah, for SSD drives, yes, trimming is a whole separate thing.

olivierlambert

I'm talking about VHD trimming, which is unrelated on how it's stored behind Obviously, to do this, the Dom0 needs to know from the VM which blocks are unused, which is given by trim commands.

Surprisingly, trim is available in the blkif protocol (used to communicate between the VM and the Dom0 & Xen). But it's not used in SMAPIv1, probably because of shared LVM on HBA or iSCSI (suspecting it cause some corruptions). It should work locally but was never implemented.

kamil-v4

@olivierlambert @florent thank you for the suggestions.
So far I have attempted to do the following:

Remote backup to NFS
Remote backup to NFS target which is connected to another storage server using iSCSI
Move the VM from one SR to another (on a completely different pool)
Copy the VM to another pool

Unfortunately, my VM size is not reducing.

@florent said in Backup size larger than it should be:

thin mode doesn't allocate space that is not needed. But once allocated, they stays there.

This looks like an answer to me.
In the worst-case scenario, I will have to recreate the VM and copy the data onto it