Offline
@afk the agent is as dumb as possible
also if you encrypt the backup, the agent will need to decrypt the metadata to detect the chains, thus having access to the encryption key, which need getting the encryption key out of XO and transferred to the immutability agent
I think it will be easier to provide more feedback on the immutabiltiy backup, XO has access to the chain , and / or alert when something seems to be strange
@peo do you have anything in the xo logs ? can you post the json of a failed backup job ?
@peo in object storage world ( S3 compatible) , there is not really the notion of "directory", it's more a convention that / is a logical level in the file key, but rclone has errors on "Directory not empty". It let me think that rclone do things a little differently than aws s3.
We try to be as compatible as possible with S3 implementation, but each object storage has its own quirk. Thus we only supports the most used setups for now.
@SudoOracle could you post the full json log of the backup ?
(you can get it by clicking on the download button on top of a failed execution job)
if possible, one per type of failed backup job
@manilx are there multiple job running on the same VM ?
this happens when 2 backup jobs are working on the same VM , or a backup and a background merge worker .
If it's the first case, you can use a job sequence to ensure the backup run one after another
If it's the latter, you have a check box to ensure the merge is done directly inside the job (bottom of the advanced settings of the backup job ), instead of doing this in a background job. It will slow down a little the job, but ensure it's completely done.
@abudef
incremental backups on block storage are compressed with brotli in fastest mode. But this is enough to have a massive gain , since we compress all the zeroes.
incremental backup ( *.vhd files ) on non block storage aren't compressed
Full backup (as in xva file ) aren't compressed by xo, but can be compressed by xcp ng ( using gzip or zstd )
for now, this is not configurable, we are waiting for the future remote form to handle this.
@afk the newer VMFS put more lock on the files, locking the full chain of snapshot and base disks instead of locking only the active disk.
Even VMFS5 sometimes lock the full chain.
The more open storage is a NFS, that can additionally be accessed directly by XO, not going through the soap api , giving anice performance boost and not hammering the esxi too much ( only for the metadata )
We don't intend to migrate the VM running in vmware from XO automatically. A user could script it though, by using a combination of the vmware apis and xo-cli.
We never succeed in getting the disks though NBD, but it should be possible ( https://github.com/vexxhost/migratekit/blob/main/internal/vmware_nbdkit/vmware_nbdkit.go#L91 ) and ( https://vdc-download.vmware.com/vmwb-repository/dcr-public/8ed923df-bad4-49b3-b677-45bca5326e85/d2d90bb6-d1b3-4266-8ce5-443680187a9a/vim.vm.device.VirtualDevice.BackingInfo.html )
What we are mostly missing here, is internal knowledge on the Vmware side : how to get the NBD server address,how to authenticate and how to get the exportname of a disk through the soap API .
Since we already have the knowledge of using NBD to read massive volume of data as long as we can connect.
they are created to ensure the samba mount is not removed by the os or another thread on XO
I would not delete them , especially since they are 0bytes files
@rtjdamen said in Feedback on immutability:
@florent so this does mean it will never work when a forever incremental is used?
you can't have a immutable forever backup without having a infinite length, and an infinite
It may be possible only if we release the constraints.
The immutable script could release the immutability , merge the disks, but that means : the immutability will be lifted from time to time, and the responsibilities of the immutability script will be greater, and we'll need a way to track the vhd to merge and transmit the information to the immutability script
@rtjdamen for the immutability to be useful, the full chain must be immutable and must never be out of immutability
the merge process can't lift/ put back the immutability , and increasing synchronization between process will extend the attack surface.
immutability duration must be longer than or equal to 2 time the full backup interval -1
the retention must be strictly longer than the immutability .
for example, if you have a full backup interval of 7 a retention of 14 and immutability duration of 13 , key backup are K, delta are D. Immutable backup are in bold . unprotected chain are striked
KDDDDDDKDDDDDD worst case, only one full chain protected
KDDDDDKDDDDDDK
KDDDDKDDDDDDKD
KDDDKDDDDDDKDD
KDDKDDDDDDKDDD
KDKDDDDDDKDDDD
KKDDDDDDKDDDDD best case almost 2 full chain protected
@rtjdamen great work
An additional note : to ensure that an incremental backup is really protected during n days, you must have
@flakpyro for now there is no tag selector , but you can now select the VM list to be replicated
The rules is "any backup that check a condition to be kept is kept", you can combine LTR retention and retention by number
so you can keep the last 3 backups , and also have a GFS strategy. It should remove most of the use case of backups with multiple schedules.
first point : any deleted backup is not recuperable, and this is a new feature, so test the feature progressively, in case we missed anything, especially on critical backups as we can iterate on the feature and improve it
we'll write more documentaion shortly
We have some interesting thing in the works, but I think a chain of 84 snapshots for the CR/DR is quite long
you can mitigate the risk of chain corruption by setting the full backup interval, this will transfer a full from time to time
@flakpyro said in CBT: the thread to centralize your feedback:
This is a completely different 5 host pool backed by a Pure storage array with SRs mounted via NFSv3, migrating a VM between hosts results in the same issue.
Before migration: [01:41 xcpng-prd-03 b04d9910-8671-750f-050e-8b55c64fbede]# cbt-util get -c -n 83035854-b5a9-4f7e-869f-abe43ddc658d.cbtlog e28065ff-342f-4eae-a910-b91842dd39ca After migration [01:41 xcpng-prd-03 b04d9910-8671-750f-050e-8b55c64fbede]# cbt-util get -c -n 83035854-b5a9-4f7e-869f-abe43ddc658d.cbtlog 00000000-0000-0000-0000-000000000000I dont think i have anything "custom" running that would be causing this so no idea why this is happening but its happening on multiple pools for us.
This is a very interesting clue, and we will investigate it with damien
there is a lot of edges case that can happens ( a lying network/drive/... )
and most of the time , xcp/xapi are self healing, but sometimes XO have to do a little work to cleanup. The CBT should be reset correctly after storage migration.
We'll add the async call to enable/ disable CBT since it could lead to bogus state, and maybe a more in depth cleaning of cbt after a "vdi not related error "
@rtjdamen we found a clue with @Bastien-Nollet : there was a race condition between the timeframe allowed to enable CBT and the snapshot, leading to a snapshot taken before CBT , thus failing to compute correctly the list of changed block at the next backup
The fix is deployed, and we'll see this night. If everything goes well, this night will be a full, but the disks will keep CBT enabled. And the next night, we'll have delta
if everything is ok, it will be released in a second patch ( 5.100.2 )
this branch ( already deployed on @rtjdamen systems) add a better handing of host that took too much time to compute the changed block list :
https://github.com/vatesfr/xen-orchestra/pull/8120
it will be release in patch this week
I am still investigating an error that still occurs occasionally : XapiError: SR_BACKEND_FAILURE_460(, Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated], )
@Tristis-Oris do you have preferNbd in your config.toml ? it was the first way to enable NBD but is not recommended anymore
@Tristis-Oris I will try to reproduce it here ASAP and will keep you informed
