@planedrop said in Backup / Migration Performance: @KPS Regarding the 2TiB limitation, it'll definitely be nice when we have SMAPIv3 so we can go over this, but it's worth noting that IMO no VMs should be larger than this anyway. Generally speaking if you need that kind of space it'd be better to just use a NAS/iSCSI setup. Something like TrueNAS can delivery that at high speed, and then handle it's own backups and replication of it. I know most probably already know this, and all environments are different (I manage one that requires a 7TiB local disk, at least for the time being, plan is to migrate it to a NAS once the software vendor supports it), but it's worth noting anytime I see the 2TiB limit come up, ideally it should be architected around so the VMs are nimble. I do something similar w/ a pretty massive SMB share and TrueNAS can back this up at whatever speed the WAN can handle, in my case 2 gigabits and it'll maintain that 2 gigabit upload for 8+ hours without slowing down. (and I'm confident even 10 gigabit would be possible with this box) We have 1 exception and that is for Windows file servers which is backing our DFS. Except from those we dont allow VM's larger than 1Tb and if they're that big we do not back them up because it usually breaks and cause all kinds of problems.

@frank-s Think of it like this... You take the original backup with snapshot on October 1st. You then have a differential backup on October 8th. That differential only writes the changes that make the state of the server on October 8th different (hence the term differential) from the state of the server as it was on the full snapshot on October 1st. Now you delete the snapshot of the full system state taken on October 1st, leaving only the October 8th differential. It's now October 15th and you want to restore something from the October 8th differential and you figure you'll just make a new full snapshot from October 15th and then try to apply the October 8th differential. However, this won't work because the differential from October 8th will be missing key reference points it will be looking for in order to overlay its changes onto the snapshot. What else on that server changed from October 1st to October 15th that isn't contained in the differential? What changes occurred from October 8th to October 15th that are newer than what is on the differential? You could, potentially, brick your entire system if the differential started overwriting changes it sees from its system state that are actually newer than the data it contains. Differentials are entirely dependent on the full snapshot on which they are based. Any new snapshot taken after their creation will be totally foreign to the differential... those differentials will be looking for very specific system states that existed at the time the original snapshot was created, and those will be completely different from the system state as it will appear on a new snapshot created after the differential.

@olivierlambert I've finally had time to sit down and do some more troubleshooting. It seems that the issue is somehow tied to the backup-jobs themselves. Deploying XOA and subsequently importing XO-config from my XOCE instance. I continued to see several issues between both instances. These issues pretty much went away, when I re-did the backup-jobs from scratch. And are now much more in-line with what I'm excepting to see. This was done on both XOA Stable, Latest and XOCE I am no longer able to repoduce this at all. So my guess is that a bug of some sort got introduced with this being a constantly updated from source instance. Marking as solved. As I don't believe there is much more to do at this time.

@stephane-m-dev In the schedule You can select force full backup

So the tools are visible but XO can't find them. Thanks for the feedback, we'll dig.

Why you wouldn't be able to deploy another XOA? It's meant to be re-deployed anytime you lose it, because even a fresh one can access a backup repo and restore everything.

@ph7 I am using an Incremental Replication backup (aka Continuous Replication) and I understand this is the same as Delta however Delta uses remotes whereas I.R. uses host storage.

That's indeed a solution. We could also maybe "revert" the condition for people with like 500 VMs to healthcheck except one.

@julien-f Tried again with today's release Xen Orchestra, commit 3d054 Master, commit 3d054 Same error again https://paste.vates.tech/?4941198dd850ed6d#ArUk6wqkwE2c3CQRbjwAXRERJd2Y4HWF5oq9ysMdkbBp Mirror incremental backup is broken here

XOA trial is enough to test it. Just ask another trial.

@florent Did you find anything? I just retested the job on latest XO build and have the same error.

I understand your requirements, I'm trying to pinpoint the issue. For now, everything converge on dom0 load when having many zstd processes at once.

@olivierlambert thanks, just to be sure.

@Nephilim Nothing shown here would indicate a problem AFAICS. P.S. Added proper markup to your post

Thanks @olivierlambert!

I don't know if it's still what he does

@McHenry Looks like you want disaster recovery option. It creates a ready-to-use VM on a separate XCP-ng server. If your main server fails you can start the vm directly off the second server. In any case, backups can be restored with XO to any server and storage available in XCP-ng.

Yes.

@Danp I don't really need something complex. I see the full & delta vhd files in each VM's folder but trying to mount the delta vhd with guestmount throws an error. There has to be a way to fold the delta vhds into the main vhd for the most up to date vhd..

@olivierlambert said in XO (from source): backup error (to Wasabi): If you need another trial, no problem. To me, the problem is not in XOA (the broken commit was done after the monthly release) and fixed today, so it means it would be never shipped in XOA. Then I am fine, building XO from Sources provides me with the ability to backup my VMs, which was the only XOA "Feature" I was using... Thanks anyway!!!