crosstestet with XOA
All working fine as expected.
So it seems that first Host has some Problems and is not providing any useful data when it comes to exporting the first delta snapshot.
Unfortunately i have to change to customer support right now and have to stop my testings for today.
I will keep going tomorrow.
Thx for your support so far.
ok, so i tested with XOA.
fully new (empty) NFS-Export mounted as remote in XOA.
i will crosstest now with a VM on the working Host.
its a 12-Disk Synology-NAS with btrfs.
On that NFS-Storage are multiple folders exported as remotes in my XOfs Installations.
As i said. All other Hosts/XOfs Installations work fine on that NFS-Storage.
Only this one specific XCP-ng Host has these Problems. So i think its not XenOrchestra related. It seems to be a problem with XCP-ng on this specific Host, but the identical second Host does not have this Problem.
Thx @olivierlambert for extending the trial.
I will make new tests with the XOA and tell my findings.
Meanwhile i found out why the backup-job is hanging forever on my new XOfs Installation.
When i mounted the NFS-remote, i checked the option "Store backup as multiple data".
I removed the remote-nfs and reconnected it without that option.
Now the full backup is working as expected and the first delta fails with "Error: stream has ended with not enough data (actual: 430, expected: 512)".
So maybe there is a difference in the error handling when this option is active and the exception is not handled correctly.
Just for your Information.
I will come back when i tested with XOA.
@olivierlambert
Many thanks for your help.
Wrote you a p.m. with my e-mail. 
Hi,
i have a really strange behaviour on one of your xcp-ng Hosts.
We have some XCP-ng Pools and 2 identical StandAlone Hosts.
We use Delta-Backups (nightly) on a Xen-Orchestra VM from sources.
A few weeks ago Delta Backups suddenly stopped working on only one of the two Standalone-Hosts, while Delta Backups keep working without any Problems on all other Hosts/Pools.
The two identical Stand-Alone Hosts are:
Both StandAlone Hosts are absolutely identical (even Firmware up2date and latest XCP-ng Patchlevel and rebootet in the last days, to try if anything will fix the problem)
As the error initially appeared, the backup-logs started saying "stream has ended with not enough data", at the transfer-stage of the delta backups.
I then started to clean snapshots and old backups on some VMs.
After that, the first full backup of a that VMs was working fine, but the second then delta backup showed the same error.
To dig deeper, i installed a fully new ubuntu 22 VM and installed Xen Orchestra from sources again and connected the 2 Standalone Hosts on that new XOfs-VM with a remote NFS-backup-remote.
Same again. Initial Full-Backup works fine, first Delta fails one that one Host only, while working without problems on the other Host.
But this time with staying in "transfer" forever. This status is staying even for days and the backup Job never finishes, so the job next day fails with "Error: the job (x) is already running".
Today i restarted the XOfs-VM and updated to commit "afadc" and tried to reproduce with a new backup job with just one single VM.
It seems to be a XCP-ng related thing, cause the other identical Host is working perfect.
On that one Host i have the same thing. Initial Full is working, Delta comes never back and stuck at stage "transfer".
When i watch the xe task-list while the backup is running, it seems the export-task is working fine for the delta and there is new data on the nfs-remote. Then at 100% the task dissapears, but the Backup-job stays in transfer and never comes Back.
To eliminate all things maybe related to my "from sources" Installation (even the error is only on this one host and all others are working fine), i deployed a XOA-VM, but i cant start a free trial (you already consumed ...) and so i can not test Delta Backup.
Do you have any ideas or maybe had a similar issue in the past?
Kind regards
Alex
@stormi
Thx for your reply.
I try to wait so i can do it all in one Task.
Kind Regards
Alex
Sorry for my impatience.
Is there any news on ETA?
We are actually planning a maintenance downtime for firmware Upgrades and some other changes on a bunch of hosts and it would be cool do the updates on the same time slot to avoid double reboot/shutdown in short time.
Kind Regards
Alex
Updated and tested:
Everthing seems to work as expected.
(new VM, live-migration, snapshots, clone VMs, import/export VMs)
@stormi
is there some information about that regression?
maybe i can help to test.
have a lab pool here with 3 hosts on a iscsi SR.
@mtill
Sorry, missed your point "The base vdk of the VM we want to recover was un-usable".
(In Case you mean the first full image of your backups storage)
@mtill
just a short question.
Why do you want to restore XVA when you have actual delta Backups?
maybe missunderstandig delta-backups?
if you make delta backups using xoa or XenOrchestra, just restore your latest delta backup to a working xcp-ng oder Xenserver with sufficient free storage.
Delta Backups always have an inital Full-Copy.
Or did i get your question wrong?
@stormi
Did some Testing over the Weekend too.
Setup with 2 Hosts in a Pool and shared iSCSI-LMV Storage with multipath 8 paths per LUN.
Anything seems to work fine (migrate/import/cross-pool-migrate/snapshots/backups).
Even our longtime Problem (snapshots taking much too long) is getting much better (still not good, but much better).
@julien-f
thx for the quick fix. works perfectly now.
@julien-f : thx for investigating.
Tried a fresh install half of the day with the same error and thought i am doing something wrong.
Then i tried updateing a running installation and hit the same error.
I keep investigating by myself too.
Hi all,
just for your Information.
For requests from germany, mirrors.xcp-ng.org is always redirecting to xcp-ng.schulzalex.de.
The SSL-Cert for xcp-ng.schulzalex.de has expired two days ago, so all tries to update are not working actually.
Is there some possibility to check the Certificates before redirecting on mirrors.xcp-ng.org and redirect to another (working) mirror if its not working?
thanks in advance
Alex
Hi all,
first of all my Environment:
XCP-NG 7.6 Pool (4 Members)
Shared (LVM) Block-Storage (FC and iSCSI)
XenOrchestra (from sources, 5.42.1 on Ubuntu 18).
Backup-Remote via NFS.
Nightly Delta Backup 00:30 - Retention 8
From time to time i have the problem of getting an Error on one or more VMs in my Delta-Backup-Job.
Error (example): "invalid header checksum 1143961443, expected 4294883451"
I have seen this over the last few Months about 10 to 15 Times on different VMs.
(a working solution is to delete/move the existing backups of this VM, so it starts with a new full one)
After some investigation we now found the cause of the problem.
Whenever we resize one or more disks of a VM, all looks good for that moment.
But when the retention Limit is reached, the Backup of the VM(s) with a former resized Disk will fail with error "invalid header checksum".
All later Backups will even fail, so the existing Backups will remain from the day of the resize till the reach of retention.
(in my case, Retention 8. The VM today shows 8 restorable Backups from 01.06. to 08.06. - The Disk was resized on 31.05.)
Does anyone else expieriance the same Problem?
Might this be an exeption that is simply not intended?
Or could it be that i missed some dependencies in my from source built?
Kind regards
Alex