Unable to export reliable backups

[bsodmike]

Hi all,

I've extensively tested the above and here's the approach taken. I'm running XCPng on two separate hosts,

• xcpng-host1
• xcpng-host0 (newer server).

I have been able to verify via iperf3, ~9+Gbit/s works just fine, when connecting to my primary Freenas server at 10.0.2.2. I have modified /etc/fstab/ to mount a CIFS connection

# Connect to freenas-primary over 10G interface
//10.0.2.2/xoa-data-backup       /mnt/xoa-data-backup              cifs    uid=0,credentials=/root/xoa-backup.cifs,iocharset=utf8,vers=3.0,noperm 0 0

I have also tried adding an NFS share via XOAs "New file system remote", with the mount point \10.0.2.2:/mnt/big-primary/xcpng_data and options vers=3,soft,proto=tcp

Starting a backup, always looks like this:

Notice there's a blip of a transfer.

Ultimately the backup is marked as successful (takes ages) and the file size is between 1-5GB.

If I try to re-import via XOA, it just fails (never succeeds with error "IMPORT_ERROR_PREMATURE_EOF()". A manual xe vm-import reports The VM could not be imported; the end of the file was reached prematurely.

However, via the xcpng host I am able to issue xe vm-export and generate ~40GB+ XVA backups.

Thanks.

[olivierlambert]

1. What about delta backup?
2. Do you have servers with the same XCP-ng version?

[bsodmike]

@olivierlambert

• Delta backup, similar behaviour, only 8% and then it appears to be "stuck" for the time being.
• host1 (the older one) is running xcpng 7.6.0. The newer host0 is running 8.2.0

Both are however running xo-server 5.73.0 and xo-web 5.76.0

[bsodmike]

@olivierlambert The delta backup finally shows an error:

Duration: 18 minutes
Error: VDI_IO_ERROR(Device I/O errors)

[Danp]

@bsodmike Seems like you are still having issues with your FreeNAS setup. Do you have a different (non-FreeNAS) storage device that you can use for testing?

I noticed that your host0 is missing some patches. Probably best to address that, too. Have you tried doing a CR backup between the two hosts?

[bsodmike]

@danp Hi Dan, I am easily able to export to FreeNAS via xe vm-export. The only issue I'm facing right now is getting XOA to run backups to NFS (or even CIFS) shares.

Right now, I do not have any other hosts to backup to, but I fail to see the FreeNAS host I'm using as the failure point given above. Thoughts?

[bsodmike]

@danp Also FYI, XOA failing backups (to the NFS share mentioned above) should not create smaller files and go "green" marking the backup as successful.

These are all failed attempts on host0.

[Danp]

@bsodmike Are you using XOA (the pre-built appliance) or XO from sources?

[bsodmike]

@danp I tried both actually, but right now I've switched to install via source using https://github.com/ronivay/XenOrchestraInstallerUpdater

[bsodmike]

@danp Here's an example that I just ran of a manual export to a NFS share that I setup on the host,

[23:22 xcpng-host0-mgmt xcpng-manual-export]# xe vm-export vm=docker1 filename=/mnt/xcpng_data/xcpng-manual-export/docker1_`date +"%d%m%Y"`.xva
Export succeeded
[23:28 xcpng-host0-mgmt xcpng-manual-export]# pwd
/mnt/xcpng_data/xcpng-manual-export
[23:29 xcpng-host0-mgmt xcpng-manual-export]# ls -lah
total 8.7G
drwxr-xr-x 2 1001 1005   3 Dec 20 23:23 .
drwxrwx--- 9 1004 1005  12 Dec 20 23:16 ..
-rw------- 1 root 1005 36G Dec 20 23:28 docker1_20122020.xva
[23:29 xcpng-host0-mgmt xcpng-manual-export]# mount -t nfs
freenas-primary.intranet:/mnt/big-primary/xcpng_data on /mnt/xcpng_data type nfs (rw,relatime,vers=3,rsize=131072,wsize=131072,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=10.0.0.149,mountvers=3,mountport=727,mountproto=tcp,local_lock=none,addr=10.0.0.149)

[olivierlambert]

Obviously there's something wrong in the setup. Basic backups or even delta issues like this would have trigger QA before a release, not talking about the 3000+ active users.

Now the question is: what's wrong exactly? Hard to tell, but I have suspicions on your storage itself. VDI_IO_ERROR(Device I/O errors) is related to XCP-ng issue on export/import.

[bsodmike]

@olivierlambert That makes sense. FYI I only got the VDI error when attempting delta backup via XO. xe vm-export has not provided any errors.

Just right now I was able to export a 35Gig XVA file, reimport using xe vm-import and the VM boots just fine.

All over the SAME NFS share that doesn't work with XOA. Hmm...

[olivierlambert]

Check dmesg in both host and XO VM to see if you can spot anything.

[bsodmike]

@olivierlambert I have a fedora workstation, which I could use to serve an NFS share. Hmm, I could try this to isolate the Freenas share?

[olivierlambert]

That would be interesting. This won't rule out everything but at least we might notice something different.