[SOLVED] XO From source restore backup fails with VDI_IO_ERROR

BitSprocket

I've posted this on GitHub but was directed here. Here's what I posted there: This is URGENT as I have rebuilt my two servers with XCP-NG 8.0 and need to restore these backups for production tomorrow (Monday)

Context

XO origin: the sources
Versions:
    Node: 8.10.0
    xo-web: 5.51.1
    xo-server: 5.51.1

Expected behavior

Restoring a backup from remote NFS storage should restore vm to server without fail
Current behavior

Moments after initiating a restore, the job fails citing VDI_IO_ERROR(Device I/O Errors). It seems to be very similar to the problem here: #3242 Not sure where to look for further information but here's the log details:2019-11-02T23_57_31.230Z - XO.log.txt

Danp

This is caused by a recent commit. You can avoid the issue by building from an earlier point in time. Try this --

Switch to the root directory containing the XO source code
Issue the command git checkout 46da11a52e86deac9bfaecfa147e292cc15a2f0a
Issue the command yarn; yarn build
Restart the xo-server process

BitSprocket

Danp Thank you for a VERY fast response indeed! Less than 5 minutes - that's gotta be a record! I will revert to that commit and give it a spin. Is there a thread on this (couldn't find one) so I can monitor for newer commits that fix this problem?

BitSprocket

That fixed it. Thanks for your help Danp !!

chevybeef

Danp Unfortunately this didn't resolve the problem in my case. Both SMB (Windows) and NFS (FreeNAS) Delta Backups fail with VDI_IO_ERROR with "some" but "not all" VM backups. I'm using Debian 9 and the XenOrchestraInstallerUpdater script to build XO. Also tried Ubuntu 1804 with the xenorchestra_installer script with the same results, gets to 44% and then gives the VDI_IO_ERROR. Anyone know what XOA is built on, Debian, Ubuntu, other, what version?

Danp

chevybeef There are many different possible causes of these errors, and you'll need to provide more details for us to assist you. Please tell us more about your particular setup, how long you've been using XO, when did these issues first appear, etc.

XenOrchestraInstallerUpdater

I haven't personally used this script, but perhaps ronivay will jump in with some advice. Also, you could try building the XO VM manually using the official guide.

Anyone know what XOA is built on

I believe the latest version is Debian 10. Have you tried testing with XOA? If so, what were the results?

ronivay

Danp said in [SOLVED] XO From source restore backup fails with VDI_IO_ERROR:

I haven't personally used this script, but perhaps ronivay will jump in with some advice. Also, you could try building the XO VM manually using the official guide.

chevybeef Indeed hard to tell with this information only. I understood that two different install scripts give the same outcome? Also only some of these fails. Are they always the same ones?

Which XCP-NG version you’re running? What kind of delay and transfer speed you have to your remote from XO?

Check this: https://github.com/vatesfr/xen-orchestra/issues/3242

I suppose this could also have same effect while restoring.

mike59999 created this issue in vatesfr/xen-orchestra

closed [Backup Ng deltaBackup] VDI_IO_ERROR(Device I/O errors) #3242

chevybeef

ronivay, Danp Yes, 2 different install scripts, 2 different distros and 2 different remotes, NFS and SMB, same results. They are always the same ones failing, the commonality being they are not running VMs, they are larger that the others i.e. 40GB and 16GB, and they are located on a HD not SSD, the HD also being used by another VM to capture CCTV. Transfer speed is 1Gbps, SMB share is on same HD as well.

I'm running version 8.0.0. and the latest XO running on updated version of Debian 9 with your latest version of the script (a note here is that it hadn't installed cifs-utils but even after installing that manually it didn't help). Debian is running with a single CPU and 2GB of RAM.

The host itself is a relatively low spec machine, G4600 (2c4t) with 16GB, 2 x SSDs and 1 x HD.

Issue 3242 looks interesting.

ronivay

chevybeef said in [SOLVED] XO From source restore backup fails with VDI_IO_ERROR:

ronivay, Danp Yes, 2 different install scripts, 2 different distros and 2 different remotes, NFS and SMB, same results. They are always the same ones failing, the commonality being they are not running VMs, they are larger that the others i.e. 40GB and 16GB, and they are located on a HD not SSD, the HD also being used by another VM to capture CCTV. Transfer speed is 1Gbps, SMB share is on same HD as well.

I'm running version 8.0.0. and the latest XO running on updated version of Debian 9 with your latest version of the script (a note here is that it hadn't installed cifs-utils but even after installing that manually it didn't help). Debian is running with a single CPU and 2GB of RAM.

The host itself is a relatively low spec machine, G4600 (2c4t) with 16GB, 2 x SSDs and 1 x HD.

Issue 3242 looks interesting.

Okay, thanks. cifs-utils dependency is actually fixed recently, but manually installing it should do the trick as well.

Unfortunately i can’t figure out any reason in the installation procedure my script makes that could cause this issue.

I’ve seen some issues even with installation when XO VM had 2GB RAM (even though that should be enough). Maybe you could try increasing that just to rule it out?

Danp

chevybeef said in [SOLVED] XO From source restore backup fails with VDI_IO_ERROR:

Debian is running with a single CPU and 2GB of RAM.

Also, try bumping the vCPUs up to 2. I vaguely recall an issue where only one CPU was in use, but don't recall the specifics.

chevybeef

Danp ronivay Thanks I'll try both of those things as soon as I can get passed "Job canceled to protect the VDI chain"

chevybeef

Danp ronivay So here's what I tried. I moved the XO VM to another machine where I could give it more resources, 2 vCPUs and 4GB RAM. I couldn't give these resources on the main machines without over-committing. Then I tried to migrate the 24GB VM I want to backup onto the second machine but unfortunately that didn't work and it failed at 45% with "VDI_COPY_FAILED(End_of_file)". I thought perhaps it's corrupt so tried migrating another 16GB VM and that also failed with the same error. Any ideas how I might manually try moving the VMs?

chevybeef

chevybeef Using XCP-ng Center 8.0.1 the move also fails with: "The VDI copy action has failed"

olivierlambert

So this is likely not an XO issue

chevybeef

olivierlambert said in [SOLVED] XO From source restore backup fails with VDI_IO_ERROR:

So this is likely not an XO issue

Any ideas how I might manually try moving the VMs?

olivierlambert

This is not related to the original, feel free to create a new one regarding this issue.

You can't "manually" move a VM, there's always a client to do that (XO, xe, XCP-ng Center etc.). So you need to understand why it's failing first.