XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    [SOLVED] XO From source restore backup fails with VDI_IO_ERROR

    Scheduled Pinned Locked Moved Solved Xen Orchestra
    16 Posts 5 Posters 1.8k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B
      BitSprocket
      last edited by BitSprocket

      I've posted this on GitHub but was directed here. Here's what I posted there: This is URGENT as I have rebuilt my two servers with XCP-NG 8.0 and need to restore these backups for production tomorrow (Monday)

      Context

      XO origin: the sources
Versions:
    Node: 8.10.0
    xo-web: 5.51.1
    xo-server: 5.51.1

      Expected behavior

      Restoring a backup from remote NFS storage should restore vm to server without fail
      Current behavior

      Moments after initiating a restore, the job fails citing VDI_IO_ERROR(Device I/O Errors). It seems to be very similar to the problem here: #3242 Not sure where to look for further information but here's the log details:2019-11-02T23_57_31.230Z - XO.log.txt

      1 Reply Last reply Reply Quote 0
      • DanpD
        Danp Vates 🪐 Pro Support Team
        last edited by

        This is caused by a recent commit. You can avoid the issue by building from an earlier point in time. Try this --

        • Switch to the root directory containing the XO source code
        • Issue the command git checkout 46da11a52e86deac9bfaecfa147e292cc15a2f0a
        • Issue the command yarn; yarn build
        • Restart the xo-server process
        C 1 Reply Last reply Reply Quote 1
        • B
          BitSprocket
          last edited by

          @Danp Thank you for a VERY fast response indeed! Less than 5 minutes - that's gotta be a record! I will revert to that commit and give it a spin. Is there a thread on this (couldn't find one) so I can monitor for newer commits that fix this problem?

          1 Reply Last reply Reply Quote 0
          • B
            BitSprocket
            last edited by

            That fixed it. Thanks for your help @Danp !!

            1 Reply Last reply Reply Quote 1
            • C
              chevybeef @Danp
              last edited by

              @Danp Unfortunately this didn't resolve the problem in my case. Both SMB (Windows) and NFS (FreeNAS) Delta Backups fail with VDI_IO_ERROR with "some" but "not all" VM backups. I'm using Debian 9 and the XenOrchestraInstallerUpdater script to build XO. Also tried Ubuntu 1804 with the xenorchestra_installer script with the same results, gets to 44% and then gives the VDI_IO_ERROR. Anyone know what XOA is built on, Debian, Ubuntu, other, what version?

              DanpD 1 Reply Last reply Reply Quote 0
              • DanpD
                Danp Vates 🪐 Pro Support Team @chevybeef
                last edited by

                @chevybeef There are many different possible causes of these errors, and you'll need to provide more details for us to assist you. Please tell us more about your particular setup, how long you've been using XO, when did these issues first appear, etc.

                XenOrchestraInstallerUpdater

                I haven't personally used this script, but perhaps @ronivay will jump in with some advice. Also, you could try building the XO VM manually using the official guide.

                Anyone know what XOA is built on

                I believe the latest version is Debian 10. Have you tried testing with XOA? If so, what were the results?

                ronivayR 1 Reply Last reply Reply Quote 0
                • ronivayR
                  ronivay Top contributor @Danp
                  last edited by ronivay

                  @Danp said in [SOLVED] XO From source restore backup fails with VDI_IO_ERROR:

                  I haven't personally used this script, but perhaps @ronivay will jump in with some advice. Also, you could try building the XO VM manually using the official guide.

                  @chevybeef Indeed hard to tell with this information only. I understood that two different install scripts give the same outcome? Also only some of these fails. Are they always the same ones?

                  Which XCP-NG version you’re running? What kind of delay and transfer speed you have to your remote from XO?

                  Check this: https://github.com/vatesfr/xen-orchestra/issues/3242

                  I suppose this could also have same effect while restoring.

                  mike59999 created this issue in vatesfr/xen-orchestra

                  closed [Backup Ng deltaBackup] VDI_IO_ERROR(Device I/O errors) #3242

                  C 1 Reply Last reply Reply Quote 0
                  • C
                    chevybeef @ronivay
                    last edited by

                    @ronivay, @Danp Yes, 2 different install scripts, 2 different distros and 2 different remotes, NFS and SMB, same results. They are always the same ones failing, the commonality being they are not running VMs, they are larger that the others i.e. 40GB and 16GB, and they are located on a HD not SSD, the HD also being used by another VM to capture CCTV. Transfer speed is 1Gbps, SMB share is on same HD as well.

                    I'm running version 8.0.0. and the latest XO running on updated version of Debian 9 with your latest version of the script (a note here is that it hadn't installed cifs-utils but even after installing that manually it didn't help). Debian is running with a single CPU and 2GB of RAM.

                    The host itself is a relatively low spec machine, G4600 (2c4t) with 16GB, 2 x SSDs and 1 x HD.

                    Issue 3242 looks interesting.

                    ronivayR DanpD 2 Replies Last reply Reply Quote 0
                    • ronivayR
                      ronivay Top contributor @chevybeef
                      last edited by

                      @chevybeef said in [SOLVED] XO From source restore backup fails with VDI_IO_ERROR:

                      @ronivay, @Danp Yes, 2 different install scripts, 2 different distros and 2 different remotes, NFS and SMB, same results. They are always the same ones failing, the commonality being they are not running VMs, they are larger that the others i.e. 40GB and 16GB, and they are located on a HD not SSD, the HD also being used by another VM to capture CCTV. Transfer speed is 1Gbps, SMB share is on same HD as well.

                      I'm running version 8.0.0. and the latest XO running on updated version of Debian 9 with your latest version of the script (a note here is that it hadn't installed cifs-utils but even after installing that manually it didn't help). Debian is running with a single CPU and 2GB of RAM.

                      The host itself is a relatively low spec machine, G4600 (2c4t) with 16GB, 2 x SSDs and 1 x HD.

                      Issue 3242 looks interesting.

                      Okay, thanks. cifs-utils dependency is actually fixed recently, but manually installing it should do the trick as well.

                      Unfortunately i can’t figure out any reason in the installation procedure my script makes that could cause this issue.

                      I’ve seen some issues even with installation when XO VM had 2GB RAM (even though that should be enough). Maybe you could try increasing that just to rule it out?

                      1 Reply Last reply Reply Quote 0
                      • DanpD
                        Danp Vates 🪐 Pro Support Team @chevybeef
                        last edited by

                        @chevybeef said in [SOLVED] XO From source restore backup fails with VDI_IO_ERROR:

                        Debian is running with a single CPU and 2GB of RAM.

                        Also, try bumping the vCPUs up to 2. I vaguely recall an issue where only one CPU was in use, but don't recall the specifics.

                        C 1 Reply Last reply Reply Quote 0
                        • C
                          chevybeef @Danp
                          last edited by

                          @Danp @ronivay Thanks I'll try both of those things as soon as I can get passed "Job canceled to protect the VDI chain"

                          1 Reply Last reply Reply Quote 0
                          • C
                            chevybeef
                            last edited by

                            @Danp @ronivay So here's what I tried. I moved the XO VM to another machine where I could give it more resources, 2 vCPUs and 4GB RAM. I couldn't give these resources on the main machines without over-committing. Then I tried to migrate the 24GB VM I want to backup onto the second machine but unfortunately that didn't work and it failed at 45% with "VDI_COPY_FAILED(End_of_file)". I thought perhaps it's corrupt so tried migrating another 16GB VM and that also failed with the same error. Any ideas how I might manually try moving the VMs?

                            C 1 Reply Last reply Reply Quote 0
                            • C
                              chevybeef @chevybeef
                              last edited by

                              @chevybeef Using XCP-ng Center 8.0.1 the move also fails with: "The VDI copy action has failed"

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                So this is likely not an XO issue 😉

                                C 1 Reply Last reply Reply Quote 0
                                • C
                                  chevybeef @olivierlambert
                                  last edited by

                                  @olivierlambert said in [SOLVED] XO From source restore backup fails with VDI_IO_ERROR:

                                  So this is likely not an XO issue 😉

                                  Any ideas how I might manually try moving the VMs?

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    This is not related to the original, feel free to create a new one regarding this issue.

                                    You can't "manually" move a VM, there's always a client to do that (XO, xe, XCP-ng Center etc.). So you need to understand why it's failing first.

                                    1 Reply Last reply Reply Quote 0
                                    • First post
                                      Last post