File Restore : error scanning disk for recent delta backups but not old
-
Hi,
I don't think it is the same failure as @daKju, because in my case the failure is before that, at function
backupNg.listPartitions, whereas @daKju's is withbackupNg.listFiles.I'm not sure which function fails after that, since we're not a JavaScript shop, but I'd start looking in the file
file-restore-ng.js(on master) starting at lines 303:303 const diskPath = handler._getFilePath('/' + diskId) 304 const mountDir = await tmpDir() 305 $defer.onFailure(rmdir, mountDir) 306 307 await execa('vhdimount', [diskPath, mountDir]) [...] -
@julien-f will take a look when he can, he's pretty busy right now
In the mean time, feel free to track the issue deeper on your side if you can -
If you want me to add additional log messages, I can do that. Just need an efficient method to:
- modify the source,
- re-compile the modified source,
- execute.
I'm not at all familiar with the build system basically.
I do want to help since solving this issue is very important for my company.
Thanks
-
It's all in the install doc: https://xen-orchestra.com/docs/installation.html#fetching-the-code
Also, if it's really important for your company, you might be interested to have professional support for it
-
Thanks, but exactly which command incrementally builds the code, if only a single file is modified? yarn, or yarn build? The build process is quite long, and since I am not familiar with JavaScript I need to iterate a lot.
Let me rephrase that: I do want to help since I am helping reproduce and investigate a bug that is very important for both our companies.
-
Thing is, we don't have similar reports for now, so it's possibly something on your side. That's why having pro support if it's very urgent for you makes sense, but it's your call
@julien-f will give you some hints to get a rebuild on only what you need
-
We are considering paid support. How much time do you evaluate fixing such a problem would take?
-
First step before fixing an issue is to be able to reproduce it. If it's linked to a data issue itself on those VHDs (or during export), then it's not a "bug" per se.
That's why it's a bit hard to be able to give you detailed input.
Alternatively, you can use
vhdimounton your side to access those VHDs and see if it works or not.@julien-f will give you the commands when he's available.
edit: you can open a support ticket on xen-orchestra.com Also open a support tunnel in your XOA so we can take a look remotely
-
Following your suggestion to use vhdimount.
I tried to mount vhd files that work and those that don't work:
Working version:
sudo vhdimount /run/xo-server/mounts/8118efa3-4968-4e87-96b1-7612e80222b8/xo-vm-backups/6b291d91-c7cc-2588-b90b-e53c2e8e21fd/vdis/2a7b744e-d238-4ad6-a032-541c111f72ae/77ef5fea-0040-4b8e-b1ef-1704712870c6/20200519T040005Z.vhd /tmp/vhdimnt/ vhdimount 20170223 xoa@yul1-xoa-001v:~$ sudo ls -lah /tmp/vhdimnt total 36K dr-xr-xr-x 2 root root 0 Jun 10 14:24 . drwxrwxrwt 11 root root 32K Jun 10 14:24 .. -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi1 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi10 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi11 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi12 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi13 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi14 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi15 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi16 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi17 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi18 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi19 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi2 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi20 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi21 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi22 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi23 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi24 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi25 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi26 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi27 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi28 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi29 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi3 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi30 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi31 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi32 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi33 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi34 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi35 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi36 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi37 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi38 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi39 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi4 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi40 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi41 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi42 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi43 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi44 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi45 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi46 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi47 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi48 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi49 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi5 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi50 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi51 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi52 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi53 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi54 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi55 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi56 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi57 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi58 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi59 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi6 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi60 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi61 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi62 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi63 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi64 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi65 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi66 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi67 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi68 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi69 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi7 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi70 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi71 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi72 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi73 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi74 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi75 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi76 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi77 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi78 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi79 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi8 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi80 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi81 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi82 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi83 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi84 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi85 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi86 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi87 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi88 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi89 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi9 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi90 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi91 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi92 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi93 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi94 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi95 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi96 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi97 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi98 -r--r--r-- 1 root root 960G Jun 10 14:24 vhdi99 xoa@yul1-xoa-001v:~$ sudo fusermount -u /tmp/vhdimntNot working one:
xoa@yul1-xoa-001v:~$ sudo vhdimount /run/xo-server/mounts/8118efa3-4968-4e87-96b1-7612e80222b8/xo-vm-backups/6b291d91-c7cc-2588-b90b-e53c2e8e21fd/vdis/2a7b744e-d238-4ad6-a032-541c111f72ae/77ef5fea-0040-4b8e-b1ef-1704712870c6/20200520T040005Z.vhd /tmp/vhdimnt/ vhdimount 20170223 xoa@yul1-xoa-001v:~$ sudo ls -lha /tmp/vhdimnt total 0The fact that vhdiXX has only 2 digits in the filename is suspicious... there are exactly 99 vhdi files in the mount directory. And May 19 corresponds to 99 days since February 11th when the first backup was made on this disk.
The bug is therefore not on our side, except if there's a note somewhere in the documentation that only 99 delta backups can be made.
Based on this:
https://github.com/libyal/libvhdi/blob/58c6aa277fd2b245e8ea208028238654407f364c/vhditools/mount_file_entry.c#L675It looks like vhdimount doesn't like more than 99 backups.
-
@mtango said in File Restore : error scanning disk for recent delta backups but not old:
It looks like vhdimount doesn't like more than 99 backups.
Indeed, it's a known issue: https://github.com/vatesfr/xen-orchestra/issues/4032
-
If we pay for support, are there any feasible options to remove this limitation in the next few days? Our target is to be able to have at least 365 backups.
Since it is very easy to reproduce, you shouldn't need access set up, I assume. -
It probably won't be easy to remove this limitation, especially in the next few days.
If you can use the full backup interval setting, it should remove this issue.
-
Indeed, the full backup interval seems the right option.
-
It is not clear to me if I can simply activate that option right now, in the state that I am with backups of 121 days that don't work past 99. Will vhdimount be needed to perform the first full backup or not?
Also we're planning on setting that interval somewhere around 60 (2 months). Can this open issue be encountered: https://github.com/vatesfr/xen-orchestra/issues/4987 ?
closed Delta backup "out of range" error in merge phase #4987
-
@mtango said in File Restore : error scanning disk for recent delta backups but not old:
Will vhdimount be needed to perform the first full backup or not?
No, it's only used for file restore.
@mtango said in File Restore : error scanning disk for recent delta backups but not old:
Can this open issue be encountered: https://github.com/vatesfr/xen-orchestra/issues/4987 ?
We have no idea what trigger this condition, it may be old corrupted VHD files.
If you want to check your backups, you can take a look at this post
-
Thank you for your help @julien-f. We will change our backup strategy according to this limitation.
I'd suggest adding a warning in the web UI and xo-server console logs regarding File Recovery if the number of delta backups is set to a number greater than 99. It is a bit troubling that it took so long to find out this was a simple limitation known for more than a year.
-
Hi, I've noticed the addition of the commit about the 99 limit warning. That's good news.
I am now having trouble trying to understand the meaning of the Full Backup Interval setting.
I have the following scenario:
- Backups for the last 30 days, daily.
- Backups for the last 52 weeks, weekly.
- I want to make sure that the delta backups are not corrupt periodically (let's say once a month).
I don't understand how to set the schedules, the backup retention, the full backup interval.
I also need answers to the following questions, and I haven't been able to understand this from the documention.
- Do I need one delta backup with 2 schedules, or 2 backups with one schedule each ?
- I've noticed that full backup interval is per backup, whereas retention is per schedule. How do these two settings interact with multiple schedules?
- On what dates will the transfers be large ?
- How many full backups will be stored on the remote at any given time ? When will the merges happen?
Thanks
