Thank you for your help @julien-f. We will change our backup strategy according to this limitation.

I'd suggest adding a warning in the web UI and xo-server console logs regarding File Recovery if the number of delta backups is set to a number greater than 99. It is a bit troubling that it took so long to find out this was a simple limitation known for more than a year.

Hi, I've noticed the addition of the commit about the 99 limit warning. That's good news.

I am now having trouble trying to understand the meaning of the Full Backup Interval setting.

I have the following scenario:

• Backups for the last 30 days, daily.
• Backups for the last 52 weeks, weekly.
• I want to make sure that the delta backups are not corrupt periodically (let's say once a month).

I don't understand how to set the schedules, the backup retention, the full backup interval.

I also need answers to the following questions, and I haven't been able to understand this from the documention.

• Do I need one delta backup with 2 schedules, or 2 backups with one schedule each ?
• I've noticed that full backup interval is per backup, whereas retention is per schedule. How do these two settings interact with multiple schedules?
• On what dates will the transfers be large ?
• How many full backups will be stored on the remote at any given time ? When will the merges happen?

Thanks

Thank you for your help @julien-f. We will change our backup strategy according to this limitation.

I'd suggest adding a warning in the web UI and xo-server console logs regarding File Recovery if the number of delta backups is set to a number greater than 99. It is a bit troubling that it took so long to find out this was a simple limitation known for more than a year.

@olivierlambert @julien-f

It is not clear to me if I can simply activate that option right now, in the state that I am with backups of 121 days that don't work past 99. Will vhdimount be needed to perform the first full backup or not?

Also we're planning on setting that interval somewhere around 60 (2 months). Can this open issue be encountered: https://github.com/vatesfr/xen-orchestra/issues/4987 ?

jcharaoui created this issue in vatesfr/xen-orchestra

closed Delta backup "out of range" error in merge phase #4987

@julien-f @olivierlambert

If we pay for support, are there any feasible options to remove this limitation in the next few days? Our target is to be able to have at least 365 backups.
Since it is very easy to reproduce, you shouldn't need access set up, I assume.

Following your suggestion to use vhdimount.

I tried to mount vhd files that work and those that don't work:

Working version:

sudo vhdimount /run/xo-server/mounts/8118efa3-4968-4e87-96b1-7612e80222b8/xo-vm-backups/6b291d91-c7cc-2588-b90b-e53c2e8e21fd/vdis/2a7b744e-d238-4ad6-a032-541c111f72ae/77ef5fea-0040-4b8e-b1ef-1704712870c6/20200519T040005Z.vhd /tmp/vhdimnt/
vhdimount 20170223

xoa@yul1-xoa-001v:~$ sudo ls -lah /tmp/vhdimnt 
total 36K
dr-xr-xr-x  2 root root    0 Jun 10 14:24 .
drwxrwxrwt 11 root root  32K Jun 10 14:24 ..
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi1
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi10
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi11
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi12
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi13
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi14
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi15
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi16
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi17
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi18
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi19
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi2
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi20
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi21
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi22
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi23
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi24
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi25
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi26
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi27
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi28
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi29
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi3
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi30
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi31
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi32
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi33
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi34
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi35
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi36
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi37
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi38
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi39
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi4
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi40
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi41
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi42
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi43
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi44
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi45
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi46
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi47
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi48
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi49
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi5
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi50
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi51
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi52
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi53
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi54
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi55
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi56
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi57
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi58
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi59
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi6
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi60
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi61
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi62
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi63
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi64
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi65
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi66
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi67
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi68
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi69
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi7
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi70
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi71
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi72
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi73
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi74
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi75
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi76
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi77
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi78
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi79
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi8
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi80
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi81
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi82
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi83
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi84
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi85
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi86
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi87
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi88
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi89
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi9
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi90
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi91
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi92
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi93
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi94
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi95
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi96
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi97
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi98
-r--r--r--  1 root root 960G Jun 10 14:24 vhdi99
xoa@yul1-xoa-001v:~$ sudo fusermount -u /tmp/vhdimnt 

Not working one:

xoa@yul1-xoa-001v:~$ sudo vhdimount /run/xo-server/mounts/8118efa3-4968-4e87-96b1-7612e80222b8/xo-vm-backups/6b291d91-c7cc-2588-b90b-e53c2e8e21fd/vdis/2a7b744e-d238-4ad6-a032-541c111f72ae/77ef5fea-0040-4b8e-b1ef-1704712870c6/20200520T040005Z.vhd /tmp/vhdimnt/
vhdimount 20170223

xoa@yul1-xoa-001v:~$ sudo ls -lha /tmp/vhdimnt 
total 0

The fact that vhdiXX has only 2 digits in the filename is suspicious... there are exactly 99 vhdi files in the mount directory. And May 19 corresponds to 99 days since February 11th when the first backup was made on this disk.

The bug is therefore not on our side, except if there's a note somewhere in the documentation that only 99 delta backups can be made.

Based on this:
https://github.com/libyal/libvhdi/blob/58c6aa277fd2b245e8ea208028238654407f364c/vhditools/mount_file_entry.c#L675

It looks like vhdimount doesn't like more than 99 backups.

We are considering paid support. How much time do you evaluate fixing such a problem would take?

Thanks, but exactly which command incrementally builds the code, if only a single file is modified? yarn, or yarn build? The build process is quite long, and since I am not familiar with JavaScript I need to iterate a lot.

Let me rephrase that: I do want to help since I am helping reproduce and investigate a bug that is very important for both our companies.

If you want me to add additional log messages, I can do that. Just need an efficient method to:

• modify the source,
• re-compile the modified source,
• execute.

I'm not at all familiar with the build system basically.

I do want to help since solving this issue is very important for my company.

Thanks

Hi,

I don't think it is the same failure as @daKju, because in my case the failure is before that, at function backupNg.listPartitions, whereas @daKju's is with backupNg.listFiles.

I'm not sure which function fails after that, since we're not a JavaScript shop, but I'd start looking in the file file-restore-ng.js (on master) starting at lines 303:

303     const diskPath = handler._getFilePath('/' + diskId)
304     const mountDir = await tmpDir()
305     $defer.onFailure(rmdir, mountDir)
306 
307     await execa('vhdimount', [diskPath, mountDir])
[...]

Hi,

It is not random.

It happens every time with the same backups from May 20 and newer on this particular VM disk (i have one every day), but it works with May 19 and older. On other VMs i have a different threshold date, but same behaviour. The files on the NAS are all there. I tried a full backup and it worked.

The backups are on a NFS share on a NAS. The remote is reachable (i've run the test).