XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    S3 Backup Retention Not Deleting Old Backups

    Scheduled Pinned Locked Moved Xen Orchestra
    14 Posts 3 Posters 1.2k Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S Offline
      stevewest15
      last edited by

      Hi,

      Been testing the S3 backup and it seems our b2 storage keeps growing even though the amount of data being backed up is pretty constant. I'm wondering if the S3 backup deletes old backups from b2 as part of the backup retention set in XO backup schedule? Or if I have setup something wrong on either XO or b2 which is causing the storage on b2 to keep growing without rotation.

      Here is what I have as part of my backup:

      125608b7-db1e-4c9f-99f6-c735729c7089-image.png
      5e79fb78-24dc-498c-94bf-0145592079e8-image.png

      f36a3fa2-9795-4976-a8bb-60e3ffdc7302-image.png

      In b2, I have the following:
      c23d0a5b-8110-4724-912f-2c8f1b886bf9-image.png

      3fee8c20-80af-4bc9-9217-edbbd9e3d294-image.png

      Thx,

      SW

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Hi,

        Do not interfere in file retention directly in Backblaze, that might remove legit files from XO backup.

        Also, in order to help you, we must know if you are using XOA or XO from the sources, and if you are fully up to date 🙂

        S 1 Reply Last reply Reply Quote 0
        • S Offline
          stevewest15 @olivierlambert
          last edited by

          @olivierlambert Good Morning, sorry didn't include that info:

          XO from sources, commit 4e88c
          xo-server 5.86.1
          xo-web 5.91.0

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by olivierlambert

            It's already 22 days old and more than 30 commits old. Please always check that your are on the latest master commit and rebuild before reporting an issue 🙂

            S 1 Reply Last reply Reply Quote 0
            • S Offline
              stevewest15 @olivierlambert
              last edited by

              @olivierlambert Thank you, didn't realize it's been that long. Just updated to latest commit (b78a9). I'll see in a week or so if the issue resolves itself.

              I'm wondering if I should change the b2 Lifecycle settings from 7 days to only last version? Just not clear on how XO full & delta backups with the b2 Lifecycle settings interact with each other:

              1d1a4a9a-c4fe-44fb-80e8-2af7f385028f-image.png

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                My gut feeling: avoid any interaction, because XO is not meant to have files removed by another program, this might destroy your backup.

                S 1 Reply Last reply Reply Quote 1
                • S Offline
                  stevewest15 @olivierlambert
                  last edited by

                  @olivierlambert Thank you, I've changed the settings on b2 to "Keep all versions of the file (default)".

                  S 1 Reply Last reply Reply Quote 1
                  • S Offline
                    stevewest15 @stevewest15
                    last edited by stevewest15

                    Hi,

                    Can anyone assist as to why my full backups are over 30 days old:
                    d7ff1037-c9d4-43d2-9aaa-3a69de3899ca-image.png

                    Here is my backup settings:
                    74ad1945-dfdb-4109-97f8-6db83aa3c475-image.png

                    I want to have a total of 7 backups per week (1 full plus 6 deltas).

                    A 1 Reply Last reply Reply Quote 0
                    • A Offline
                      Andrew Top contributor @stevewest15
                      last edited by

                      @stevewest15 If you are running XO from Source then you must update to the newest master. S3 backups just started working correctly for me on the newest version. If you are using XOA or another older tag version then it has old code for S3 and has issues with delta backups.

                      Second, your backup retention is only set to 3. If you want 7 restore points then you want to increase the retention.

                      With a delta backup you will always have at least 1 or 2 full backups saved. You will then have additional deltas between them to get a restore point.

                      The Full Backup interval does not determine how many full backups you keep but how often you have a full from which future deltas are based (and merged into).

                      Full+delta,,,,+delta = restore point.

                      The system will need a 2nd full backup as your deltas age and hit your Full backup interval and new deltas are based off of the new full backup. When all the old deltas are merged into the old full backup it will be removed and you will again only have one full backup plus deltas.

                      S 1 Reply Last reply Reply Quote 0
                      • S Offline
                        stevewest15 @Andrew
                        last edited by

                        @andrew Thanks Andrew! I just updated to latest commit and will run backup job again based on the following new settings (see below). I did remove the advanced settings of "Full backup interval" to blank so not sure if that matters or not when doing deltas:

                        eb0c0a84-f7dc-4bce-b904-57dfa830a74c-image.png

                        A 1 Reply Last reply Reply Quote 0
                        • A Offline
                          Andrew Top contributor @stevewest15
                          last edited by

                          @stevewest15 Only running one full backup the first time and then deltas going forward is creating a synthetic full backup (deltas merged into the original full backup). The problem with this is if there is an error in the program (coding) or a delta backup error (or S3 error) then your "full" backup will be corrupted going forward, for ever... until the next real full backup. So you'll still want to run a true full backup regularly to make sure that you don't have long term errors in the synthetic full backup.

                          Other companies (like Veem) use a single full copy for both forward and reverse deltas to save space. For now XO only uses forward deltas so there will be more than one full backup at times.

                          @florent Testing today's XO master update, S3 backup works faster than ever. As this feature is still "beta" it's changing every update (normally improving).

                          S 1 Reply Last reply Reply Quote 0
                          • S Offline
                            stevewest15 @Andrew
                            last edited by

                            @andrew thank you for the clarification! XO isn't very clear on that as I thought after the 7 backup retention, it will automatically run a new full back. So do you think doing this will work where to offer 1 full backup per week and then 6 deltas in between:

                            78beb6d8-7119-42bc-96cc-84c17c308ed9-image.png

                            S 1 Reply Last reply Reply Quote 0
                            • S Offline
                              stevewest15 @stevewest15
                              last edited by stevewest15

                              @olivierlambert, if an admin selects to delete backups, does XO delete these from the S3 bucket?

                              Last night, I selected to delete all backups from XO that are stored on b2 S3 bucket (about 50 backups). However when I checked b2 this morning, it seems nothing got deleted from the bucket:

                              2a4910de-2f4e-4271-9836-656da4a594b2-image.png

                              The above bucket contains only XO backups (for VMs and Pool/Config backups) so the bucket size should have reduced to only a few hundred MBs instead of still being over 10TB.

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                Dunno it's @florent you should ask 😉

                                1 Reply Last reply Reply Quote 1
                                • First post
                                  Last post