XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup started to fail

    Scheduled Pinned Locked Moved Xen Orchestra
    17 Posts 6 Posters 977 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F Offline
      fred974 @julien-f
      last edited by

      @julien-f Find bellow the the full log file.
      2022-02-07T00_00_00.026Z - backup NG.txt

      Thank you 🙂

      julien-fJ 1 Reply Last reply Reply Quote 0
      • julien-fJ Offline
        julien-f Vates 🪐 Co-Founder XO Team @fred974
        last edited by

        @fred974 Thank you. Unfortunately there isn't much more info, can you take a look at your remote and tell me how many files you have in /xo-config-backups/4866b683-59a1-42ba-93d8-b704b5bbbbe6 and what do they look like?

        F 1 Reply Last reply Reply Quote 0
        • F Offline
          fred974 @julien-f
          last edited by

          @julien-f this is what I see inside /xo-config-backups/4866b683-59a1-42ba-93d8-b704b5bbbbe6

          d34f5659-babf-4cb9-afa6-7f7e057bd04b-image.png

          There are 48 folders in there and here the content of the last folder.

          a2456743-019e-4f24-ac85-100071c8cc18-image.png

          Thank you

          F 1 Reply Last reply Reply Quote 0
          • F Offline
            fred974 @fred974
            last edited by

            According to backblaze support, there is over 5 million files in the bucket. Is that not too much for my setting of daily backup?

            julien-fJ 1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              5M blocks of 2MiB, means near 10TiB of data. Is that the expected volume of your backups?

              1 Reply Last reply Reply Quote 0
              • julien-fJ Offline
                julien-f Vates 🪐 Co-Founder XO Team @fred974
                last edited by

                @fred974 What's important for XO is that there is no more than 1000 files per directories.

                If this condition is respected, then maybe it's an issue between our implementation and Blackblaze (particularly Delimiter support for ListObjectsV2).

                Our implementation has been tested against AWS S3 and MinIO.

                1 Reply Last reply Reply Quote 0
                • florentF Offline
                  florent Vates 🪐 XO Team
                  last edited by

                  Hi fred,

                  I looked into your problem and did not reproduced it on our test lab. Have you got a xo support subscription ? That would allow us to open a support tunnel and search a solution more efficiently

                  Regards

                  Florent

                  F 1 Reply Last reply Reply Quote 0
                  • F Offline
                    fred974 @florent
                    last edited by

                    @florent said in Backup started to fail:

                    Have you got a xo support subscription  ?

                    No I don't. We use the community edition alonside XOA..

                    S 1 Reply Last reply Reply Quote 0
                    • S Offline
                      stevewest15 @fred974
                      last edited by stevewest15

                      XO 5.67.0

                      Hi,

                      @florent @julien-f I just updated XO via the web browser from an older version to the latest version 5.67.0 and now I'm getting the same issue as @fred974 :

                      Mar 07 08:20:29 xoa xo-server[13189]:       at S3Handler._list (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/src/s3.js:199:21)
Mar 07 08:20:29 xoa xo-server[13189]:       at runMicrotasks (<anonymous>)
Mar 07 08:20:29 xoa xo-server[13189]:       at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Mar 07 08:20:29 xoa xo-server[13189]:     dir: '/xo-vm-backups'
Mar 07 08:20:29 xoa xo-server[13189]:   }
Mar 07 08:20:29 xoa xo-server[13189]: }
Mar 07 08:34:12 xoa xo-server[13189]: 2022-03-07T13:34:12.720Z xo:xo-mixins:backups-ng WARN listVmBackups for remote 53745311-c2b2-41d9-bf98-67d9789b16a3: {
Mar 07 08:34:12 xoa xo-server[13189]:   error: Error: more than 1000 objects, unsupported in this implementation
Mar 07 08:34:12 xoa xo-server[13189]:       at S3Handler._list (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/src/s3.js:199:21)
Mar 07 08:34:12 xoa xo-server[13189]:       at runMicrotasks (<anonymous>)
Mar 07 08:34:12 xoa xo-server[13189]:       at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Mar 07 08:34:12 xoa xo-server[13189]:     dir: '/xo-vm-backups'
Mar 07 08:34:12 xoa xo-server[13189]:   }
Mar 07 08:34:12 xoa xo-server[13189]: }

                      7af56e44-41b2-4819-8e02-185b0b69117e-image.png

                      Backblaze Bucket:
                      4bdedcbf-cddf-43e0-a19c-9c669598f167-image.png

                      After doing the XO update, I can't see the list of backups when I go to Backup > Restore:
                      22bd4928-de73-435e-bb7c-ad1b2732467f-image.png

                      Prior to updating XO to the latest version, I was able to see the backups on Backblaze when going to Backup > Restore.

                      Here is the prior version of XOA installed which didn't have this issue:

                      Mar  7 06:52:13 xoa xoa-updater[479]: stable channel selected
Mar  7 06:52:13 xoa xoa-updater[479]: Following packages will be removed:
Mar  7 06:52:13 xoa xoa-updater[479]:   xen-orchestra-upload-ova 0.1.4
Mar  7 06:52:13 xoa xoa-updater[479]:   xen-orchestra 5.65.3
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server-telemetry 0.5.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server-xoa 0.12.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server 5.85.1
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-web-free 5.90.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-cli 0.28.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-updater 0.39.0
Mar  7 06:52:13 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:13 xoa xoa-updater[479]:   node 16.13.2 (14.17.5 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   npm 8.4.0 (6.14.13 installed)
Mar  7 06:52:13 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-updater 0.39.0 (0.39.0 installed)
Mar  7 06:52:13 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:13 xoa xoa-updater[479]:   xen-orchestra 5.67.0 (5.65.3 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server 5.87.0 (5.85.1 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-web-free 5.92.0 (5.90.0 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-cli 0.29.0 (0.28.0 installed)
Mar  7 06:52:16 xoa xoa-updater[479]: stable channel selected
Mar  7 06:52:16 xoa xoa-updater[479]: Following packages will be removed:
Mar  7 06:52:16 xoa xoa-updater[479]:   xen-orchestra-upload-ova 0.1.4
Mar  7 06:52:16 xoa xoa-updater[479]:   xen-orchestra 5.65.3
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server-telemetry 0.5.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server-xoa 0.12.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server 5.85.1
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-web-free 5.90.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-cli 0.28.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-updater 0.39.0
Mar  7 06:52:16 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:16 xoa xoa-updater[479]:   node 16.13.2 (14.17.5 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   npm 8.4.0 (6.14.13 installed)
Mar  7 06:52:16 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-updater 0.39.0 (0.39.0 installed)
Mar  7 06:52:16 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:16 xoa xoa-updater[479]:   xen-orchestra 5.67.0 (5.65.3 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server 5.87.0 (5.85.1 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-web-free 5.92.0 (5.90.0 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-cli 0.29.0 (0.28.0 installed)
                      florentF 1 Reply Last reply Reply Quote 0
                      • florentF Offline
                        florent Vates 🪐 XO Team @stevewest15
                        last edited by

                        hi @stevewest15 , can you create a support ticket on https://help.vates.fr/ ? Then we will open a tunnel support to look into this in depth

                        Regards,
                        Florent

                        S 1 Reply Last reply Reply Quote 0
                        • S Offline
                          stevewest15 @florent
                          last edited by

                          @florent Thank you but can't seem to login to the support ticket system:

                          Login failed. Have you double-checked your credentials and completed the email verification step?

                          Not sure how to get the email verification or to reset password as I don't see that option on the login page.

                          DarkbeldinD 1 Reply Last reply Reply Quote 0
                          • DarkbeldinD Offline
                            Darkbeldin Vates 🪐 Pro Support Team @stevewest15
                            last edited by

                            @stevewest15
                            Hi,

                            It's the same login you used just before on the website, it's not a new account you just have to confirm your password.

                            S 1 Reply Last reply Reply Quote 0
                            • S Offline
                              stevewest15 @Darkbeldin
                              last edited by

                              @Darkbeldin Thank you! I was able to get in after 1st logging to the xen-orchestra.com website, updating my profile, and then clicking on support from that portal.

                              Here is the Ticket#775709.

                              Look forward to your update!

                              Best Regards,

                              SW

                              DarkbeldinD 1 Reply Last reply Reply Quote 0
                              • DarkbeldinD Offline
                                Darkbeldin Vates 🪐 Pro Support Team @stevewest15
                                last edited by

                                @stevewest15 Happy to help 😉

                                1 Reply Last reply Reply Quote 0
                                • First post
                                  Last post