XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup started to fail

    Scheduled Pinned Locked Moved Xen Orchestra
    17 Posts 6 Posters 977 Views 2 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • julien-fJ Offline
      julien-f Vates 🪐 Co-Founder XO Team @fred974
      last edited by

      @fred974 Please show me the full log (I need the stacktrace of this error), there is a button at the top of the log modal.

      F 1 Reply Last reply Reply Quote 0
      • F Offline
        fred974 @julien-f
        last edited by

        @julien-f Find bellow the the full log file.
        2022-02-07T00_00_00.026Z - backup NG.txt

        Thank you 🙂

        julien-fJ 1 Reply Last reply Reply Quote 0
        • julien-fJ Offline
          julien-f Vates 🪐 Co-Founder XO Team @fred974
          last edited by

          @fred974 Thank you. Unfortunately there isn't much more info, can you take a look at your remote and tell me how many files you have in /xo-config-backups/4866b683-59a1-42ba-93d8-b704b5bbbbe6 and what do they look like?

          F 1 Reply Last reply Reply Quote 0
          • F Offline
            fred974 @julien-f
            last edited by

            @julien-f this is what I see inside /xo-config-backups/4866b683-59a1-42ba-93d8-b704b5bbbbe6

            d34f5659-babf-4cb9-afa6-7f7e057bd04b-image.png

            There are 48 folders in there and here the content of the last folder.

            a2456743-019e-4f24-ac85-100071c8cc18-image.png

            Thank you

            F 1 Reply Last reply Reply Quote 0
            • F Offline
              fred974 @fred974
              last edited by

              According to backblaze support, there is over 5 million files in the bucket. Is that not too much for my setting of daily backup?

              julien-fJ 1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                5M blocks of 2MiB, means near 10TiB of data. Is that the expected volume of your backups?

                1 Reply Last reply Reply Quote 0
                • julien-fJ Offline
                  julien-f Vates 🪐 Co-Founder XO Team @fred974
                  last edited by

                  @fred974 What's important for XO is that there is no more than 1000 files per directories.

                  If this condition is respected, then maybe it's an issue between our implementation and Blackblaze (particularly Delimiter support for ListObjectsV2).

                  Our implementation has been tested against AWS S3 and MinIO.

                  1 Reply Last reply Reply Quote 0
                  • florentF Offline
                    florent Vates 🪐 XO Team
                    last edited by

                    Hi fred,

                    I looked into your problem and did not reproduced it on our test lab. Have you got a xo support subscription ? That would allow us to open a support tunnel and search a solution more efficiently

                    Regards

                    Florent

                    F 1 Reply Last reply Reply Quote 0
                    • F Offline
                      fred974 @florent
                      last edited by

                      @florent said in Backup started to fail:

                      Have you got a xo support subscription  ?

                      No I don't. We use the community edition alonside XOA..

                      S 1 Reply Last reply Reply Quote 0
                      • S Offline
                        stevewest15 @fred974
                        last edited by stevewest15

                        XO 5.67.0

                        Hi,

                        @florent @julien-f I just updated XO via the web browser from an older version to the latest version 5.67.0 and now I'm getting the same issue as @fred974 :

                        Mar 07 08:20:29 xoa xo-server[13189]:       at S3Handler._list (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/src/s3.js:199:21)
Mar 07 08:20:29 xoa xo-server[13189]:       at runMicrotasks (<anonymous>)
Mar 07 08:20:29 xoa xo-server[13189]:       at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Mar 07 08:20:29 xoa xo-server[13189]:     dir: '/xo-vm-backups'
Mar 07 08:20:29 xoa xo-server[13189]:   }
Mar 07 08:20:29 xoa xo-server[13189]: }
Mar 07 08:34:12 xoa xo-server[13189]: 2022-03-07T13:34:12.720Z xo:xo-mixins:backups-ng WARN listVmBackups for remote 53745311-c2b2-41d9-bf98-67d9789b16a3: {
Mar 07 08:34:12 xoa xo-server[13189]:   error: Error: more than 1000 objects, unsupported in this implementation
Mar 07 08:34:12 xoa xo-server[13189]:       at S3Handler._list (/usr/local/lib/node_modules/xo-server/node_modules/@xen-orchestra/fs/src/s3.js:199:21)
Mar 07 08:34:12 xoa xo-server[13189]:       at runMicrotasks (<anonymous>)
Mar 07 08:34:12 xoa xo-server[13189]:       at processTicksAndRejections (node:internal/process/task_queues:96:5) {
Mar 07 08:34:12 xoa xo-server[13189]:     dir: '/xo-vm-backups'
Mar 07 08:34:12 xoa xo-server[13189]:   }
Mar 07 08:34:12 xoa xo-server[13189]: }

                        7af56e44-41b2-4819-8e02-185b0b69117e-image.png

                        Backblaze Bucket:
                        4bdedcbf-cddf-43e0-a19c-9c669598f167-image.png

                        After doing the XO update, I can't see the list of backups when I go to Backup > Restore:
                        22bd4928-de73-435e-bb7c-ad1b2732467f-image.png

                        Prior to updating XO to the latest version, I was able to see the backups on Backblaze when going to Backup > Restore.

                        Here is the prior version of XOA installed which didn't have this issue:

                        Mar  7 06:52:13 xoa xoa-updater[479]: stable channel selected
Mar  7 06:52:13 xoa xoa-updater[479]: Following packages will be removed:
Mar  7 06:52:13 xoa xoa-updater[479]:   xen-orchestra-upload-ova 0.1.4
Mar  7 06:52:13 xoa xoa-updater[479]:   xen-orchestra 5.65.3
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server-telemetry 0.5.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server-xoa 0.12.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server 5.85.1
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-web-free 5.90.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-cli 0.28.0
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-updater 0.39.0
Mar  7 06:52:13 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:13 xoa xoa-updater[479]:   node 16.13.2 (14.17.5 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   npm 8.4.0 (6.14.13 installed)
Mar  7 06:52:13 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-updater 0.39.0 (0.39.0 installed)
Mar  7 06:52:13 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:13 xoa xoa-updater[479]:   xen-orchestra 5.67.0 (5.65.3 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-server 5.87.0 (5.85.1 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   xo-web-free 5.92.0 (5.90.0 installed)
Mar  7 06:52:13 xoa xoa-updater[479]:   xoa-cli 0.29.0 (0.28.0 installed)
Mar  7 06:52:16 xoa xoa-updater[479]: stable channel selected
Mar  7 06:52:16 xoa xoa-updater[479]: Following packages will be removed:
Mar  7 06:52:16 xoa xoa-updater[479]:   xen-orchestra-upload-ova 0.1.4
Mar  7 06:52:16 xoa xoa-updater[479]:   xen-orchestra 5.65.3
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server-telemetry 0.5.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server-xoa 0.12.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server 5.85.1
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-web-free 5.90.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-cli 0.28.0
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-updater 0.39.0
Mar  7 06:52:16 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:16 xoa xoa-updater[479]:   node 16.13.2 (14.17.5 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   npm 8.4.0 (6.14.13 installed)
Mar  7 06:52:16 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-updater 0.39.0 (0.39.0 installed)
Mar  7 06:52:16 xoa xoa-updater[479]: New versions available:
Mar  7 06:52:16 xoa xoa-updater[479]:   xen-orchestra 5.67.0 (5.65.3 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-server 5.87.0 (5.85.1 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   xo-web-free 5.92.0 (5.90.0 installed)
Mar  7 06:52:16 xoa xoa-updater[479]:   xoa-cli 0.29.0 (0.28.0 installed)
                        florentF 1 Reply Last reply Reply Quote 0
                        • florentF Offline
                          florent Vates 🪐 XO Team @stevewest15
                          last edited by

                          hi @stevewest15 , can you create a support ticket on https://help.vates.fr/ ? Then we will open a tunnel support to look into this in depth

                          Regards,
                          Florent

                          S 1 Reply Last reply Reply Quote 0
                          • S Offline
                            stevewest15 @florent
                            last edited by

                            @florent Thank you but can't seem to login to the support ticket system:

                            Login failed. Have you double-checked your credentials and completed the email verification step?

                            Not sure how to get the email verification or to reset password as I don't see that option on the login page.

                            DarkbeldinD 1 Reply Last reply Reply Quote 0
                            • DarkbeldinD Offline
                              Darkbeldin Vates 🪐 Pro Support Team @stevewest15
                              last edited by

                              @stevewest15
                              Hi,

                              It's the same login you used just before on the website, it's not a new account you just have to confirm your password.

                              S 1 Reply Last reply Reply Quote 0
                              • S Offline
                                stevewest15 @Darkbeldin
                                last edited by

                                @Darkbeldin Thank you! I was able to get in after 1st logging to the xen-orchestra.com website, updating my profile, and then clicking on support from that portal.

                                Here is the Ticket#775709.

                                Look forward to your update!

                                Best Regards,

                                SW

                                DarkbeldinD 1 Reply Last reply Reply Quote 0
                                • DarkbeldinD Offline
                                  Darkbeldin Vates 🪐 Pro Support Team @stevewest15
                                  last edited by

                                  @stevewest15 Happy to help 😉

                                  1 Reply Last reply Reply Quote 0
                                  • First post
                                    Last post