XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

    Scheduled Pinned Locked Moved Unsolved Backup
    105 Posts 20 Posters 12.1k Views 21 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • marcoiM Offline
      marcoi
      last edited by

      looks like mine also just started working.
      9093a86c-0ec3-48b8-ab41-236e86cf37f8-image.jpeg

      J 1 Reply Last reply Reply Quote 0
      • J Offline
        JB
        last edited by

        6123a044-a5bd-4816-b937-6a096b26bc0a-image.jpeg

        Master, commit be960

        1 Reply Last reply Reply Quote 0
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO @JB
          last edited by

          @JB I'd like to test a different way of investigating the problem. So I won't do it for many people, just send me a private message with a way to SSH to your host and XOA, promise it's not a scam 😆

          1 Reply Last reply Reply Quote 0
          • J Offline
            JB @marcoi
            last edited by

            @marcoi Try doing it three times, and the problem comes back.

            marcoiM 1 Reply Last reply Reply Quote 0
            • marcoiM Offline
              marcoi @JB
              last edited by

              @JB seems okay after a few back to back runs.
              bd76af32-4767-42fe-b956-6270fde29325-image.jpeg

              J 2 Replies Last reply Reply Quote 0
              • J Offline
                JB @marcoi
                last edited by

                @marcoi Does your backup include METADATA and XO CONFIG?

                marcoiM 1 Reply Last reply Reply Quote 0
                • marcoiM Offline
                  marcoi @JB
                  last edited by marcoi

                  @JB both
                  Also Xen Orchestra, commit be960

                  ab69a1e2-43cd-4491-8a2f-b48765886b8c-image.jpeg

                  J 2 Replies Last reply Reply Quote 0
                  • J Offline
                    JB @marcoi
                    last edited by

                    @marcoi The problem is with the METADATA backup. XO Config is working fine!

                    1 Reply Last reply Reply Quote 0
                    • J Offline
                      JB @marcoi
                      last edited by

                      @marcoi So the problem is with my infrastructure.

                      1 Reply Last reply Reply Quote 0
                      • J Offline
                        JB @marcoi
                        last edited by

                        @marcoi I see that you use NFS to store the backup. I use SMB.

                        1 Reply Last reply Reply Quote 0
                        • J Offline
                          JB
                          last edited by

                          This post is deleted!
                          1 Reply Last reply Reply Quote 0
                          • J Offline
                            JB
                            last edited by

                            070b68c3-bcf2-40ca-8667-ad41a9dd6f42-image.jpeg

                            1 Reply Last reply Reply Quote 0
                            • marcoiM Offline
                              marcoi
                              last edited by marcoi

                              i switched one of the backups for my extra pool from nfs to smb and its been sitting so i expect it to fail. might be a smb issue. ill update this post with results once it either fails or works

                              2d4d9b1d-9aa5-4f1d-a1c1-d8ea5e079047-image.jpeg

                              i did test smb and it works.

                              back to nfs works
                              b2b807b8-d422-4719-8088-817b08d3c5ad-image.jpeg

                              1 Reply Last reply Reply Quote 0
                              • M Online
                                MajorP93 @olivierlambert
                                last edited by

                                @olivierlambert said:

                                Hi! Can you try to replicate on the latest commit?

                                Hello!
                                Yes I was able to replicate this issue after updating my Xen Orchestra.

                                188a7d72-1210-4077-b5ef-aa7ebe60ef6d-image.jpeg

                                949e05f6-184d-4f11-9e53-23ddbd662150-image.jpeg

                                The thing is that the "XO-config-and-pool-metadata" does not always fail.
                                It fails from time to time then it looks exactly like shown in the screenshot.
                                Attached logs please find above in my earlier post.

                                This exact job has been working for a long time before.
                                After some Xen Orchestra commit this issue got introduced.

                                I use the same backup remotes for multiple backup jobs.
                                Only the "XO-config-and-pool-metadata" job is failing with this body timeout error.

                                Best regards

                                P 1 Reply Last reply Reply Quote 0
                                • P Offline
                                  pierrebrunet Vates 🪐 XO Team @MajorP93
                                  last edited by

                                  @MajorP93 Hi,
                                  Thank you, thank you! I think you found it! It seems to be SMB, @acebmxer can you confirm that you also use smb please?
                                  If this is the case, the bad news is I think we will need to update our docs to not advise SMB at all: it was already said for delta but maybe we will have to extend this to all kind of BR 😕

                                  M acebmxerA 3 Replies Last reply Reply Quote 0
                                  • M Online
                                    MajorP93 @pierrebrunet
                                    last edited by

                                    @pierrebrunet Hello, thank you for working on this issue.

                                    Are you completely sure that this isn't something that got introduced within Xen Orchestra code?

                                    Since I started using XCP-ng / Xen Orchestra I have been using these backup repositories via SMB.
                                    All backup jobs (including this one) have been working fine for almost a year.

                                    This repeated body timeout error is a rather new issue which is why I was wondering if this could have been introduced by XO code changes?

                                    I did not switch to a different linux kernel major version or similar. So the SMB client kernel module is at the same version number.

                                    Best regards

                                    1 Reply Last reply Reply Quote 0
                                    • acebmxerA Online
                                      acebmxer @pierrebrunet
                                      last edited by acebmxer

                                      @pierrebrunet

                                      I use nfs for my remotes and still get the error occasionally.

                                      Edit - Note i am only seeing this error in my work production environment. I am not seeing this issue in my home lab. Both use NFS for remote storage and vm storage.

                                      P 1 Reply Last reply Reply Quote 0
                                      • M Online
                                        MajorP93 @pierrebrunet
                                        last edited by

                                        @pierrebrunet Oh one more thing that I just remembered.
                                        If I recall correctly the first time I saw this issue was right after I patched my XCP-ng hosts without updating Xen Orchestra.
                                        So maybe this is caused by XAPI code changes?

                                        1 Reply Last reply Reply Quote 0
                                        • J Offline
                                          JB
                                          last edited by

                                          6b58102f-353c-4da8-aed5-6817621e13e1-image.jpeg

                                          a4477b40-7812-4f27-8711-917c81c818da-image.jpeg

                                          commit 20bed

                                          SMB

                                          1 Reply Last reply Reply Quote 0
                                          • P Offline
                                            pierrebrunet Vates 🪐 XO Team @acebmxer
                                            last edited by

                                            @acebmxer ok, too bad... I thought we had it...
                                            @majorp93 this can be a lead. I will ask xapi team if they changed something in pool metadata export.

                                            acebmxerA 1 Reply Last reply Reply Quote 1

                                            Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                            Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                            With your input, this post could be even better 💗

                                            Register Login
                                            • First post
                                              Last post