XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

    Scheduled Pinned Locked Moved Unsolved Backup
    105 Posts 20 Posters 12.1k Views 21 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • acebmxerA Online
      acebmxer @olivierlambert
      last edited by

      @olivierlambert

      Mine has not done it since last post but i have opened a support ticket along with tunnel being opened - Ticket#7762089

      1 Reply Last reply Reply Quote 1
      • J Online
        JB @olivierlambert
        last edited by

        @olivierlambert Yes, I can grant it.

        olivierlambertO 1 Reply Last reply Reply Quote 0
        • marcoiM Offline
          marcoi
          last edited by

          looks like mine also just started working.
          9093a86c-0ec3-48b8-ab41-236e86cf37f8-image.jpeg

          J 1 Reply Last reply Reply Quote 0
          • J Online
            JB
            last edited by

            6123a044-a5bd-4816-b937-6a096b26bc0a-image.jpeg

            Master, commit be960

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO @JB
              last edited by

              @JB I'd like to test a different way of investigating the problem. So I won't do it for many people, just send me a private message with a way to SSH to your host and XOA, promise it's not a scam 😆

              1 Reply Last reply Reply Quote 0
              • J Online
                JB @marcoi
                last edited by

                @marcoi Try doing it three times, and the problem comes back.

                marcoiM 1 Reply Last reply Reply Quote 0
                • marcoiM Offline
                  marcoi @JB
                  last edited by

                  @JB seems okay after a few back to back runs.
                  bd76af32-4767-42fe-b956-6270fde29325-image.jpeg

                  J 2 Replies Last reply Reply Quote 0
                  • J Online
                    JB @marcoi
                    last edited by

                    @marcoi Does your backup include METADATA and XO CONFIG?

                    marcoiM 1 Reply Last reply Reply Quote 0
                    • marcoiM Offline
                      marcoi @JB
                      last edited by marcoi

                      @JB both
                      Also Xen Orchestra, commit be960

                      ab69a1e2-43cd-4491-8a2f-b48765886b8c-image.jpeg

                      J 2 Replies Last reply Reply Quote 0
                      • J Online
                        JB @marcoi
                        last edited by

                        @marcoi The problem is with the METADATA backup. XO Config is working fine!

                        1 Reply Last reply Reply Quote 0
                        • J Online
                          JB @marcoi
                          last edited by

                          @marcoi So the problem is with my infrastructure.

                          1 Reply Last reply Reply Quote 0
                          • J Online
                            JB @marcoi
                            last edited by

                            @marcoi I see that you use NFS to store the backup. I use SMB.

                            1 Reply Last reply Reply Quote 0
                            • J Online
                              JB
                              last edited by

                              This post is deleted!
                              1 Reply Last reply Reply Quote 0
                              • J Online
                                JB
                                last edited by

                                070b68c3-bcf2-40ca-8667-ad41a9dd6f42-image.jpeg

                                1 Reply Last reply Reply Quote 0
                                • marcoiM Offline
                                  marcoi
                                  last edited by marcoi

                                  i switched one of the backups for my extra pool from nfs to smb and its been sitting so i expect it to fail. might be a smb issue. ill update this post with results once it either fails or works

                                  2d4d9b1d-9aa5-4f1d-a1c1-d8ea5e079047-image.jpeg

                                  i did test smb and it works.

                                  back to nfs works
                                  b2b807b8-d422-4719-8088-817b08d3c5ad-image.jpeg

                                  1 Reply Last reply Reply Quote 0
                                  • M Offline
                                    MajorP93 @olivierlambert
                                    last edited by

                                    @olivierlambert said:

                                    Hi! Can you try to replicate on the latest commit?

                                    Hello!
                                    Yes I was able to replicate this issue after updating my Xen Orchestra.

                                    188a7d72-1210-4077-b5ef-aa7ebe60ef6d-image.jpeg

                                    949e05f6-184d-4f11-9e53-23ddbd662150-image.jpeg

                                    The thing is that the "XO-config-and-pool-metadata" does not always fail.
                                    It fails from time to time then it looks exactly like shown in the screenshot.
                                    Attached logs please find above in my earlier post.

                                    This exact job has been working for a long time before.
                                    After some Xen Orchestra commit this issue got introduced.

                                    I use the same backup remotes for multiple backup jobs.
                                    Only the "XO-config-and-pool-metadata" job is failing with this body timeout error.

                                    Best regards

                                    P 1 Reply Last reply Reply Quote 0
                                    • P Online
                                      pierrebrunet Vates 🪐 XO Team @MajorP93
                                      last edited by

                                      @MajorP93 Hi,
                                      Thank you, thank you! I think you found it! It seems to be SMB, @acebmxer can you confirm that you also use smb please?
                                      If this is the case, the bad news is I think we will need to update our docs to not advise SMB at all: it was already said for delta but maybe we will have to extend this to all kind of BR 😕

                                      M acebmxerA 3 Replies Last reply Reply Quote 0
                                      • M Offline
                                        MajorP93 @pierrebrunet
                                        last edited by

                                        @pierrebrunet Hello, thank you for working on this issue.

                                        Are you completely sure that this isn't something that got introduced within Xen Orchestra code?

                                        Since I started using XCP-ng / Xen Orchestra I have been using these backup repositories via SMB.
                                        All backup jobs (including this one) have been working fine for almost a year.

                                        This repeated body timeout error is a rather new issue which is why I was wondering if this could have been introduced by XO code changes?

                                        I did not switch to a different linux kernel major version or similar. So the SMB client kernel module is at the same version number.

                                        Best regards

                                        1 Reply Last reply Reply Quote 0
                                        • acebmxerA Online
                                          acebmxer @pierrebrunet
                                          last edited by acebmxer

                                          @pierrebrunet

                                          I use nfs for my remotes and still get the error occasionally.

                                          Edit - Note i am only seeing this error in my work production environment. I am not seeing this issue in my home lab. Both use NFS for remote storage and vm storage.

                                          P 1 Reply Last reply Reply Quote 0
                                          • M Offline
                                            MajorP93 @pierrebrunet
                                            last edited by

                                            @pierrebrunet Oh one more thing that I just remembered.
                                            If I recall correctly the first time I saw this issue was right after I patched my XCP-ng hosts without updating Xen Orchestra.
                                            So maybe this is caused by XAPI code changes?

                                            1 Reply Last reply Reply Quote 0

                                            Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                            Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                            With your input, this post could be even better 💗

                                            Register Login
                                            • First post
                                              Last post