XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

    Scheduled Pinned Locked Moved Unsolved Backup
    81 Posts 19 Posters 6.5k Views 19 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J Offline
      JB
      last edited by

      6b58102f-353c-4da8-aed5-6817621e13e1-image.jpeg

      a4477b40-7812-4f27-8711-917c81c818da-image.jpeg

      commit 20bed

      SMB

      1 Reply Last reply Reply Quote 0
      • P Offline
        pierrebrunet Vates 🪐 XO Team @acebmxer
        last edited by

        @acebmxer ok, too bad... I thought we had it...
        @majorp93 this can be a lead. I will ask xapi team if they changed something in pool metadata export.

        acebmxerA 1 Reply Last reply Reply Quote 1
        • acebmxerA Offline
          acebmxer @pierrebrunet
          last edited by

          @pierrebrunet

          No worries,
          Mine seems to be intermittent and restarting that pools tool stack resolves the issue for now.

          J 1 Reply Last reply Reply Quote 0
          • J Offline
            JB @acebmxer
            last edited by

            @acebmxer

            514388c6-4dc9-4602-a72d-8ec157d492ab-image.jpeg

            Exactly! I restarted the pools tool stack and made 3 backups. If I restart it again, I'll be able to make more backups.

            acebmxerA 1 Reply Last reply Reply Quote 0
            • G Offline
              GregBinSD
              last edited by

              @jb
              You have hit on something that ties into what I have discovered. Since July 1, on my XCP-ng / XO-CE cluster, the metadata and xo-config backup job only runs successfully when I have all 3 hosts in my cluster active, and after I have reset the tool stack on the Master host.

              Because I am running the XCP-ng and XO-CE on an off-grid solar/battery system, independent of my electric bill, I try to limit total power to 250 watts for all devices (network switches, iSCSI NAS, XCP-ng hosts, etc). Consequently, I run only one host on a regular basis. When it is necessary to update XCP-ng, I turn on all hosts and do a rolling pool update. If I manually perform a metadata and xo-config backup while all of the hosts are on, only then will the backup be successful.

              Before July 1, the metadata and xo-config back was always successful, whether it was scheduled or manual.

              Hopefully, this will turn on a light bulb in the head of a Vates engineer, and he or she will solve this issue!

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Online
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Are you sure it really worked? Because we fixed something that swallowed the error before. Maybe it was successful while in fact not working. When you have some hosts down in a pool, you cannot safely export metadata because XAPI can't tell if the other hosts are dead, disconnected or simply stopped.

                1 Reply Last reply Reply Quote 0
                • acebmxerA Offline
                  acebmxer @JB
                  last edited by

                  @JB @gregbinsd @olivierlambert

                  Yes restarting the tool stack works for me. I currently do not have any host offline at least not intentionally. If there is a network glitch that i am not aware of that might cause the issue if that is the actual cause.

                  As @olivierlambert stated maybe we were having issues all along and they just changed something that is making the issue visible now. The support tunnel should still be open on my environment if you or the team want to look at logs.

                  1 Reply Last reply Reply Quote 0
                  • G Offline
                    GregBinSD
                    last edited by

                    @jb
                    @olivierlambert
                    Oliver, you may be right, wondering whether they really worked. How can I prove that the metadata and xo-config backups were truly successful?

                    Below, I pasted in the GUI backup log from XO5.

                    425ae32e-ada6-4498-af93-eff3c5bb451a-image.jpeg

                    1 Reply Last reply Reply Quote 0
                    • J Offline
                      JB
                      last edited by

                      All my hosts are online, yet the error still occurs. However, if I restart, the backup works for two or three runs.

                      1 Reply Last reply Reply Quote 0
                      • bogikornelB Offline
                        bogikornel
                        last edited by

                        The problem is that my VM backups don't run either, because I still get a timeout. This morning out of 11 VM backups, 2 failed, on Tuesday 1 failed, 20 were good.

                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO Online
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          5 minutes is clearly an HTTP timeout, the question is way we are reaching this timeout in the first place.

                          1 Reply Last reply Reply Quote 0

                          Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                          Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                          With your input, this post could be even better 💗

                          Register Login
                          • First post
                            Last post