XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

    Scheduled Pinned Locked Moved Unsolved Backup
    94 Posts 19 Posters 10.4k Views 19 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P Offline
      pierrebrunet Vates 🪐 XO Team @bogikornel
      last edited by

      @bogikornel Can you tell us on which versions you are please?
      the ones where it works and the ones where it does not work.
      Thanks for your help!

      bogikornelB 1 Reply Last reply Reply Quote 0
      • bogikornelB Offline
        bogikornel @pierrebrunet
        last edited by

        @pierrebrunet said:

        @bogikornel Can you tell us on which versions you are please?
        the ones where it works and the ones where it does not work.
        Thanks for your help!

        Not working: commit bb4da build date: 2026.07.01
        Not working: commit 3bc70 build date: 2026.07.02
        Working:: commit 71fa8 build date: ~ 2025.11.29
        Working: commit 96b76 build date: ~ 2024.07.22

        P 1 Reply Last reply Reply Quote 0
        • P Offline
          pierrebrunet Vates 🪐 XO Team @bogikornel
          last edited by

          @bogikornel @jb @gregbinsd @majorp93
          We tested metadata backups every 5 min and it works in XO. It seems it might be coming from XCP-NG, can you provide us a xensource.log from the problematic timeline?

          bogikornelB 1 Reply Last reply Reply Quote 0
          • bogikornelB Offline
            bogikornel @pierrebrunet
            last edited by

            @pierrebrunet I'll try to collect the logs.
            However, I can't even backup VMs. What's interesting is that the error always occurs with only 1 VM or 1 Metadata backup. This morning's backup report:
            Job ID: 3c17f0c1-85f5-4b98-b43a-0d117c812a9a
            Run ID: 1783461900005
            Mode: full
            Start time: Wednesday, July 8th 2026, 12:05:00 am
            End time: Wednesday, July 8th 2026, 1:22:44 am
            Duration: an hour
            Successes: 14 / 15
            Transfer size: 276.37 GiB

            Or with another host and another XO:
            Job ID: 53c582e5-ba01-4a82-8a45-948a991add99
            Job name: metadata
            Run ID: 1783483200004
            Start time: Wednesday, July 8th 2026, 6:00:00 am
            End time: Wednesday, July 8th 2026, 6:05:00 am
            Duration: 5 minutes
            Successes: 5 / 6
            Error: backup task failed with undefined error

            poddingueP 1 Reply Last reply Reply Quote 0
            • poddingueP Online
              poddingue Vates 🪐 @bogikornel
              last edited by

              Thanks @bogikornel, that version comparison really helps.
              Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out". 👍

              On the log Pierre asked for: the useful one is /var/log/xensource.log from the pool master, covering one failed run's window.
              You already have the Job and Run IDs and the 00:05 to 01:22 timestamps from this morning's run, so that exact span is perfect.
              If you can pull it from the master and attach it here (or just the slice around the timeout), that's what lets them line the failure up against what xapi was doing.
              It's looking like it may be coming from the XCP-ng side rather than XO, so a mention to @Team-Storage might help too. I could be wrong about where it actually lands, but the log is the thing that'll tell us.

              bogikornelB 1 Reply Last reply Reply Quote 0
              • poddingueP poddingue referenced this topic on
              • bogikornelB Offline
                bogikornel @poddingue
                last edited by

                @poddingue said:

                Narrowing it to something that changed between the late-November 2025 build and the July 2026 ones gives everyone a better place to start than "it just times out".

                The backup function was definitely still good in the 2026-05-28 build. So I think you should look at the last 1 month.

                poddingueP 1 Reply Last reply Reply Quote 0
                • poddingueP Online
                  poddingue Vates 🪐 @bogikornel
                  last edited by

                  The 2026-05-28 build still being good narrows the window a lot more than "sometime since November". 👍
                  What I keep coming back to is the shape of the failure: 14 out of 15, then 5 out of 6, so it is always exactly one that falls over and never the whole run. I don't know whether that points at a per-VM timeout or at something the last task in a run does differently, and someone on the XO team will read that better than me. 🤔
                  @pierrebrunet still needs /var/log/xensource.log from the pool master covering one failed run's window, so even a slice from a job where only one VM failed should be enough.

                  1 Reply Last reply Reply Quote 0
                  • J Offline
                    JB
                    last edited by

                    Any solution?

                    1 Reply Last reply Reply Quote 0
                    • christopher-petzelC Offline
                      christopher-petzel
                      last edited by

                      @poddingue Maybe this log will help @pierrebrunet . I've been having the Body Timeout Error for a couple of weeks on Metadata/Config backups. I'm using XO from sources. The problem started after upgrading to commit 63f8d. I was previously at commit e6443. The errors will occur for one or more hosts, and which host(s) has the error seems to be random. The error will occur for hosts which have VMs and for hosts that have no VMs. Attached is this morning's xensource.log from 00:10 when the backup started. The backup ends at 00:15 but I've included log data through 00:20. xensource-truncated.log.txt

                      I have reverted to a snapshot of XO running at commit e6443 and executed multiple Metadata/Config backups without any problem.

                      J 1 Reply Last reply Reply Quote 1
                      • J Offline
                        JB @christopher-petzel
                        last edited by

                        @christopher-petzel Thanks!

                        1 Reply Last reply Reply Quote 0
                        • M Online
                          MajorP93
                          last edited by

                          Hello,

                          the issue is back on my end.
                          Pool and metadata backup failed on all remotes due to "body timeout error".

                          dc6631af-5178-4347-9cb9-e9ef388b3606-image.jpeg

                          Attached please find full backup log:
                          https://paste.vates.tech/?f781d8cb757e56b1#8MN4cyjHhkWKzhhvvggckf8mqjT9gxDY6t4vkSVQcWUY

                          I can confirm what @christopher-petzel said. Before commit 63f8d got pushed this issue did not occur.

                          I am currently running commit 1d82c (feat: release 6.6.2).

                          Best regards

                          1 Reply Last reply Reply Quote 1
                          • olivierlambertO Online
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            Hi! Can you try to replicate on the latest commit?

                            J M 2 Replies Last reply Reply Quote 0
                            • J Offline
                              JB @olivierlambert
                              last edited by

                              @olivierlambert commit 59306backup-metadados.jpg

                              florentF 1 Reply Last reply Reply Quote 1
                              • florentF Offline
                                florent Vates 🪐 XO Team @JB
                                last edited by

                                @JB this looks like an issue while reading the export part. Did you configure a default backup network in the pool ?
                                is there any firewall/sdn controller that can filter out some reading ?

                                note that we have removed several point that where hiding some errors this month.

                                J 2 Replies Last reply Reply Quote 1
                                • J Offline
                                  JB @florent
                                  last edited by

                                  @florent Between one error and another, the backup sometimes succeeds.backup-metadados2.jpg

                                  1 Reply Last reply Reply Quote 1
                                  • J Offline
                                    JB @florent
                                    last edited by

                                    @florent backup-metadados3.jpg

                                    florentF 1 Reply Last reply Reply Quote 1
                                    • poddingueP poddingue marked this topic as a question
                                    • florentF Offline
                                      florent Vates 🪐 XO Team @JB
                                      last edited by

                                      @JB this really looks like a network issue

                                      are the host/xoa remotes ?

                                      J 1 Reply Last reply Reply Quote 0
                                      • J Offline
                                        JB @florent
                                        last edited by

                                        @florent hi! Hosts/XOA on the same 192.168.0.x network.

                                        J 1 Reply Last reply Reply Quote 0
                                        • J Offline
                                          JB @JB
                                          last edited by

                                          backup-metadados4.jpg

                                          1 Reply Last reply Reply Quote 0
                                          • A Offline
                                            abudef
                                            last edited by

                                            Hi,
                                            for several weeks, since 23 June 2026 (see below), I have been experiencing the same issue. XO (from source; daily updated) is running as a VM on the host (updated on the day or day + 1 the patches were released) being backed up.

                                            result	
name	"BodyTimeoutError"
code	"UND_ERR_BODY_TIMEOUT"
message	"Body Timeout Error"
stack	"BodyTimeoutError: Body Timeout Error\n    at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607312004/node_modules/undici/lib/dispatcher/client-h1.js:688:28)\n    at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202607312004/node_modules/undici/lib/util/timers.js:162:13)\n    at listOnTimeout (node:internal/timers:605:17)\n    at processTimers (node:internal/timers:541:7)"

                                            Configuration	2026-07-31 20:22	2026-07-31 20:22	a few seconds		
Configuration	2026-07-31 20:16	2026-07-31 20:21	5 minutes		
Configuration	2026-07-31 20:09	2026-07-31 20:14	5 minutes		
Configuration	2026-07-31 19:50	2026-07-31 19:55	5 minutes		
Configuration	2026-07-30 20:53	2026-07-30 20:53	a few seconds		
Configuration	2026-07-30 19:50	2026-07-30 19:50	a few seconds		
Configuration	2026-07-29 19:50	2026-07-29 19:50	a few seconds		
Configuration	2026-07-28 20:05	2026-07-28 20:05	a few seconds		
Configuration	2026-07-28 19:50	2026-07-28 19:55	5 minutes		
Configuration	2026-07-27 19:50	2026-07-27 19:50	a few seconds		
Configuration	2026-07-26 20:49	2026-07-26 20:49	a few seconds		
Configuration	2026-07-26 19:50	2026-07-26 19:55	5 minutes		
Configuration	2026-07-25 19:50	2026-07-25 19:50	a few seconds		
Configuration	2026-07-24 19:50	2026-07-24 19:50	a few seconds		
Configuration	2026-07-23 20:48	2026-07-23 20:48	a few seconds		
Configuration	2026-07-23 19:50	2026-07-23 19:55	5 minutes		
Configuration	2026-07-22 19:50	2026-07-22 19:50	a few seconds		
Configuration	2026-07-21 19:50	2026-07-21 19:50	a few seconds		
Configuration	2026-07-20 19:50	2026-07-20 19:50	a few seconds		
Configuration	2026-07-19 19:50	2026-07-19 19:50	a few seconds		
Configuration	2026-07-19 07:47	2026-07-19 07:47	a few seconds		
Configuration	2026-07-18 19:50	2026-07-18 19:50	a few seconds		
Configuration	2026-07-17 19:50	2026-07-17 19:50	a few seconds		
Configuration	2026-07-16 19:50	2026-07-16 19:50	a few seconds		
Configuration	2026-07-15 19:50	2026-07-15 19:50	a few seconds		
Configuration	2026-07-14 19:50	2026-07-14 19:55	6 minutes		
Configuration	2026-07-13 23:03	2026-07-13 23:03	a few seconds		
Configuration	2026-07-13 19:50	2026-07-13 19:55	5 minutes		
Configuration	2026-07-12 19:50	2026-07-12 19:50	a few seconds		
Configuration	2026-07-11 19:50	2026-07-11 19:50	a few seconds		
Configuration	2026-07-10 19:50	2026-07-10 19:50	a few seconds		
Configuration	2026-07-09 19:50	2026-07-09 19:55	5 minutes		
Configuration	2026-07-08 19:50	2026-07-08 19:50	a few seconds		
Configuration	2026-07-07 19:50	2026-07-07 19:55	5 minutes		
Configuration	2026-07-06 19:50	2026-07-06 19:50	a few seconds		
Configuration	2026-07-06 13:34	2026-07-06 13:34	a few seconds		
Configuration	2026-07-05 19:50	2026-07-05 19:55	6 minutes		
Configuration	2026-07-04 19:50	2026-07-04 19:50	a few seconds		
Configuration	2026-07-03 19:50	2026-07-03 19:50	a few seconds		
Configuration	2026-07-02 19:50	2026-07-02 19:50	a few seconds		
Configuration	2026-07-01 19:50	2026-07-01 19:50	a few seconds		
Configuration	2026-07-01 06:28	2026-07-01 06:28	a few seconds		
Configuration	2026-07-01 00:52	2026-07-01 00:57	5 minutes		
Configuration	2026-06-30 19:50	2026-06-30 19:55	5 minutes		
Configuration	2026-06-29 20:23	2026-06-29 20:23	a few seconds		
Configuration	2026-06-29 19:50	2026-06-29 19:55	5 minutes		
Configuration	2026-06-28 19:50	2026-06-28 19:50	a few seconds		
Configuration	2026-06-27 19:50	2026-06-27 19:50	a few seconds		
Configuration	2026-06-26 19:50	2026-06-26 19:50	a few seconds		
Configuration	2026-06-25 19:50	2026-06-25 19:50	a few seconds		
Configuration	2026-06-24 19:50	2026-06-24 19:50	a few seconds		
Configuration	2026-06-24 07:03	2026-06-24 07:03	a few seconds		
Configuration	2026-06-23 19:50	2026-06-23 19:55	5 minutes
                                            1 Reply Last reply Reply Quote 0

                                            Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                                            Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                                            With your input, this post could be even better 💗

                                            Register Login
                                            • First post
                                              Last post