XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Backup timeout error

    Scheduled Pinned Locked Moved Backup
    10 Posts 3 Posters 151 Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • H Offline
      henri9813
      last edited by

      Hello,

      I have this error in xen-orchestra.

      Currently, my xen orchestra is in a different network than my hypervisors.

      My backup target is in another DC of course thought NFS ( via Wireguard VPN ).

      Usually, it's doens't cause issue, but now, i have this error

      xen-orchestra  | 2025-04-30T16:01:03.955Z xo:backups:AbstractVmRunner WARN writer step failed {
xen-orchestra  |   error: Error: read ETIMEDOUT
xen-orchestra  |       at TLSWrap.onStreamRead (node:internal/stream_base_commons:218:20)
xen-orchestra  |       at TLSWrap.callbackTrampoline (node:internal/async_hooks:130:17) {
xen-orchestra  |     errno: -110,
xen-orchestra  |     code: 'ETIMEDOUT',
xen-orchestra  |     syscall: 'read'
xen-orchestra  |   },
xen-orchestra  |   step: 'writer.transfer()',
xen-orchestra  |   writer: 'IncrementalRemoteWriter'
xen-orchestra  | }

      It's problematic because, here, i think the issue is between Xen-orchestra and the final hypervisor, right ?

      ( it could be great to add some details on the error to ease the diagnostics 🙂 )

      thanks !

      H 1 Reply Last reply Reply Quote 0
      • H Offline
        henri9813 @henri9813
        last edited by

        @henri9813 I found this in my logs don't know if it's related

        Apr 30 21:24:00 gco-002-rbx-002 xapi: [debug||8618534 HTTPS 10.255.0.250->:::80|put_import|import] import handler
Apr 30 21:24:00 gco-002-rbx-002 xapi: [debug||8618535 /var/lib/xcp/xapi|post_root|dummytaskhelper] task dispatch:session.slave_login D:bd14de344b0e created by task D:3fcf04a3282e
Apr 30 21:24:00 gco-002-rbx-002 xapi: [ info||8618535 /var/lib/xcp/xapi|session.slave_login D:fefe31620afb|xapi_session] Session.create trackid=7ded2f51c51fbf82e2c5ac7cee9931dc pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618533 ||xapi_compression] nice failed to decompress: exit code 1
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] Raised End_of_file
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 1/13 xapi Raised at file unix/tar_unix.ml, line 34
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 2/13 xapi Called from file unix/tar_unix.ml, line 26
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 3/13 xapi Called from file lib/tar.ml, line 829
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 4/13 xapi Called from file lib/tar.ml, line 660
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 5/13 xapi Called from file lib/tar.ml, line 666
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 6/13 xapi Called from file lib/tar.ml, line 673
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 7/13 xapi Called from file lib/tar.ml, line 749
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 8/13 xapi Called from file lib/tar.ml, line 867
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 9/13 xapi Called from file ocaml/xapi/import.ml, line 2132
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 10/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 11/13 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 12/13 xapi Called from file ocaml/xapi/import.ml, line 2554
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace] 13/13 xapi Called from file lib/backtrace.ml, line 177
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|backtrace]
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|VM.import R:9d05d125c060|import] Prematurely reached end-of-file during import
Apr 30 21:24:00 gco-002-rbx-002 xapi: [debug||8618536 /var/lib/xcp/xapi|post_root|dummytaskhelper] task dispatch:pool.get_all D:8b21a6838cb8 created by task D:fefe31620afb
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace] VM.import R:9d05d125c060 failed with exception Server_error(IMPORT_ERROR_PREMATURE_EOF, [  ])
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace] Raised Server_error(IMPORT_ERROR_PREMATURE_EOF, [  ])
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace] 1/7 xapi Raised at file lib/backtrace.ml, line 210
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace] 2/7 xapi Called from file ocaml/xapi/import.ml, line 2204
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace] 3/7 xapi Called from file ocaml/xapi/server_helpers.ml, line 72
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace] 4/7 xapi Called from file ocaml/xapi/server_helpers.ml, line 94
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace] 5/7 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace] 6/7 xapi Called from file ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml, line 39
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace] 7/7 xapi Called from file ocaml/libs/log/debug.ml, line 250
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 :::80|VM.import D:7c2fd25192d0|backtrace]
Apr 30 21:24:00 gco-002-rbx-002 xapi: [debug||8618537 /var/lib/xcp/xapi|post_root|dummytaskhelper] task dispatch:session.logout D:f7e71c366389 created by task D:7c2fd25192d0
Apr 30 21:24:00 gco-002-rbx-002 xapi: [ info||8618537 /var/lib/xcp/xapi|session.logout D:218a803a945b|xapi_session] Session.destroy trackid=b2e981e2a3cbc9fc91324f34ef240f5d
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|put_import|backtrace] VM.import D:7c2fd25192d0 failed with exception Server_error(IMPORT_ERROR_PREMATURE_EOF, [  ])
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|put_import|backtrace] Raised Server_error(IMPORT_ERROR_PREMATURE_EOF, [  ])
Apr 30 21:24:00 gco-002-rbx-002 xapi: [error||8618530 HTTPS 10.255.0.250->:::80|put_import|backtrace] 1/14 xapi Raised at file ocaml/libs/log/debug.ml, line 267
        1 Reply Last reply Reply Quote 0
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          Hi,

          What kind of backup are you using and which version of XO source or XOA?

          H 1 Reply Last reply Reply Quote 0
          • H Offline
            henri9813 @olivierlambert
            last edited by

            Hello, @olivierlambert

            Latest version,

            XCP-NG 8.3 latest version, i'm trying to perform disaster recovery backup,

            I think the problems seems to come from my firewall which "cut" the TCP session.

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              DR (so full replication) and not CR (incremental replication), right?

              1. In DR, have you enabled compression? If yes, try to disable it
              2. If not tested, try in CR
              H 1 Reply Last reply Reply Quote 0
              • H Offline
                henri9813 @olivierlambert
                last edited by

                Hello @olivierlambert

                I confirm my issue came from my Firewall so, not related to XO.

                However, it could be great to make logs more "clear", i mean:

                Error: read ETIMEDOUT"

                Become

                Error: read ETIMEDOUT while connect to X.X.X.X:ABC

                That would permit to understand more quickly my "real and weird" issue 😄

                Best regards,

                1 Reply Last reply Reply Quote 1
                • olivierlambertO Offline
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  Can you detail more what was the issue with the firewall exactly?

                  Also, adding @florent in the loop for the feedback 🙂

                  H 1 Reply Last reply Reply Quote 0
                  • H Offline
                    henri9813 @olivierlambert
                    last edited by

                    @olivierlambert our opnsense resets the TCP states so the firewall block packet because it forgot about the tcp session.

                    And then, a timeout occured in the middle of the export.

                    lsouai-vatesL 1 Reply Last reply Reply Quote 1
                    • olivierlambertO Offline
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      Also pinging @lsouai-vates

                      1 Reply Last reply Reply Quote 0
                      • lsouai-vatesL Offline
                        lsouai-vates Vates 🪐 XO Team @henri9813
                        last edited by

                        @henri9813 hello, thnaks for the feedback.
                        I propose that you open a git feature in xen orchestra project so we can add it to our roadmap.

                        Have a good day 🙂

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post