XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Continuous replication fails every time

    Scheduled Pinned Locked Moved Xen Orchestra
    31 Posts 8 Posters 7.4k Views 5 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R Offline
      RKENS
      last edited by

      I'm trying to set up continuous replication between two identical hosts with local storage. I have two different VMs that I'm trying to replicate. When the jobs run, they both fail regularly with VDI-IO-ERROR, which it states is a device i/o error. However, the VMs work normally and as far as I know, there are no device problems. The failure occurs 2 hours into the transfer.CR failure error 10-29-19.JPG

      Any suggestions on how I can dig deeper into this?

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Online
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Before to dig deeper:

        • update your XO on latest version
        • update all your XCP-ng host
        1 Reply Last reply Reply Quote 0
        • txsastreT Offline
          txsastre
          last edited by

          I have to say that I'm having the same problems.

          everything is updated to the last version.

          xcp-ng 8.0.0

          xo-server 5.51.1
          xo-web 5.51.0

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Online
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Please open a support ticket if you can so we can take a look

            1 Reply Last reply Reply Quote 0
            • R Offline
              RKENS
              last edited by

              Has anyone gotten Continuous replication to work normally?

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Online
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by olivierlambert

                RKENS The question is more "has anyone getting issues with cont. replication" because we wouldn't keep a feature broken, especially with a feature that's used for our hundreds of customers 😉

                It might be related to your XO install, can you try with XOA?

                1 Reply Last reply Reply Quote 0
                • R Offline
                  RKENS
                  last edited by

                  I will update my XOA next week. I just wanted to confirm that others are in fact able to use CR successfully, since the other guy (txsastre) mentioned he had already updated his XOA and he still had the same issue.

                  Thanks,
                  Russ

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Online
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    That's because likely txsastre isn't using XOA. But XO from the sources or via a 3rd party script.

                    LPJonL 1 Reply Last reply Reply Quote 0
                    • LPJonL Offline
                      LPJon
                      last edited by

                      I have been trying to update from XOACE 5.49.0 to 5.51.1 . That's for both XO-Web and XO-Server. On 5.49.0 continuous Replication works perfectly but after an update to 5.51.1 they will fail with the exact same as in the image above. After a downgrade the Continuous Replication again works perfectly. The backup storage repository is a local remote storage repository dedicated for CR backups. It's as though something in the code has changed in the way that it accesses a local based storage repository. For now we have had to freeze updates until we can figure out why CR backups do not work. Please note that any backup that uses Delta or CR does not work anymore with XOACE 5.51.1 and a local storage repository. Is there any suggestions you may have on a different setup or possibly a missing plugin.

                      Another strange thing is that a previous SSL certificate that works with 5.49.0 does not work with 5.51.1 for the HTTPS setup of XO-Server.

                      1 Reply Last reply Reply Quote 0
                      • LPJonL Offline
                        LPJon @olivierlambert
                        last edited by

                        olivierlambert So does that mean that XO from sources will not support CR or delta backups?

                        1 Reply Last reply Reply Quote 0
                        • txsastreT Offline
                          txsastre
                          last edited by

                          hi there, yes we're using XOCE and I said that we have this problem because may be something wrong in the latests releases.

                          just trying to help the developers to show that there is a problem in the new versions, as I can see XOA is more tested may be because the XOCE users help to test it.

                          so my posts are intended to help not to complain about this wonderful product.

                          1 Reply Last reply Reply Quote 1
                          • olivierlambertO Online
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            LPJon not at all! It's just that the only tested environment is XOA. From the sources, you have to rely on community or yourself if you have problems. It's impossible to test on all platforms, that's why XOA is recommended for production 🙂

                            Also, bugs could happen into XO itself, on master branch (which shouldn't to often because we use features branches), but there's no guarantees 🙂

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Online
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by olivierlambert

                              txsastre that's exactly how I got it, no worries 😉 Having users on master branch helps to find bugs that we won't have to discover during QA phase before a XOA release 🙂

                              In this case, it's might be related to the guessVhdsize variable or something like that.

                              LPJonL 1 Reply Last reply Reply Quote 1
                              • LPJonL Offline
                                LPJon @olivierlambert
                                last edited by

                                olivierlambert I have looked into that with no success. I'm not sure if you just change that variable to false or if you just comment it out completely. I have tried both but neither had an affect on my situation. It seems the guessVhdsize variable is related to working with an nfs share for the remote which I don't have.

                                More or less you saying that this could be a bug in the master branch of XOCE but that XO is not showing this issue? What branch could I switch to that might be more stable?

                                1 Reply Last reply Reply Quote 0
                                • olivierlambertO Online
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  1. No, guessVhdSize is NOT related to the remote, but to the way we send data to the destination server. Double check you enabled it correctly and restart XO after the modification.
                                  2. No, I'm not saying this. Also, there's no stable branch on the sources, if you want something tested and stable, it's XOA 🙂
                                  LPJonL 1 Reply Last reply Reply Quote 0
                                  • LPJonL Offline
                                    LPJon @olivierlambert
                                    last edited by LPJon

                                    olivierlambert It came enabled by default with the upgrade to 5.51.1. What I might be having issues with is how to disable it the proper way for testing. I don't see anywhere where is says how to disable it, only how to enable it since there were failures happening with it disabled...lol. That's a tongue twister, anyway do you have any suggestions or might this be a question for Julien?

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Online
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      I have no idea, guessVhdSize is only an intuition on my side. I'm pinging julien-f who will try to answer Monday 🙂

                                      1 Reply Last reply Reply Quote 0
                                      • julien-fJ Offline
                                        julien-f Vates 🪐 Co-Founder XO Team
                                        last edited by

                                        LPJon Like all settings from config.toml and sample.config.toml, you can override it in your user config (probably /etc/xo-server/config.toml?):

                                        guessVhdSizeOnImport = false
                                        
                                        LPJonL 1 Reply Last reply Reply Quote 0
                                        • LPJonL Offline
                                          LPJon @julien-f
                                          last edited by

                                          julien-f Ok....thanks Olivier and Julien. Changing that setting in that way had no effect on the issue. You have any idea what would cause this from one version to the other. XCP-ng is version 7.5 with a second local storage repository.

                                          1 Reply Last reply Reply Quote 0
                                          • rizaemet 0R Offline
                                            rizaemet 0
                                            last edited by

                                            Hello,
                                            I experience this problem with CR job. xo-server 5.51.1
                                            I revent xo-server 5.50.1. No problem with this version.

                                            I found these errors in xensource.log. maybe not related:

                                            Importing raw VDI R:55902cfbe5b0|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR
                                            vhd-tool output: vhd-tool: Non-zero size required (either a pre-existing destination file or specified via --destination-size on the command line)
                                            

                                            and this:

                                            xapi: [error|XenServer-08|12899 INET :::80|Importing raw VDI R:55902cfbe5b0|import] Caught exception: VDI_IO_ERROR: [ Device I/O errors ]
                                            xapi: [debug|XenServer-08|12899 INET :::80|Importing raw VDI R:55902cfbe5b0|taskhelper] the status of R:55902cfbe5b0 is failure; cannot set it to `failure
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] Importing raw VDI R:55902cfbe5b0 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 1/12 xapi @ XenServer-08 Raised at file ocaml/xapi/vhd_tool_wrapper.ml, line 59
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 2/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 3/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 4/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 5/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 6/12 xapi @ XenServer-08 Called from file ocaml/xapi/import_raw_vdi.ml, line 81
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 7/12 xapi @ XenServer-08 Called from file ocaml/xapi/import_raw_vdi.ml, line 105
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 8/12 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 80
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 9/12 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 99
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 10/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 11/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 12/12 xapi @ XenServer-08 Called from file lib/backtrace.ml, line 177
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace]
                                            xapi: [debug|XenServer-08|13023 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:8800126a89c1 created by task D:8f42c874d788
                                            xapi: [ info|XenServer-08|13023 UNIX /var/lib/xcp/xapi|session.logout D:04108f92fdfe|xapi] Session.destroy trackid=4319d4a602955613971f7a77e5c9e8cf
                                            xapi: [debug|XenServer-08|640 |xapi events D:0b0842fbd22a|xenops] Event on VM 20192c0a-8918-4e11-8e80-1ac10ff145d1; resident_here = true
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|import] Caught exception in import handler: VDI_IO_ERROR: [ Device I/O errors ]
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] VDI.import D:8f42c874d788 failed with exception Unix.Unix_error(Unix.EPIPE, "single_write", "")
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] Raised Unix.Unix_error(Unix.EPIPE, "single_write", "")
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 1/9 xapi @ XenServer-08 Raised at file unix.ml, line 326
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 2/9 xapi @ XenServer-08 Called from file lib/xapi-stdext-unix/unixext.ml, line 463
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 3/9 xapi @ XenServer-08 Called from file lib/xapi-stdext-unix/unixext.ml, line 465
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 4/9 xapi @ XenServer-08 Called from file ocaml/xapi/import_raw_vdi.ml, line 141
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 5/9 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 80
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 6/9 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 99
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 7/9 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 8/9 xapi @ XenServer-08 Called from file map.ml, line 135
                                            xapi: [debug|XenServer-08|640 |xapi events D:0b0842fbd22a|dummytaskhelper] task timeboxed_rpc D:647b3dd768d7 created by task D:0b0842fbd22a
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 9/9 xapi @ XenServer-08 Called from file sexp_conv.ml, line 148
                                            xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace]
                                            xapi: [error|XenServer-08|12899 INET :::80||backtrace] VDI.import D:9903e89b7f74 failed with exception Unix.Unix_error(Unix.EPIPE, "single_write", "")
                                            xapi: [error|XenServer-08|12899 INET :::80||backtrace] Raised Unix.Unix_error(Unix.EPIPE, "single_write", "")
                                            xapi: [error|XenServer-08|12899 INET :::80||backtrace] 1/1 xapi @ XenServer-08 Raised at file (Thread 12899 has no backtrace table. Was with_backtraces called?, line 0
                                            xapi: [error|XenServer-08|12899 INET :::80||backtrace]
                                            
                                            
                                            txsastreT 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post