Continuous replication fails every time

Reply to Continuous replication fails every time on Tue, 19 Nov 2019 16:26:29 GMT

olivierlambert — Tue, 19 Nov 2019 16:26:29 GMT

Your CR issue might be related to a XO bug on master, not too a XCP-ng issue

Reply to Continuous replication fails every time on Tue, 19 Nov 2019 14:13:33 GMT

RKENS — Tue, 19 Nov 2019 14:13:33 GMT

That's too bad. I don't see where anyone really had any solution.

Reply to Continuous replication fails every time on Tue, 19 Nov 2019 09:21:11 GMT

olivierlambert — Tue, 19 Nov 2019 09:21:11 GMT

No, since few release, ZFS should support all operations.

Reply to Continuous replication fails every time on Tue, 19 Nov 2019 06:18:36 GMT

RKENS — Tue, 19 Nov 2019 06:18:36 GMT

I saw this note relating to ZFS: It is not yet possible to import, migrate or export a VDI to/from a ZFS SR

Could that be why my continuous replication is failing? I'm trying to replicate from one ZFS SR to another on a different server.

Reply to Continuous replication fails every time on Fri, 08 Nov 2019 03:27:20 GMT

Fungusware — Fri, 08 Nov 2019 03:27:20 GMT

Can any comment on the fact that all of these issues are caused because 'import_raw_vdi' doesn't support Chunked transfers?

This used to be fine casuse the loophole that allowed a massive Content-Length has been closed.

(here: xapi-project/vhd-tool@f978789#diff-a1476b2332c40710d3f1791146a29aeaL484 ).

Regards.

0 jonludlam committed to xapi-project/vhd-tool

XSI-19 locate VHD footer based on file size

I order to locate the footer in a VHD file or stream, the total length
of that file or stream must be known. This commit passes it when it is
known. This depends on a corresponding change in the ocaml-vhd library.

Signed-off-by: Christian Lindig

Reply to Continuous replication fails every time on Wed, 06 Nov 2019 12:54:14 GMT

txsastre — Wed, 06 Nov 2019 12:54:14 GMT

@Danp thanks ! but I have rolled back to a previous version ( 5.50.1) and now I'm testing the backup again.

Reply to Continuous replication fails every time on Wed, 06 Nov 2019 12:24:25 GMT

Danp — Wed, 06 Nov 2019 12:24:25 GMT

@txsastre I outlined the procedure here --

https://xcp-ng.org/forum/post/18974

Reply to Continuous replication fails every time on Wed, 06 Nov 2019 10:14:42 GMT

txsastre — Wed, 06 Nov 2019 10:14:42 GMT

@rizaemet-0 thanks you've got the right. I already doing this, but I didn't know if you uninstalled it and installed a previous version.

it was just curiosity. So I'm gonna rollback to a previous snapshot too.

Thank you

Reply to Continuous replication fails every time on Wed, 06 Nov 2019 08:43:16 GMT

rizaemet 0 — Wed, 06 Nov 2019 08:43:16 GMT

@txsastre I'm taking snapshot of XO vm before every update. Simple revent it.

Because it is from source so I have no support subscription. I recommend this to you.

Reply to Continuous replication fails every time on Wed, 06 Nov 2019 08:27:12 GMT

txsastre — Wed, 06 Nov 2019 08:27:12 GMT

@rizaemet-0 could you tell me how have you reverted to 5.50.1 ? thanks

Reply to Continuous replication fails every time on Tue, 05 Nov 2019 07:54:50 GMT

julien-f — Tue, 05 Nov 2019 07:54:50 GMT

Could this be related to this: https://github.com/vatesfr/xen-orchestra/issues/4640?

BluejacketScott created this issue in vatesfr/xen-orchestra

closed XO From source restore backup fails with VDI_IO_ERROR #4640

Reply to Continuous replication fails every time on Mon, 04 Nov 2019 09:01:35 GMT

rizaemet 0 — Mon, 04 Nov 2019 09:01:35 GMT

Hello,
I experience this problem with CR job. xo-server 5.51.1
I revent xo-server 5.50.1. No problem with this version.

I found these errors in xensource.log. maybe not related:

Importing raw VDI R:55902cfbe5b0|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR
vhd-tool output: vhd-tool: Non-zero size required (either a pre-existing destination file or specified via --destination-size on the command line)

and this:

xapi: [error|XenServer-08|12899 INET :::80|Importing raw VDI R:55902cfbe5b0|import] Caught exception: VDI_IO_ERROR: [ Device I/O errors ]
xapi: [debug|XenServer-08|12899 INET :::80|Importing raw VDI R:55902cfbe5b0|taskhelper] the status of R:55902cfbe5b0 is failure; cannot set it to `failure
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] Importing raw VDI R:55902cfbe5b0 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 1/12 xapi @ XenServer-08 Raised at file ocaml/xapi/vhd_tool_wrapper.ml, line 59
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 2/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 3/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 4/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 5/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 6/12 xapi @ XenServer-08 Called from file ocaml/xapi/import_raw_vdi.ml, line 81
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 7/12 xapi @ XenServer-08 Called from file ocaml/xapi/import_raw_vdi.ml, line 105
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 8/12 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 80
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 9/12 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 99
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 10/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 11/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 12/12 xapi @ XenServer-08 Called from file lib/backtrace.ml, line 177
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace]
xapi: [debug|XenServer-08|13023 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:8800126a89c1 created by task D:8f42c874d788
xapi: [ info|XenServer-08|13023 UNIX /var/lib/xcp/xapi|session.logout D:04108f92fdfe|xapi] Session.destroy trackid=4319d4a602955613971f7a77e5c9e8cf
xapi: [debug|XenServer-08|640 |xapi events D:0b0842fbd22a|xenops] Event on VM 20192c0a-8918-4e11-8e80-1ac10ff145d1; resident_here = true
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|import] Caught exception in import handler: VDI_IO_ERROR: [ Device I/O errors ]
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] VDI.import D:8f42c874d788 failed with exception Unix.Unix_error(Unix.EPIPE, "single_write", "")
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] Raised Unix.Unix_error(Unix.EPIPE, "single_write", "")
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 1/9 xapi @ XenServer-08 Raised at file unix.ml, line 326
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 2/9 xapi @ XenServer-08 Called from file lib/xapi-stdext-unix/unixext.ml, line 463
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 3/9 xapi @ XenServer-08 Called from file lib/xapi-stdext-unix/unixext.ml, line 465
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 4/9 xapi @ XenServer-08 Called from file ocaml/xapi/import_raw_vdi.ml, line 141
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 5/9 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 80
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 6/9 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 99
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 7/9 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 8/9 xapi @ XenServer-08 Called from file map.ml, line 135
xapi: [debug|XenServer-08|640 |xapi events D:0b0842fbd22a|dummytaskhelper] task timeboxed_rpc D:647b3dd768d7 created by task D:0b0842fbd22a
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 9/9 xapi @ XenServer-08 Called from file sexp_conv.ml, line 148
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace]
xapi: [error|XenServer-08|12899 INET :::80||backtrace] VDI.import D:9903e89b7f74 failed with exception Unix.Unix_error(Unix.EPIPE, "single_write", "")
xapi: [error|XenServer-08|12899 INET :::80||backtrace] Raised Unix.Unix_error(Unix.EPIPE, "single_write", "")
xapi: [error|XenServer-08|12899 INET :::80||backtrace] 1/1 xapi @ XenServer-08 Raised at file (Thread 12899 has no backtrace table. Was with_backtraces called?, line 0
xapi: [error|XenServer-08|12899 INET :::80||backtrace]

Reply to Continuous replication fails every time on Sun, 03 Nov 2019 14:10:28 GMT

LPJon — Sun, 03 Nov 2019 14:10:28 GMT

@julien-f Ok....thanks Olivier and Julien. Changing that setting in that way had no effect on the issue. You have any idea what would cause this from one version to the other. XCP-ng is version 7.5 with a second local storage repository.

Reply to Continuous replication fails every time on Sun, 03 Nov 2019 13:55:57 GMT

julien-f — Sun, 03 Nov 2019 13:55:57 GMT

@LPJon Like all settings from config.toml and sample.config.toml, you can override it in your user config (probably /etc/xo-server/config.toml?):

guessVhdSizeOnImport = false

Reply to Continuous replication fails every time on Sat, 02 Nov 2019 17:57:35 GMT

olivierlambert — Sat, 02 Nov 2019 17:57:35 GMT

I have no idea, guessVhdSize is only an intuition on my side. I'm pinging @julien-f who will try to answer Monday

Reply to Continuous replication fails every time on Sun, 03 Nov 2019 13:12:00 GMT

LPJon — Sun, 03 Nov 2019 13:12:00 GMT

@olivierlambert It came enabled by default with the upgrade to 5.51.1. What I might be having issues with is how to disable it the proper way for testing. I don't see anywhere where is says how to disable it, only how to enable it since there were failures happening with it disabled...lol. That's a tongue twister, anyway do you have any suggestions or might this be a question for Julien?

Reply to Continuous replication fails every time on Sat, 02 Nov 2019 16:19:58 GMT

olivierlambert — Sat, 02 Nov 2019 16:19:58 GMT

No, guessVhdSize is NOT related to the remote, but to the way we send data to the destination server. Double check you enabled it correctly and restart XO after the modification.
No, I'm not saying this. Also, there's no stable branch on the sources, if you want something tested and stable, it's XOA

Reply to Continuous replication fails every time on Sat, 02 Nov 2019 15:53:09 GMT

LPJon — Sat, 02 Nov 2019 15:53:09 GMT

@olivierlambert I have looked into that with no success. I'm not sure if you just change that variable to false or if you just comment it out completely. I have tried both but neither had an affect on my situation. It seems the guessVhdsize variable is related to working with an nfs share for the remote which I don't have.

More or less you saying that this could be a bug in the master branch of XOCE but that XO is not showing this issue? What branch could I switch to that might be more stable?

Reply to Continuous replication fails every time on Sat, 02 Nov 2019 11:40:19 GMT

olivierlambert — Sat, 02 Nov 2019 11:40:19 GMT

@txsastre that's exactly how I got it, no worries Having users on master branch helps to find bugs that we won't have to discover during QA phase before a XOA release

In this case, it's might be related to the guessVhdsize variable or something like that.

Reply to Continuous replication fails every time on Sat, 02 Nov 2019 11:39:11 GMT

olivierlambert — Sat, 02 Nov 2019 11:39:11 GMT

@LPJon not at all! It's just that the only tested environment is XOA. From the sources, you have to rely on community or yourself if you have problems. It's impossible to test on all platforms, that's why XOA is recommended for production

Also, bugs could happen into XO itself, on master branch (which shouldn't to often because we use features branches), but there's no guarantees

Reply to Continuous replication fails every time on Sat, 02 Nov 2019 11:03:52 GMT

txsastre — Sat, 02 Nov 2019 11:03:52 GMT

hi there, yes we're using XOCE and I said that we have this problem because may be something wrong in the latests releases.

just trying to help the developers to show that there is a problem in the new versions, as I can see XOA is more tested may be because the XOCE users help to test it.

so my posts are intended to help not to complain about this wonderful product.

Reply to Continuous replication fails every time on Sat, 02 Nov 2019 01:20:11 GMT

LPJon — Sat, 02 Nov 2019 01:20:11 GMT

@olivierlambert So does that mean that XO from sources will not support CR or delta backups?

Reply to Continuous replication fails every time on Sat, 02 Nov 2019 01:00:52 GMT

LPJon — Sat, 02 Nov 2019 01:00:52 GMT

I have been trying to update from XOACE 5.49.0 to 5.51.1 . That's for both XO-Web and XO-Server. On 5.49.0 continuous Replication works perfectly but after an update to 5.51.1 they will fail with the exact same as in the image above. After a downgrade the Continuous Replication again works perfectly. The backup storage repository is a local remote storage repository dedicated for CR backups. It's as though something in the code has changed in the way that it accesses a local based storage repository. For now we have had to freeze updates until we can figure out why CR backups do not work. Please note that any backup that uses Delta or CR does not work anymore with XOACE 5.51.1 and a local storage repository. Is there any suggestions you may have on a different setup or possibly a missing plugin.

Another strange thing is that a previous SSL certificate that works with 5.49.0 does not work with 5.51.1 for the HTTPS setup of XO-Server.

Reply to Continuous replication fails every time on Fri, 01 Nov 2019 21:43:45 GMT

olivierlambert — Fri, 01 Nov 2019 21:43:45 GMT

That's because likely @txsastre isn't using XOA. But XO from the sources or via a 3rd party script.