Hello.
I want to share a new backup speed record after a long time. Top speed 727.9 MBps. Avarage 680-700 MBps. 1 CR Job with 5 concurrency.
Speed is not constant because there are many virtual machines whose vdi-export time is shorter than snapshot time.
XCP-ng 8.2, XO from source, xo-server 5.80.0, xo-web 5.84.0
Hello,
This is not an important issue, but I wanted to give information.
When I move the cursor over the statistics, the data is shown in the wrong location. This problem exists only in Firefox browser (Windows 10, Firefox 84). Edge and chrome work as expected.
Screenshot:
Thank you.
@olivierlambert with https, only single cpu core running 100% of XO vm. But with http all cpu core of XO vm running simultaneously.
So I give 8x3.0 GHz cpu to XO. It is never use more than 5 GB of RAM.
Also concurrent backup setting and XO on master host are improve the speed.
Hello and sorry for my bad English,
Just a feedback and thanks.
My new backup speed record: 430-450 MB per second! A CR job with 6 concurrency.
Thank you very much to both Xcp-Ng and Xen Orchestra team.
NICs: Everyting 10Gbps
Source: Netapp AFF A300 (full ssd)
Destination: Synology RS3614xs+ (without ssd disks. raid6)
Host: Intel Xeon E5-2690 v2 @ 3.00GHz (2 socket)
http traffic instead of https with this suggestion.
Hello,
Http ~7 times faster than https on our system for cr jobs. Https ~50MB/sec. Http ~350MB/sec.
xo from source 5.60
xcp-ng 8.0
10 gbit connection everywhere
@txsastre I'm taking snapshot of XO vm before every update. Simple revent it.
Because it is from source so I have no support subscription. I recommend this to you.
Hello,
I experience this problem with CR job. xo-server 5.51.1
I revent xo-server 5.50.1. No problem with this version.
I found these errors in xensource.log. maybe not related:
Importing raw VDI R:55902cfbe5b0|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR
vhd-tool output: vhd-tool: Non-zero size required (either a pre-existing destination file or specified via --destination-size on the command line)
and this:
xapi: [error|XenServer-08|12899 INET :::80|Importing raw VDI R:55902cfbe5b0|import] Caught exception: VDI_IO_ERROR: [ Device I/O errors ]
xapi: [debug|XenServer-08|12899 INET :::80|Importing raw VDI R:55902cfbe5b0|taskhelper] the status of R:55902cfbe5b0 is failure; cannot set it to `failure
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] Importing raw VDI R:55902cfbe5b0 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 1/12 xapi @ XenServer-08 Raised at file ocaml/xapi/vhd_tool_wrapper.ml, line 59
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 2/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 3/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 4/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 5/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 6/12 xapi @ XenServer-08 Called from file ocaml/xapi/import_raw_vdi.ml, line 81
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 7/12 xapi @ XenServer-08 Called from file ocaml/xapi/import_raw_vdi.ml, line 105
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 8/12 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 80
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 9/12 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 99
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 10/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 11/12 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 35
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace] 12/12 xapi @ XenServer-08 Called from file lib/backtrace.ml, line 177
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|backtrace]
xapi: [debug|XenServer-08|13023 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:8800126a89c1 created by task D:8f42c874d788
xapi: [ info|XenServer-08|13023 UNIX /var/lib/xcp/xapi|session.logout D:04108f92fdfe|xapi] Session.destroy trackid=4319d4a602955613971f7a77e5c9e8cf
xapi: [debug|XenServer-08|640 |xapi events D:0b0842fbd22a|xenops] Event on VM 20192c0a-8918-4e11-8e80-1ac10ff145d1; resident_here = true
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:8f42c874d788|import] Caught exception in import handler: VDI_IO_ERROR: [ Device I/O errors ]
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] VDI.import D:8f42c874d788 failed with exception Unix.Unix_error(Unix.EPIPE, "single_write", "")
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] Raised Unix.Unix_error(Unix.EPIPE, "single_write", "")
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 1/9 xapi @ XenServer-08 Raised at file unix.ml, line 326
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 2/9 xapi @ XenServer-08 Called from file lib/xapi-stdext-unix/unixext.ml, line 463
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 3/9 xapi @ XenServer-08 Called from file lib/xapi-stdext-unix/unixext.ml, line 465
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 4/9 xapi @ XenServer-08 Called from file ocaml/xapi/import_raw_vdi.ml, line 141
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 5/9 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 80
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 6/9 xapi @ XenServer-08 Called from file ocaml/xapi/server_helpers.ml, line 99
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 7/9 xapi @ XenServer-08 Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 24
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 8/9 xapi @ XenServer-08 Called from file map.ml, line 135
xapi: [debug|XenServer-08|640 |xapi events D:0b0842fbd22a|dummytaskhelper] task timeboxed_rpc D:647b3dd768d7 created by task D:0b0842fbd22a
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace] 9/9 xapi @ XenServer-08 Called from file sexp_conv.ml, line 148
xapi: [error|XenServer-08|12899 INET :::80|VDI.import D:9903e89b7f74|backtrace]
xapi: [error|XenServer-08|12899 INET :::80||backtrace] VDI.import D:9903e89b7f74 failed with exception Unix.Unix_error(Unix.EPIPE, "single_write", "")
xapi: [error|XenServer-08|12899 INET :::80||backtrace] Raised Unix.Unix_error(Unix.EPIPE, "single_write", "")
xapi: [error|XenServer-08|12899 INET :::80||backtrace] 1/1 xapi @ XenServer-08 Raised at file (Thread 12899 has no backtrace table. Was with_backtraces called?, line 0
xapi: [error|XenServer-08|12899 INET :::80||backtrace]
@karlisi said in daily Continuous Replication backup in XO:
... this is happening only on first schedule after each XO restart. Next schedules are working without issues.
Hello again,
My CR jobs are same. This is happening only first schedule after each XO restart as @karlisi said.
Hello,
Same problem here. Job start 2 seconds before scheduled time. And start again 2 seconds later exact scheduled time. First run finish successfully. Second run fail with this error: "the job (...) is already running"
XO 5.50.1 from source.
'guessVhdSizeOnImport: true' looks like solved the problem 9 day ago. But I'm getting same error sometimes and randomly since that day.
Daily CR job have 31 VMs. Some day successful 31/31. But some days one or two VM replication fail with tis message: VDI_IO_ERROR(Device I/O errors).
I think my problem still related with this topic.
@olivierlambert I've just tried and yes, 'guessVhdSizeOnImport: true' solved the problem.
Same problem here after upgrade xcp-ng 7.6 to 8.0
Successful upgrade 8 host in 1 pool from xcp-ng 7.6 to xcp-ng 8.0.
81 VMs. No downtime.
Thank you xcp-ng team.
No "Edit" button issue on my xcp-ng center 8.0.1. It's appear as normal.
@stormi installed on our pool hosts. Hosts boot normal. No vm boot issue Linux and Windows.
@stormi said in Updates announcements and testing:
Anyone?
I updated one host in a pool and reboot. It is working as before. Seems no problem. I can't see any diffrence vcpu usage as @borzel say.
@olivierlambert Similar performance for my environment. I did not any specific test. Just copy-paste files.
By the way, is multipath connection (or pnfs) possible xcp-ng between storage with nfs4.1?
I create a nfs sr with nfs 4.1 with xcp-ng center v7.6.3.21. success.
live storage migrate a vm to this sr. success.
live vm migrate to another host. success.
backup vm with xo. success.
I tested copy-paste file in itself and nfs4, nfs4.1, fc between each other. nfs 4.1 speed is like nfs4 speed for my environment.
heavy i/o for 24 hours. no any performance or another issue.
@stormi
I have an netapp storage. it support nfs 4.1. how can I add a nfs sr with nfs 4.1? I want test it.