Testing ZFS with XCP-ng
-
@stormi said in Testing ZFS with XCP-ng:
@borzel It would be nice also to improve https://github.com/xcp-ng/xcp/wiki/ZFS-on-XCP-ng-7.5-and-later because I wrote it and I never used zfs!
I have no write permissions
-
Good ressource with a lot of information, how to really use ZFS, build storages, tuning, etc.: https://www.zfsbuild.com
-
Zfs is a local sr and imho does not add too much advantages vs classic ext sr
The real value of zfs is to use built in replication to remote zfs system.
Proxmox has built a very interesting solutions around it: async replication of vm on a another node.I know xoa already offer a similar solution but is snapshot based and everyone knows the performance and size problem with this approach.
-
@buzzzo said in Testing ZFS with XCP-ng:
Zfs is a local sr and imho does not add too much advantages vs classic ext sr
meeeep
- ZFS uses (in standard configuration) half the RAM as read cache
- with
sync=disabled
your writes are faster (but synced to discs every 5 seconds; in one sequential batch - efficient use of slow spinning drives!) - it has checksums from fs-layer down to the device
- it's awesome
know xoa already offer a similar solution but is snapshot based and everyone knows the performance and size problem with this approach.
If we all had 48 hours a day and more people in the team, we could fully integrate ZFS into XCP-ng (with all bells and whistles). But for now we are happy to have the first step done - loose integrate ZFS to start using it and testing stability of the dom0 with ZFS in it.
-
olivierlambert Vates 🪐 Co-Founder CEOlast edited by olivierlambert 8 Nov 2018, 21:34 11 Aug 2018, 19:33
@buzzzo Double meeeep
On file level SR (ext, NFS, local ZFS, XOSAN), there is no space issue doing snapshots. Also, perfs are good even with snapshots if you don't have an asthmatic SR.
-
@olivierlambert meeep!!!
-
-
I tested latest and greatest XO (5.24.1 from sources) with latest XCP-ng 7.5 (incl. xcp-ng-extra repo/ZFS). Backups do not work
If the VDI is on a ext-SR, all is fine, backup working.
If the VDI is on a File-SR (ZFS based), the backup says "VDI_ERROR" or "VDI Chain to long". I will provide logfiles tomorrow if needed.Setup of my File-SR:
xe sr-create host-uuid=<xxx> name-label=zfssr type=file other-config:o_direct=false device-config:location=/zfspool/zfssr
-
VDI chain too long
doesn't seems related. It means you have a VDI chain that's around 30 files.Can you provide the exact/detailed error log?
-
-
I searched in xensource.log, SMlog and syslog, no related entries found
How can I increase the loglevel of XO?
Here's the log output if I start this job again:
Aug 15 01:10:35 ubuntu xo-server[616]: 2018-08-14T23:10:35.241Z xo:xapi Snapshotting VM pmg as [XO Backup test] pmg Aug 15 01:10:36 ubuntu xo-server[616]: 2018-08-14T23:10:36.948Z xo:xapi exporting VDI pmg 0 Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.289Z xo:xapi Deleting VM [XO Backup test] pmg Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.585Z xo:xapi Deleting VDI pmg 0
It's nearly nothing...
-
@borzel > The error is often in the Settings->Logs panel in XO in this case
-
@nraynaud no, it's as empty as it could be
-
I found something in /var/log/xensource.log (used
tail -f /var/log/xensource.log | grep -i 'error'
)Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool output: vhd-tool: internal error, uncaught exception:#012 Unix.Unix_error(Unix.EINVAL, "open", "/var/run/sr-mount/9fc67ea3-c9b3-5e62-ecc3-7d1debf81744/6781ab7b-c475-4d72-a7d0-b2aeca2e81e3.vhd")#012 Raised at file "src/core/lwt.ml", line 3008, characters 20-29#012 Called from file "src/unix/lwt_main.ml", line 42, characters 8-18#012 Called from file "src/impl.ml", line 811, characters 4-23#012 Called from file "src/cmdliner_term.ml", line 27, characters 19-24#012 Called from file "src/cmdliner.ml", line 27, characters 27-34#012 Called from file "src/cmdliner.ml", line 106, characters 32-39 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] [XO] VDI Export R:3001f7e5aa17 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ]) Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ]) Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 1/13 xapi @ hw Raised at file ocaml/xapi/vhd_tool_wrapper.ml, line 59 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 4/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 47 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 6/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 7/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 54 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 8/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 65 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] VDI.export_raw_vdi D:315adbf2ec6c failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ]) Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ]) Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 1/13 xapi @ hw Raised at file lib/debug.ml, line 187 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 4/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 6/13 xapi @ hw Called from file ocaml/xapi/xapi_http.ml, line 185 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 7/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 8/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] VDI.export_raw_vdi D:fc8d0f58f1be failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ]) Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ]) Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] 1/1 xapi @ hw Raised at file (Thread 63005 has no backtrace table. Was with_backtraces called?, line 0 Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||xapi] Unhandled Api_errors.Server_error(VDI_IO_ERROR, [ Device I/O errors ])
-
I'll try to run some tests on my side.
-
@borzel I'm not very familiar with github wikis, but I think you can do pull requests against the wiki.
-
@stormi wiki issue is allready solved
-
@borzel good catch, it's another O_DIRECT issue.
-
@borzel I am struggling to match the error lines in your log with a source code, can you tell me what version of vhd-tool you have installed please ? (
yum info vhd-tool
) -
# yum info vhd-tool Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Installed Packages Name : vhd-tool Arch : x86_64 Version : 0.20.0 Release : 4.2.xcp.el7.centos.extras Size : 40 M Repo : installed From repo : xcp-ng-extras Summary : Command-line tools for manipulating and : streaming .vhd format files URL : https://github.com/xapi-project/vhd-tool License : LGPL+linking exception Description : Simple command-line tools for manipulating an : streaming .vhd format file.