<![CDATA[Testing ZFS with XCP-ng]]>I all,

I spent some time trying to get ZFS to work.

The main issue is that ZFS doesn't support O_DIRECT and that blktap only opens .vhd files with O_DIRECT.

I patched blktap to remove the O_DIRECT and replace it with O_DSYNC and try to run ZFS on a file SR. Here are some changes: https://gist.github.com/nraynaud/b32bd612f5856d1efc1418233d3aeb0f

The steps are the following:

  • remove the current blktap from your host
  • install the patched blktap
  • install the ZFS support RPMs
  • create the and mount the ZFS volume
  • create the file SR pointing to the mounted directory

Here is a ZFS test script:

XCP_HOST_UNDER_TEST=root@yourmachine
rsync -r zfs blktap-3.5.0-1.12test.x86_64.rpm ${XCP_HOST_UNDER_TEST}:
ssh ${XCP_HOST_UNDER_TEST} yum remove -y blktap
ssh ${XCP_HOST_UNDER_TEST} yum install -y blktap-3.5.0-1.12test.x86_64.rpm
ssh ${XCP_HOST_UNDER_TEST} yum install -y zfs/*.rpm
ssh ${XCP_HOST_UNDER_TEST} depmod -a
ssh ${XCP_HOST_UNDER_TEST} modprobe zfs
ssh ${XCP_HOST_UNDER_TEST} zpool create -f -m /mnt/zfs tank /dev/mydev
ssh ${XCP_HOST_UNDER_TEST} zfs set sync=disabled tank
ssh ${XCP_HOST_UNDER_TEST} zfs set compression=lz4 tank
ssh ${XCP_HOST_UNDER_TEST} zfs list

SR_ZFS_UUID=`ssh ${XCP_HOST_UNDER_TEST} "xe sr-create host-uuid=${XCP_HOST_UNDER_TEST_UUID} name-label=test-zfs-sr type=file other-config:o_direct=false device-config:location=/mnt/zfs/test-zfs-sr"`

In details

ZFS options

Obviously, you can create the ZFS volume as you need (mirror, RAID-Z, doesn't matter, it's entirely up to you!)

Regarding options: due to O_DIRECT removal, it seems that the default sync mode is as if you got sync=always. To enjoy ZFS real perfs, it's better to use sync=disabled.

Compression (using lz4) also offers a nice boost. Finally, if you are a bit kamikaze, you can try dedup, but frankly, not worth the risk.

Note: please extend your dom0 memory to enjoy more perfs in general! 4GiB seems a minimum, remember that your host will now act as a cache for your storage.

SR creation

We added a specific flag to allow SR usage without O_DIRECT:

xe sr-create host-uuid=<YOUR_HOST_UUID> name-label=ZFS-SR type=file other-config:o_direct=false device-config:location=/tank

You can see the other-config:o_direct=false flag required.

Download

The files are here: https://cloud.vates.fr/index.php/s/zDSLcsJ4l5tYHwL?path=%2F

I would be interested in someone else reproducing my work, fiddle with ZFS and tell us if it's worth pursuing.

Thanks,

Nico.

]]>https://xcp-ng.org/forum/topic/152/testing-zfs-with-xcp-ngRSS for NodeMon, 17 Aug 2026 04:18:53 GMTMon, 18 Jun 2018 16:36:16 GMT60<![CDATA[Reply to Testing ZFS with XCP-ng on Tue, 27 Nov 2018 20:24:53 GMT]]>@dvdhngs

install onto (mdadm/ZFS) RAID 10: no

to use (mdadm/ZFS) RAID 10 as SR: yes, of course!

]]>https://xcp-ng.org/forum/post/6137https://xcp-ng.org/forum/post/6137Tue, 27 Nov 2018 20:24:53 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Tue, 27 Nov 2018 20:21:22 GMT]]>@olivierlambert
Just to solve a doubt.... there is no way to install xcp-ng on software RAID 10, right?

]]>https://xcp-ng.org/forum/post/6136https://xcp-ng.org/forum/post/6136Tue, 27 Nov 2018 20:21:22 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Fri, 23 Nov 2018 15:12:21 GMT]]>Hi,

There is 2 different things:

  • soft RAID 1 support during install, using mdadm (the standard Linux software RAID mechanism)
  • ZFS support for storage repositories (where you store your VM disks)

Regarding your use case, it depends. You can choose to install XCP-ng system without RAID and use only one disk, and the rest for ZFS SR. Or RAID1 for XCP-ng system, and the left disks for ZFS.

]]>https://xcp-ng.org/forum/post/6033https://xcp-ng.org/forum/post/6033Fri, 23 Nov 2018 15:12:21 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Fri, 23 Nov 2018 15:04:59 GMT]]>Hello everybody! Newbie here, just 24h of xcp use, I was trying set Raid 10, but xcp only support Raid 1, so I'm trying now to set raidz, like on the document, I'm using 4 disks (500gb each), but I want to do Raid on this 4 disks (sda,sdb,sdc,sdd), so I can't? Need to be 6 disks? 2 raid 1 with xcp, and 4 raidz for other things?

]]>https://xcp-ng.org/forum/post/6032https://xcp-ng.org/forum/post/6032Fri, 23 Nov 2018 15:04:59 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Thu, 04 Oct 2018 22:07:57 GMT]]>Thanks @borzel

]]>https://xcp-ng.org/forum/post/4122https://xcp-ng.org/forum/post/4122Thu, 04 Oct 2018 22:07:57 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Thu, 04 Oct 2018 20:56:17 GMT]]>Just added the setup part to https://github.com/xcp-ng/xcp/wiki/ZFS-on-XCP-ng-7.5-and-later#setup

]]>https://xcp-ng.org/forum/post/4121https://xcp-ng.org/forum/post/4121Thu, 04 Oct 2018 20:56:17 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Thu, 20 Sep 2018 20:14:32 GMT]]>@nraynaud take your time, no rush

]]>https://xcp-ng.org/forum/post/3855https://xcp-ng.org/forum/post/3855Thu, 20 Sep 2018 20:14:32 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Thu, 20 Sep 2018 17:07:26 GMT]]>I am till on the issue, the subject is way deeper than expected.

]]>https://xcp-ng.org/forum/post/3850https://xcp-ng.org/forum/post/3850Thu, 20 Sep 2018 17:07:26 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Fri, 31 Aug 2018 18:10:25 GMT]]>@nraynaud I just tested with an up-to-date XCP-ng 7.5 host, same failure

# yum info vhd-tool
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Installed Packages
Name        : vhd-tool
Arch        : x86_64
Version     : 0.20.0
Release     : 4.3.xcp.el7.centos
Size        : 40 M
Repo        : installed
From repo   : xcp-ng-updates
Summary     : Command-line tools for manipulating and streaming .vhd format files
URL         : https://github.com/xapi-project/vhd-tool
License     : LGPL+linking exception
Description : Simple command-line tools for manipulating and streaming .vhd format file.
]]>https://xcp-ng.org/forum/post/3481https://xcp-ng.org/forum/post/3481Fri, 31 Aug 2018 18:10:25 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Wed, 22 Aug 2018 10:00:37 GMT]]>Does this issue also have a role if you run glusterfs on top of zfs?
Or is it just directly on zfs.

Because i would want to runs glusterfs on top of zfs on a spearate server, and use glusterfs driver to connect to it.

Do i also have this issue then?

]]>https://xcp-ng.org/forum/post/3262https://xcp-ng.org/forum/post/3262Wed, 22 Aug 2018 10:00:37 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Mon, 20 Aug 2018 23:16:09 GMT]]>Thanks,
I have found a suspected O_DIRECT here, the solution won't come quickly.

]]>https://xcp-ng.org/forum/post/3221https://xcp-ng.org/forum/post/3221Mon, 20 Aug 2018 23:16:09 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Wed, 15 Aug 2018 22:13:19 GMT]]>@nraynaud

# yum info vhd-tool
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Installed Packages
Name        : vhd-tool
Arch        : x86_64
Version     : 0.20.0
Release     : 4.2.xcp.el7.centos.extras
Size        : 40 M
Repo        : installed
From repo   : xcp-ng-extras
Summary     : Command-line tools for manipulating and
            : streaming .vhd format files
URL         : https://github.com/xapi-project/vhd-tool
License     : LGPL+linking exception
Description : Simple command-line tools for manipulating an
            : streaming .vhd format file.
]]>https://xcp-ng.org/forum/post/3040https://xcp-ng.org/forum/post/3040Wed, 15 Aug 2018 22:13:19 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Wed, 15 Aug 2018 19:17:45 GMT]]>@borzel I am struggling to match the error lines in your log with a source code, can you tell me what version of vhd-tool you have installed please ? (yum info vhd-tool)

]]>https://xcp-ng.org/forum/post/3031https://xcp-ng.org/forum/post/3031Wed, 15 Aug 2018 19:17:45 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Wed, 15 Aug 2018 15:07:23 GMT]]>@borzel good catch, it's another O_DIRECT issue.

]]>https://xcp-ng.org/forum/post/3017https://xcp-ng.org/forum/post/3017Wed, 15 Aug 2018 15:07:23 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Wed, 15 Aug 2018 14:55:01 GMT]]>@stormi wiki issue is allready solved 😉

]]>https://xcp-ng.org/forum/post/3016https://xcp-ng.org/forum/post/3016Wed, 15 Aug 2018 14:55:01 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Wed, 15 Aug 2018 12:55:49 GMT]]>@borzel I'm not very familiar with github wikis, but I think you can do pull requests against the wiki.

]]>https://xcp-ng.org/forum/post/3004https://xcp-ng.org/forum/post/3004Wed, 15 Aug 2018 12:55:49 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Wed, 15 Aug 2018 08:15:13 GMT]]>I'll try to run some tests on my side.

]]>https://xcp-ng.org/forum/post/2992https://xcp-ng.org/forum/post/2992Wed, 15 Aug 2018 08:15:13 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Tue, 14 Aug 2018 23:21:41 GMT]]>I found something in /var/log/xensource.log (used tail -f /var/log/xensource.log | grep -i 'error')

Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR
Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool output: vhd-tool: internal error, uncaught exception:#012          Unix.Unix_error(Unix.EINVAL, "open", "/var/run/sr-mount/9fc67ea3-c9b3-5e62-ecc3-7d1debf81744/6781ab7b-c475-4d72-a7d0-b2aeca2e81e3.vhd")#012          Raised at file "src/core/lwt.ml", line 3008, characters 20-29#012          Called from file "src/unix/lwt_main.ml", line 42, characters 8-18#012          Called from file "src/impl.ml", line 811, characters 4-23#012          Called from file "src/cmdliner_term.ml", line 27, characters 19-24#012          Called from file "src/cmdliner.ml", line 27, characters 27-34#012          Called from file "src/cmdliner.ml", line 106, characters 32-39
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] [XO] VDI Export R:3001f7e5aa17 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 1/13 xapi @ hw Raised at file ocaml/xapi/vhd_tool_wrapper.ml, line 59
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 4/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 47
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 6/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 7/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 54
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 8/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 65
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace]
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] VDI.export_raw_vdi D:315adbf2ec6c failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 1/13 xapi @ hw Raised at file lib/debug.ml, line 187
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 4/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 6/13 xapi @ hw Called from file ocaml/xapi/xapi_http.ml, line 185
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 7/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 8/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace]
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] VDI.export_raw_vdi D:fc8d0f58f1be failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] 1/1 xapi @ hw Raised at file (Thread 63005 has no backtrace table. Was with_backtraces called?, line 0
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace]
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||xapi] Unhandled Api_errors.Server_error(VDI_IO_ERROR, [ Device I/O errors ])
]]>https://xcp-ng.org/forum/post/2979https://xcp-ng.org/forum/post/2979Tue, 14 Aug 2018 23:21:41 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Tue, 14 Aug 2018 23:13:50 GMT]]>@nraynaud no, it's as empty as it could be 😞

]]>https://xcp-ng.org/forum/post/2978https://xcp-ng.org/forum/post/2978Tue, 14 Aug 2018 23:13:50 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Tue, 14 Aug 2018 23:12:55 GMT]]>@borzel > The error is often in the Settings->Logs panel in XO in this case

]]>https://xcp-ng.org/forum/post/2977https://xcp-ng.org/forum/post/2977Tue, 14 Aug 2018 23:12:55 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Tue, 14 Aug 2018 23:12:37 GMT]]>I searched in xensource.log, SMlog and syslog, no related entries found 😞

How can I increase the loglevel of XO?

Here's the log output if I start this job again:

Aug 15 01:10:35 ubuntu xo-server[616]: 2018-08-14T23:10:35.241Z xo:xapi Snapshotting VM pmg as [XO Backup test] pmg
Aug 15 01:10:36 ubuntu xo-server[616]: 2018-08-14T23:10:36.948Z xo:xapi exporting VDI pmg 0
Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.289Z xo:xapi Deleting VM [XO Backup test] pmg
Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.585Z xo:xapi Deleting VDI pmg 0

It's nearly nothing...

]]>https://xcp-ng.org/forum/post/2976https://xcp-ng.org/forum/post/2976Tue, 14 Aug 2018 23:12:37 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Tue, 14 Aug 2018 23:01:25 GMT]]>0_1534286419566_560b7fad-965d-42bc-8f40-2eed276163b8-grafik.png

]]>https://xcp-ng.org/forum/post/2975https://xcp-ng.org/forum/post/2975Tue, 14 Aug 2018 23:01:25 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Tue, 14 Aug 2018 07:10:10 GMT]]>VDI chain too long doesn't seems related. It means you have a VDI chain that's around 30 files.

Can you provide the exact/detailed error log?

]]>https://xcp-ng.org/forum/post/2918https://xcp-ng.org/forum/post/2918Tue, 14 Aug 2018 07:10:10 GMT<![CDATA[Reply to Testing ZFS with XCP-ng on Mon, 13 Aug 2018 23:10:10 GMT]]>I tested latest and greatest XO (5.24.1 from sources) with latest XCP-ng 7.5 (incl. xcp-ng-extra repo/ZFS). Backups do not work 😞

If the VDI is on a ext-SR, all is fine, backup working.
If the VDI is on a File-SR (ZFS based), the backup says "VDI_ERROR" or "VDI Chain to long". I will provide logfiles tomorrow if needed.

Setup of my File-SR:
xe sr-create host-uuid=<xxx> name-label=zfssr type=file other-config:o_direct=false device-config:location=/zfspool/zfssr

]]>https://xcp-ng.org/forum/post/2897https://xcp-ng.org/forum/post/2897Mon, 13 Aug 2018 23:10:10 GMT