XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Testing ZFS with XCP-ng

    Scheduled Pinned Locked Moved Development
    80 Posts 10 Posters 40.1k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • B Offline
      buzzzo @borzel
      last edited by

      Zfs is a local sr and imho does not add too much advantages vs classic ext sr

      The real value of zfs is to use built in replication to remote zfs system.
      Proxmox has built a very interesting solutions around it: async replication of vm on a another node.

      I know xoa already offer a similar solution but is snapshot based and everyone knows the performance and size problem with this approach.

      borzelB olivierlambertO 2 Replies Last reply Reply Quote 0
      • borzelB Offline
        borzel XCP-ng Center Team @buzzzo
        last edited by borzel

        @buzzzo said in Testing ZFS with XCP-ng:

        Zfs is a local sr and imho does not add too much advantages vs classic ext sr

        meeeep 🙂

        • ZFS uses (in standard configuration) half the RAM as read cache
        • with sync=disabled your writes are faster (but synced to discs every 5 seconds; in one sequential batch - efficient use of slow spinning drives!)
        • it has checksums from fs-layer down to the device
        • it's awesome 😉

        know xoa already offer a similar solution but is snapshot based and everyone knows the performance and size problem with this approach.

        If we all had 48 hours a day and more people in the team, we could fully integrate ZFS into XCP-ng (with all bells and whistles). But for now we are happy to have the first step done - loose integrate ZFS to start using it and testing stability of the dom0 with ZFS in it.

        1 Reply Last reply Reply Quote 0
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO @buzzzo
          last edited by olivierlambert

          @buzzzo Double meeeep 😆

          On file level SR (ext, NFS, local ZFS, XOSAN), there is no space issue doing snapshots. Also, perfs are good even with snapshots if you don't have an asthmatic SR.

          B 1 Reply Last reply Reply Quote 0
          • B Offline
            buzzzo @olivierlambert
            last edited by

            @olivierlambert meeep!!! ☺

            1 Reply Last reply Reply Quote 0
            • olivierlambertO Offline
              olivierlambert Vates 🪐 Co-Founder CEO
              last edited by

              Meep meep!

              1 Reply Last reply Reply Quote 1
              • borzelB Offline
                borzel XCP-ng Center Team
                last edited by borzel

                I tested latest and greatest XO (5.24.1 from sources) with latest XCP-ng 7.5 (incl. xcp-ng-extra repo/ZFS). Backups do not work 😞

                If the VDI is on a ext-SR, all is fine, backup working.
                If the VDI is on a File-SR (ZFS based), the backup says "VDI_ERROR" or "VDI Chain to long". I will provide logfiles tomorrow if needed.

                Setup of my File-SR:
                xe sr-create host-uuid=<xxx> name-label=zfssr type=file other-config:o_direct=false device-config:location=/zfspool/zfssr

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates 🪐 Co-Founder CEO
                  last edited by

                  VDI chain too long doesn't seems related. It means you have a VDI chain that's around 30 files.

                  Can you provide the exact/detailed error log?

                  1 Reply Last reply Reply Quote 0
                  • borzelB Offline
                    borzel XCP-ng Center Team
                    last edited by borzel

                    0_1534286419566_560b7fad-965d-42bc-8f40-2eed276163b8-grafik.png

                    1 Reply Last reply Reply Quote 0
                    • borzelB Offline
                      borzel XCP-ng Center Team
                      last edited by borzel

                      I searched in xensource.log, SMlog and syslog, no related entries found 😞

                      How can I increase the loglevel of XO?

                      Here's the log output if I start this job again:

                      Aug 15 01:10:35 ubuntu xo-server[616]: 2018-08-14T23:10:35.241Z xo:xapi Snapshotting VM pmg as [XO Backup test] pmg
Aug 15 01:10:36 ubuntu xo-server[616]: 2018-08-14T23:10:36.948Z xo:xapi exporting VDI pmg 0
Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.289Z xo:xapi Deleting VM [XO Backup test] pmg
Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.585Z xo:xapi Deleting VDI pmg 0

                      It's nearly nothing...

                      1 Reply Last reply Reply Quote 0
                      • nraynaudN Offline
                        nraynaud XCP-ng Team
                        last edited by

                        @borzel > The error is often in the Settings->Logs panel in XO in this case

                        borzelB 1 Reply Last reply Reply Quote 0
                        • borzelB Offline
                          borzel XCP-ng Center Team @nraynaud
                          last edited by

                          @nraynaud no, it's as empty as it could be 😞

                          1 Reply Last reply Reply Quote 0
                          • borzelB Offline
                            borzel XCP-ng Center Team
                            last edited by

                            I found something in /var/log/xensource.log (used tail -f /var/log/xensource.log | grep -i 'error')

                            Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR
Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool output: vhd-tool: internal error, uncaught exception:#012          Unix.Unix_error(Unix.EINVAL, "open", "/var/run/sr-mount/9fc67ea3-c9b3-5e62-ecc3-7d1debf81744/6781ab7b-c475-4d72-a7d0-b2aeca2e81e3.vhd")#012          Raised at file "src/core/lwt.ml", line 3008, characters 20-29#012          Called from file "src/unix/lwt_main.ml", line 42, characters 8-18#012          Called from file "src/impl.ml", line 811, characters 4-23#012          Called from file "src/cmdliner_term.ml", line 27, characters 19-24#012          Called from file "src/cmdliner.ml", line 27, characters 27-34#012          Called from file "src/cmdliner.ml", line 106, characters 32-39
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] [XO] VDI Export R:3001f7e5aa17 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 1/13 xapi @ hw Raised at file ocaml/xapi/vhd_tool_wrapper.ml, line 59
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 4/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 47
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 6/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 7/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 54
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 8/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 65
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace]
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] VDI.export_raw_vdi D:315adbf2ec6c failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 1/13 xapi @ hw Raised at file lib/debug.ml, line 187
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 4/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 6/13 xapi @ hw Called from file ocaml/xapi/xapi_http.ml, line 185
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 7/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 8/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace]
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] VDI.export_raw_vdi D:fc8d0f58f1be failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] 1/1 xapi @ hw Raised at file (Thread 63005 has no backtrace table. Was with_backtraces called?, line 0
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace]
Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||xapi] Unhandled Api_errors.Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                            nraynaudN 1 Reply Last reply Reply Quote 0
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              I'll try to run some tests on my side.

                              1 Reply Last reply Reply Quote 0
                              • stormiS Offline
                                stormi Vates 🪐 XCP-ng Team @borzel
                                last edited by

                                @borzel I'm not very familiar with github wikis, but I think you can do pull requests against the wiki.

                                borzelB 1 Reply Last reply Reply Quote 0
                                • borzelB Offline
                                  borzel XCP-ng Center Team @stormi
                                  last edited by

                                  @stormi wiki issue is allready solved 😉

                                  1 Reply Last reply Reply Quote 0
                                  • nraynaudN Offline
                                    nraynaud XCP-ng Team @borzel
                                    last edited by

                                    @borzel good catch, it's another O_DIRECT issue.

                                    1 Reply Last reply Reply Quote 0
                                    • nraynaudN Offline
                                      nraynaud XCP-ng Team
                                      last edited by

                                      @borzel I am struggling to match the error lines in your log with a source code, can you tell me what version of vhd-tool you have installed please ? (yum info vhd-tool)

                                      borzelB 1 Reply Last reply Reply Quote 0
                                      • borzelB Offline
                                        borzel XCP-ng Center Team @nraynaud
                                        last edited by borzel

                                        @nraynaud

                                        # yum info vhd-tool
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Installed Packages
Name        : vhd-tool
Arch        : x86_64
Version     : 0.20.0
Release     : 4.2.xcp.el7.centos.extras
Size        : 40 M
Repo        : installed
From repo   : xcp-ng-extras
Summary     : Command-line tools for manipulating and
            : streaming .vhd format files
URL         : https://github.com/xapi-project/vhd-tool
License     : LGPL+linking exception
Description : Simple command-line tools for manipulating an
            : streaming .vhd format file.
                                        1 Reply Last reply Reply Quote 0
                                        • nraynaudN Offline
                                          nraynaud XCP-ng Team
                                          last edited by

                                          Thanks,
                                          I have found a suspected O_DIRECT here, the solution won't come quickly.

                                          borzelB 1 Reply Last reply Reply Quote 0
                                          • S Offline
                                            SeyedGH
                                            last edited by

                                            Does this issue also have a role if you run glusterfs on top of zfs?
                                            Or is it just directly on zfs.

                                            Because i would want to runs glusterfs on top of zfs on a spearate server, and use glusterfs driver to connect to it.

                                            Do i also have this issue then?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post