XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Testing ZFS with XCP-ng

    Scheduled Pinned Locked Moved Development
    80 Posts 10 Posters 40.0k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • borzelB Offline
      borzel XCP-ng Center Team @buzzzo
      last edited by borzel

      @buzzzo said in Testing ZFS with XCP-ng:

      Zfs is a local sr and imho does not add too much advantages vs classic ext sr

      meeeep 🙂

      • ZFS uses (in standard configuration) half the RAM as read cache
      • with sync=disabled your writes are faster (but synced to discs every 5 seconds; in one sequential batch - efficient use of slow spinning drives!)
      • it has checksums from fs-layer down to the device
      • it's awesome 😉

      know xoa already offer a similar solution but is snapshot based and everyone knows the performance and size problem with this approach.

      If we all had 48 hours a day and more people in the team, we could fully integrate ZFS into XCP-ng (with all bells and whistles). But for now we are happy to have the first step done - loose integrate ZFS to start using it and testing stability of the dom0 with ZFS in it.

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Online
        olivierlambert Vates 🪐 Co-Founder CEO @buzzzo
        last edited by olivierlambert

        @buzzzo Double meeeep 😆

        On file level SR (ext, NFS, local ZFS, XOSAN), there is no space issue doing snapshots. Also, perfs are good even with snapshots if you don't have an asthmatic SR.

        B 1 Reply Last reply Reply Quote 0
        • B Offline
          buzzzo @olivierlambert
          last edited by

          @olivierlambert meeep!!! ☺

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Online
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Meep meep!

            1 Reply Last reply Reply Quote 1
            • borzelB Offline
              borzel XCP-ng Center Team
              last edited by borzel

              I tested latest and greatest XO (5.24.1 from sources) with latest XCP-ng 7.5 (incl. xcp-ng-extra repo/ZFS). Backups do not work 😞

              If the VDI is on a ext-SR, all is fine, backup working.
              If the VDI is on a File-SR (ZFS based), the backup says "VDI_ERROR" or "VDI Chain to long". I will provide logfiles tomorrow if needed.

              Setup of my File-SR:
              xe sr-create host-uuid=<xxx> name-label=zfssr type=file other-config:o_direct=false device-config:location=/zfspool/zfssr

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Online
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                VDI chain too long doesn't seems related. It means you have a VDI chain that's around 30 files.

                Can you provide the exact/detailed error log?

                1 Reply Last reply Reply Quote 0
                • borzelB Offline
                  borzel XCP-ng Center Team
                  last edited by borzel

                  0_1534286419566_560b7fad-965d-42bc-8f40-2eed276163b8-grafik.png

                  1 Reply Last reply Reply Quote 0
                  • borzelB Offline
                    borzel XCP-ng Center Team
                    last edited by borzel

                    I searched in xensource.log, SMlog and syslog, no related entries found 😞

                    How can I increase the loglevel of XO?

                    Here's the log output if I start this job again:

                    Aug 15 01:10:35 ubuntu xo-server[616]: 2018-08-14T23:10:35.241Z xo:xapi Snapshotting VM pmg as [XO Backup test] pmg
                    Aug 15 01:10:36 ubuntu xo-server[616]: 2018-08-14T23:10:36.948Z xo:xapi exporting VDI pmg 0
                    Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.289Z xo:xapi Deleting VM [XO Backup test] pmg
                    Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.585Z xo:xapi Deleting VDI pmg 0
                    

                    It's nearly nothing...

                    1 Reply Last reply Reply Quote 0
                    • nraynaudN Offline
                      nraynaud XCP-ng Team
                      last edited by

                      @borzel > The error is often in the Settings->Logs panel in XO in this case

                      borzelB 1 Reply Last reply Reply Quote 0
                      • borzelB Offline
                        borzel XCP-ng Center Team @nraynaud
                        last edited by

                        @nraynaud no, it's as empty as it could be 😞

                        1 Reply Last reply Reply Quote 0
                        • borzelB Offline
                          borzel XCP-ng Center Team
                          last edited by

                          I found something in /var/log/xensource.log (used tail -f /var/log/xensource.log | grep -i 'error')

                          Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR
                          Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool output: vhd-tool: internal error, uncaught exception:#012          Unix.Unix_error(Unix.EINVAL, "open", "/var/run/sr-mount/9fc67ea3-c9b3-5e62-ecc3-7d1debf81744/6781ab7b-c475-4d72-a7d0-b2aeca2e81e3.vhd")#012          Raised at file "src/core/lwt.ml", line 3008, characters 20-29#012          Called from file "src/unix/lwt_main.ml", line 42, characters 8-18#012          Called from file "src/impl.ml", line 811, characters 4-23#012          Called from file "src/cmdliner_term.ml", line 27, characters 19-24#012          Called from file "src/cmdliner.ml", line 27, characters 27-34#012          Called from file "src/cmdliner.ml", line 106, characters 32-39
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] [XO] VDI Export R:3001f7e5aa17 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 1/13 xapi @ hw Raised at file ocaml/xapi/vhd_tool_wrapper.ml, line 59
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 4/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 47
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 6/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 7/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 54
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 8/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 65
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace]
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] VDI.export_raw_vdi D:315adbf2ec6c failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 1/13 xapi @ hw Raised at file lib/debug.ml, line 187
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 4/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 6/13 xapi @ hw Called from file ocaml/xapi/xapi_http.ml, line 185
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 7/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 8/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace]
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] VDI.export_raw_vdi D:fc8d0f58f1be failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] 1/1 xapi @ hw Raised at file (Thread 63005 has no backtrace table. Was with_backtraces called?, line 0
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace]
                          Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||xapi] Unhandled Api_errors.Server_error(VDI_IO_ERROR, [ Device I/O errors ])
                          
                          nraynaudN 1 Reply Last reply Reply Quote 0
                          • olivierlambertO Online
                            olivierlambert Vates 🪐 Co-Founder CEO
                            last edited by

                            I'll try to run some tests on my side.

                            1 Reply Last reply Reply Quote 0
                            • stormiS Offline
                              stormi Vates 🪐 XCP-ng Team @borzel
                              last edited by

                              @borzel I'm not very familiar with github wikis, but I think you can do pull requests against the wiki.

                              borzelB 1 Reply Last reply Reply Quote 0
                              • borzelB Offline
                                borzel XCP-ng Center Team @stormi
                                last edited by

                                @stormi wiki issue is allready solved 😉

                                1 Reply Last reply Reply Quote 0
                                • nraynaudN Offline
                                  nraynaud XCP-ng Team @borzel
                                  last edited by

                                  @borzel good catch, it's another O_DIRECT issue.

                                  1 Reply Last reply Reply Quote 0
                                  • nraynaudN Offline
                                    nraynaud XCP-ng Team
                                    last edited by

                                    @borzel I am struggling to match the error lines in your log with a source code, can you tell me what version of vhd-tool you have installed please ? (yum info vhd-tool)

                                    borzelB 1 Reply Last reply Reply Quote 0
                                    • borzelB Offline
                                      borzel XCP-ng Center Team @nraynaud
                                      last edited by borzel

                                      @nraynaud

                                      # yum info vhd-tool
                                      Loaded plugins: fastestmirror
                                      Loading mirror speeds from cached hostfile
                                      Installed Packages
                                      Name        : vhd-tool
                                      Arch        : x86_64
                                      Version     : 0.20.0
                                      Release     : 4.2.xcp.el7.centos.extras
                                      Size        : 40 M
                                      Repo        : installed
                                      From repo   : xcp-ng-extras
                                      Summary     : Command-line tools for manipulating and
                                                  : streaming .vhd format files
                                      URL         : https://github.com/xapi-project/vhd-tool
                                      License     : LGPL+linking exception
                                      Description : Simple command-line tools for manipulating an
                                                  : streaming .vhd format file.
                                      
                                      1 Reply Last reply Reply Quote 0
                                      • nraynaudN Offline
                                        nraynaud XCP-ng Team
                                        last edited by

                                        Thanks,
                                        I have found a suspected O_DIRECT here, the solution won't come quickly.

                                        borzelB 1 Reply Last reply Reply Quote 0
                                        • S Offline
                                          SeyedGH
                                          last edited by

                                          Does this issue also have a role if you run glusterfs on top of zfs?
                                          Or is it just directly on zfs.

                                          Because i would want to runs glusterfs on top of zfs on a spearate server, and use glusterfs driver to connect to it.

                                          Do i also have this issue then?

                                          1 Reply Last reply Reply Quote 0
                                          • borzelB Offline
                                            borzel XCP-ng Center Team @nraynaud
                                            last edited by

                                            @nraynaud I just tested with an up-to-date XCP-ng 7.5 host, same failure

                                            # yum info vhd-tool
                                            Loaded plugins: fastestmirror
                                            Loading mirror speeds from cached hostfile
                                            Installed Packages
                                            Name        : vhd-tool
                                            Arch        : x86_64
                                            Version     : 0.20.0
                                            Release     : 4.3.xcp.el7.centos
                                            Size        : 40 M
                                            Repo        : installed
                                            From repo   : xcp-ng-updates
                                            Summary     : Command-line tools for manipulating and streaming .vhd format files
                                            URL         : https://github.com/xapi-project/vhd-tool
                                            License     : LGPL+linking exception
                                            Description : Simple command-line tools for manipulating and streaming .vhd format file.
                                            
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post