Testing ZFS with XCP-ng


  • XCP-ng Team

    VDI chain too long doesn't seems related. It means you have a VDI chain that's around 30 files.

    Can you provide the exact/detailed error log?


  • XCP-ng Center Team

    0_1534286419566_560b7fad-965d-42bc-8f40-2eed276163b8-grafik.png


  • XCP-ng Center Team

    I searched in xensource.log, SMlog and syslog, no related entries found 😞

    How can I increase the loglevel of XO?

    Here's the log output if I start this job again:

    Aug 15 01:10:35 ubuntu xo-server[616]: 2018-08-14T23:10:35.241Z xo:xapi Snapshotting VM pmg as [XO Backup test] pmg
    Aug 15 01:10:36 ubuntu xo-server[616]: 2018-08-14T23:10:36.948Z xo:xapi exporting VDI pmg 0
    Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.289Z xo:xapi Deleting VM [XO Backup test] pmg
    Aug 15 01:10:41 ubuntu xo-server[616]: 2018-08-14T23:10:41.585Z xo:xapi Deleting VDI pmg 0
    

    It's nearly nothing...


  • XCP-ng Team

    @borzel > The error is often in the Settings->Logs panel in XO in this case


  • XCP-ng Center Team

    @nraynaud no, it's as empty as it could be 😞


  • XCP-ng Center Team

    I found something in /var/log/xensource.log (used tail -f /var/log/xensource.log | grep -i 'error')

    Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool failed, returning VDI_IO_ERROR
    Aug 14 23:19:10 hw xapi: [error|hw|63005 INET :::80|[XO] VDI Export R:3001f7e5aa17|vhd_tool_wrapper] vhd-tool output: vhd-tool: internal error, uncaught exception:#012          Unix.Unix_error(Unix.EINVAL, "open", "/var/run/sr-mount/9fc67ea3-c9b3-5e62-ecc3-7d1debf81744/6781ab7b-c475-4d72-a7d0-b2aeca2e81e3.vhd")#012          Raised at file "src/core/lwt.ml", line 3008, characters 20-29#012          Called from file "src/unix/lwt_main.ml", line 42, characters 8-18#012          Called from file "src/impl.ml", line 811, characters 4-23#012          Called from file "src/cmdliner_term.ml", line 27, characters 19-24#012          Called from file "src/cmdliner.ml", line 27, characters 27-34#012          Called from file "src/cmdliner.ml", line 106, characters 32-39
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] [XO] VDI Export R:3001f7e5aa17 failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 1/13 xapi @ hw Raised at file ocaml/xapi/vhd_tool_wrapper.ml, line 59
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 4/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 47
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 6/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 7/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 54
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 8/13 xapi @ hw Called from file ocaml/xapi/export_raw_vdi.ml, line 65
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:315adbf2ec6c|backtrace]
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] VDI.export_raw_vdi D:315adbf2ec6c failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 1/13 xapi @ hw Raised at file lib/debug.ml, line 187
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 2/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 3/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 4/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 5/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 6/13 xapi @ hw Called from file ocaml/xapi/xapi_http.ml, line 185
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 7/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 8/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 9/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 73
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 10/13 xapi @ hw Called from file ocaml/xapi/server_helpers.ml, line 91
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 11/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 22
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 12/13 xapi @ hw Called from file lib/xapi-stdext-pervasives/pervasiveext.ml, line 26
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace] 13/13 xapi @ hw Called from file lib/backtrace.ml, line 177
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80|VDI.export_raw_vdi D:fc8d0f58f1be|backtrace]
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] VDI.export_raw_vdi D:fc8d0f58f1be failed with exception Server_error(VDI_IO_ERROR, [ Device I/O errors ])
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] Raised Server_error(VDI_IO_ERROR, [ Device I/O errors ])
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace] 1/1 xapi @ hw Raised at file (Thread 63005 has no backtrace table. Was with_backtraces called?, line 0
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||backtrace]
    Aug 14 23:19:11 hw xapi: [error|hw|63005 INET :::80||xapi] Unhandled Api_errors.Server_error(VDI_IO_ERROR, [ Device I/O errors ])
    

  • XCP-ng Team

    I'll try to run some tests on my side.


  • XCP-ng Team

    @borzel I'm not very familiar with github wikis, but I think you can do pull requests against the wiki.


  • XCP-ng Center Team

    @stormi wiki issue is allready solved 😉


  • XCP-ng Team

    @borzel good catch, it's another O_DIRECT issue.


  • XCP-ng Team

    @borzel I am struggling to match the error lines in your log with a source code, can you tell me what version of vhd-tool you have installed please ? (yum info vhd-tool)


  • XCP-ng Center Team

    @nraynaud

    # yum info vhd-tool
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    Installed Packages
    Name        : vhd-tool
    Arch        : x86_64
    Version     : 0.20.0
    Release     : 4.2.xcp.el7.centos.extras
    Size        : 40 M
    Repo        : installed
    From repo   : xcp-ng-extras
    Summary     : Command-line tools for manipulating and
                : streaming .vhd format files
    URL         : https://github.com/xapi-project/vhd-tool
    License     : LGPL+linking exception
    Description : Simple command-line tools for manipulating an
                : streaming .vhd format file.
    

  • XCP-ng Team

    Thanks,
    I have found a suspected O_DIRECT here, the solution won't come quickly.



  • Does this issue also have a role if you run glusterfs on top of zfs?
    Or is it just directly on zfs.

    Because i would want to runs glusterfs on top of zfs on a spearate server, and use glusterfs driver to connect to it.

    Do i also have this issue then?


  • XCP-ng Center Team

    @nraynaud I just tested with an up-to-date XCP-ng 7.5 host, same failure

    # yum info vhd-tool
    Loaded plugins: fastestmirror
    Loading mirror speeds from cached hostfile
    Installed Packages
    Name        : vhd-tool
    Arch        : x86_64
    Version     : 0.20.0
    Release     : 4.3.xcp.el7.centos
    Size        : 40 M
    Repo        : installed
    From repo   : xcp-ng-updates
    Summary     : Command-line tools for manipulating and streaming .vhd format files
    URL         : https://github.com/xapi-project/vhd-tool
    License     : LGPL+linking exception
    Description : Simple command-line tools for manipulating and streaming .vhd format file.
    

  • XCP-ng Team

    I am till on the issue, the subject is way deeper than expected.


  • XCP-ng Center Team

    @nraynaud take your time, no rush


  • XCP-ng Center Team


  • XCP-ng Team

    Thanks @borzel



  • Hello everybody! Newbie here, just 24h of xcp use, I was trying set Raid 10, but xcp only support Raid 1, so I'm trying now to set raidz, like on the document, I'm using 4 disks (500gb each), but I want to do Raid on this 4 disks (sda,sdb,sdc,sdd), so I can't? Need to be 6 disks? 2 raid 1 with xcp, and 4 raidz for other things?


Log in to reply
 

XCP-ng Pro Support

XCP-ng Pro Support