XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    blktap module needed for kernel 4.16. kernel 4.16 needed for latest Ceph with bluestore backend.

    Scheduled Pinned Locked Moved Development
    35 Posts 4 Posters 12.2k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R Offline
      r1 XCP-ng Team
      last edited by

      Hello All,

      Feels great to be part of this community and finally being able to openly work on my favorite xenserver.

      I'm putting efforts to make CEPH work as a shared storage like LVMoiSCSI. I built a CEPH cluster using few SATA, SSD and NVMEs.. However the default 3.10 (XenServer) or 4.4 (XCP) seems to mismatch featureset required to mount RBD.

      I've upgraded the kernel on client to 4.15.6 and plan to keep up to latest stable. Good news is that I'm able to mount and use /dev/rbd0. However, I'm stuck at blktap patch for these. Hence, not able to create VDIs.

      Any help would be appreciated.

      Thanks,
      Rishi

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        I'm not sure to understand what are you doing exactly. Let me try to recap: you want to connect your Ceph cluster in iSCSI (using lvmoiscsi "driver" of XS/XCP), but it doesn't work?

        1 Reply Last reply Reply Quote 0
        • R Offline
          r1 XCP-ng Team
          last edited by

          Apologies for creating confusion by mentioning "LVMoiSCSI".

          I'm using rbd kernel module to map ceph disks to xcp-ng/xenserver client. This kernel module is present in kernel 3.10 (xenserver) as well as kernel 4.4 (xcp-ng). However its not latest. And thus gives me feature mismatch error to map a disk.

          I've compiled a recent 4.15.6 kernel which installs on both xenserver and xcp-ng. It is able to map ceph disks using rbd module however, it lacks blktap module to be able to mange VDIs.

          Essentially, I need to add blktap module to my custom kernel.

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Compiling your own kernel in dom0: that's not a good idea in general, because it's not tested at all. There is also maybe patches from Citrix/community on the host kernel, so there is no guarantee you won't open hell's door by doing this.

            So if I understood correctly, you must have a very recent kernel to connect to your Ceph cluster?

            1 Reply Last reply Reply Quote 0
            • R Offline
              r1 XCP-ng Team
              last edited by

              Yes. At least to let all features work properly, I need to have latest kernel.

              Either I will have to back port ceph or upgrade dom0 kernel to recent with blktap patch.

              I do not think having latest kernel is an issue for XenServer since Xen is fine with it. Other special device drivers may run into issues but I'm using generic ones. These are well supported on latest kernel.

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Ideally, I'd prefer that everyone use the same kernel with the right stuff in it, because it would means we'll lose community effort otherwise. I doubt everyone is able to use a recent kernel in XCP-ng (I'm including myself šŸ˜› )

                1 Reply Last reply Reply Quote 0
                • R Offline
                  r1 XCP-ng Team
                  last edited by

                  I understand but from my 7+ years of experience in working with XenServer, I always wanted to have openness about choice of kernel. There are variety reasons one may want their own. E.g. The awesome ZFS. I could have it with plain Xen but XenServer offered extensive APIs.

                  Now a days kernel is one of the fastest frequently updated (stable release) project with improvisations in all areas. Ceph is my interest area and XenServer team already gave up on it by clearly giving me a NO https://discussions.citrix.com/topic/394865-ceph-luminous-support-for-xenserver-7/.

                  I think I will continue to see if I can have blktap module work with the new kernel.

                  1 Reply Last reply Reply Quote 0
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    I completely understand, and that' up to you šŸ™‚

                    • ZFS: I already made working RPMs for XCP-ng, now the rest of the work will come for building the SR driver.
                    • Would it be possible you have your procedure on how to use a more recent kernel? This way other people could make the choice too!
                    1 Reply Last reply Reply Quote 0
                    • R Offline
                      r1 XCP-ng Team
                      last edited by

                      Congratulations on ZFS.

                      I will contribute the config file used to build latest supportive kernel. Once I know how to get blktap module baked in, it should be straight. Essentially without blktap the tap driver doesn't work which is responsible for VBD plug.

                      1 Reply Last reply Reply Quote 0
                      • L Offline
                        learningdaily
                        last edited by

                        Not sure if this is helpful, but I've had success using https://github.com/rposudnevskiy/RBDSR to connect to CEPH using XCP-ng, maybe worth a look?

                        Though having CEPH support built in like LVMoiSCSI would be great too, and built into XCP-ng even better.

                        R 1 Reply Last reply Reply Quote 0
                        • olivierlambertO Offline
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          That's something we'll consider, yes. Having cutting edge features, maybe without pro support for some of them, but at least let people use it if they like šŸ™‚

                          1 Reply Last reply Reply Quote 0
                          • R Offline
                            r1 XCP-ng Team @learningdaily
                            last edited by

                            @learningdaily said in blktap module needed for kernel 4.16. kernel 4.16 needed for latest Ceph with bluestore backend.:

                            Not sure if this is helpful, but I've had success using https://github.com/rposudnevskiy/RBDSR to connect to CEPH using XCP-ng, maybe worth a look?

                            Though having CEPH support built in like LVMoiSCSI would be great too, and built into XCP-ng even better.

                            Sure. I will try it out.

                            1 Reply Last reply Reply Quote 0
                            • R Offline
                              r1 XCP-ng Team
                              last edited by

                              @olivierlambert I've raised an issue with https://github.com/xapi-project/blktap/issues/245 do you think I can get help?

                              Those are the only errors stopping to compile the module.

                              rushikeshjadhav created this issue in xapi-project/blktap

                              closed Unable to compile blktap against latest stable kernel #245

                              1 Reply Last reply Reply Quote 0
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                That's beyond my knowledge. I suppose you should take a look on XenServer source ISO and see what kind of patches are applied to the dom0 kernel and see if you can build blktap from there.

                                1 Reply Last reply Reply Quote 0
                                • R Offline
                                  r1 XCP-ng Team
                                  last edited by

                                  Using DDK I'm able to build blktap from sources for kernel 3.10 but not for 4.16. Reason being Xen handers in 4.16 are improved in ./drivers/xen/ and creates slight incompatibility.

                                  After raising the issue, I found an old patch which upgrades blktap2 to blktap3. Im trying to see if that works. https://github.com/xenserver/buildroot/blob/master/SOURCES/blktap-gntcpy.patch

                                  However I feel upgrading to blktap3 will need me to rebuild a compatible Xen which is another challenge.

                                  All this to have dom0 support latest ceph šŸ˜ž

                                  I'm feeling I should evaluate other ways. e.g. Having a local VM on host to export ceph or evaluate driver domain concept.

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO Offline
                                    olivierlambert Vates 🪐 Co-Founder CEO
                                    last edited by

                                    Yes, that's why having Ceph kernel support in Dom0 is a pain (maybe smapiv3 will be better on this). That's also why we decided to have only Gluster cliennt in dom0, not the server side.

                                    1 Reply Last reply Reply Quote 0
                                    • R Offline
                                      r1 XCP-ng Team
                                      last edited by

                                      Guys - I could build upstream dom0 kernel (4.16.8) with required blktap2 driver as well as blktap to userspace component with some help from Citrix.

                                      https://github.com/xapi-project/blktap/issues/245

                                      With this I can have ceph Luminous client running in dom0. Did some fio randrw benchmarks and all went well. Created a VM using an rbd image as LVM SR.

                                      I'll figure out a way to publish and maintain this if there is enough interest around.

                                      Cheers. Time to sleep 2.40AM šŸ˜„

                                      rushikeshjadhav created this issue in xapi-project/blktap

                                      closed Unable to compile blktap against latest stable kernel #245

                                      1 Reply Last reply Reply Quote 0
                                      • olivierlambertO Offline
                                        olivierlambert Vates 🪐 Co-Founder CEO
                                        last edited by

                                        Hey thanks a lot!

                                        As we all can see, if we ask Citrix devs to get more of their patch upstread, they did it! This is excellent news for everyone, and helps to make everything more Open Source in the end šŸ™‚

                                        Thanks again for your work @r1 !

                                        1 Reply Last reply Reply Quote 0
                                        • R Offline
                                          r1 XCP-ng Team
                                          last edited by

                                          Yes - it helped to get userspace blktap to work. šŸ™‚

                                          However, upstream kernel is completely missing the driver for blktap2. It needed to be fetch from older XS kernel and port on newer one with required modifications. Its location was drivers/block/blktap2

                                          To make this new port usable for everyone I have following options

                                          1. Publish kernel-4.16.8-1.src.rpm (unmaintainable)
                                          2. Carve out blktap2 driver from kernel code and create a separate rpm for it (most likely)
                                          3. Try to get blktap2 in mainline kernel as additional driver (least likely)

                                          Any ideas?

                                          It also needs the patched blktap userspace components which can be built directly from blktap GitHub repo. (procedure)

                                          @olivierlambert may be you can add a "Development" category on main page where this info might be useful.

                                          1 Reply Last reply Reply Quote 0
                                          • olivierlambertO Offline
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by olivierlambert

                                            Good point. On it.

                                            Regarding the port itself, IMHO the best solutions in order of preference (not in feasibility šŸ˜‰ šŸ˜ž

                                            1. Get blktap2 in the kernel as a driver (then "nothing to do" for next kernels)
                                            2. Create a dedicated repo able to generate a RPM from its sources
                                            3. Publish a big RPM blob (fine for quick benchmarks, but not maintainable as you said

                                            edit: what are the obstacles, in your opinion, to get to the solution n°1?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post