blktap module needed for kernel 4.16. kernel 4.16 needed for latest Ceph with bluestore backend.
-
I understand but from my 7+ years of experience in working with XenServer, I always wanted to have openness about choice of kernel. There are variety reasons one may want their own. E.g. The awesome ZFS. I could have it with plain Xen but XenServer offered extensive APIs.
Now a days kernel is one of the fastest frequently updated (stable release) project with improvisations in all areas. Ceph is my interest area and XenServer team already gave up on it by clearly giving me a NO https://discussions.citrix.com/topic/394865-ceph-luminous-support-for-xenserver-7/.
I think I will continue to see if I can have blktap module work with the new kernel.
-
I completely understand, and that' up to you
- ZFS: I already made working RPMs for XCP-ng, now the rest of the work will come for building the SR driver.
- Would it be possible you have your procedure on how to use a more recent kernel? This way other people could make the choice too!
-
Congratulations on ZFS.
I will contribute the config file used to build latest supportive kernel. Once I know how to get blktap module baked in, it should be straight. Essentially without blktap the tap driver doesn't work which is responsible for VBD plug.
-
Not sure if this is helpful, but I've had success using https://github.com/rposudnevskiy/RBDSR to connect to CEPH using XCP-ng, maybe worth a look?
Though having CEPH support built in like LVMoiSCSI would be great too, and built into XCP-ng even better.
-
That's something we'll consider, yes. Having cutting edge features, maybe without pro support for some of them, but at least let people use it if they like
-
@learningdaily said in blktap module needed for kernel 4.16. kernel 4.16 needed for latest Ceph with bluestore backend.:
Not sure if this is helpful, but I've had success using https://github.com/rposudnevskiy/RBDSR to connect to CEPH using XCP-ng, maybe worth a look?
Though having CEPH support built in like LVMoiSCSI would be great too, and built into XCP-ng even better.
Sure. I will try it out.
-
@olivierlambert I've raised an issue with https://github.com/xapi-project/blktap/issues/245 do you think I can get help?
Those are the only errors stopping to compile the module.
-
That's beyond my knowledge. I suppose you should take a look on XenServer source ISO and see what kind of patches are applied to the dom0 kernel and see if you can build blktap from there.
-
Using DDK I'm able to build blktap from sources for kernel 3.10 but not for 4.16. Reason being Xen handers in 4.16 are improved in ./drivers/xen/ and creates slight incompatibility.
After raising the issue, I found an old patch which upgrades blktap2 to blktap3. Im trying to see if that works. https://github.com/xenserver/buildroot/blob/master/SOURCES/blktap-gntcpy.patch
However I feel upgrading to blktap3 will need me to rebuild a compatible Xen which is another challenge.
All this to have dom0 support latest ceph
I'm feeling I should evaluate other ways. e.g. Having a local VM on host to export ceph or evaluate driver domain concept.
-
Yes, that's why having Ceph kernel support in Dom0 is a pain (maybe smapiv3 will be better on this). That's also why we decided to have only Gluster cliennt in dom0, not the server side.
-
Guys - I could build upstream dom0 kernel (4.16.8) with required blktap2 driver as well as blktap to userspace component with some help from Citrix.
https://github.com/xapi-project/blktap/issues/245
With this I can have ceph Luminous client running in dom0. Did some fio randrw benchmarks and all went well. Created a VM using an rbd image as LVM SR.
I'll figure out a way to publish and maintain this if there is enough interest around.
Cheers. Time to sleep 2.40AM
-
Hey thanks a lot!
As we all can see, if we ask Citrix devs to get more of their patch upstread, they did it! This is excellent news for everyone, and helps to make everything more Open Source in the end
Thanks again for your work @r1 !
-
Yes - it helped to get userspace blktap to work.
However, upstream kernel is completely missing the driver for blktap2. It needed to be fetch from older XS kernel and port on newer one with required modifications. Its location was drivers/block/blktap2
To make this new port usable for everyone I have following options
- Publish kernel-4.16.8-1.src.rpm (unmaintainable)
- Carve out blktap2 driver from kernel code and create a separate rpm for it (most likely)
- Try to get blktap2 in mainline kernel as additional driver (least likely)
Any ideas?
It also needs the patched blktap userspace components which can be built directly from blktap GitHub repo. (procedure)
@olivierlambert may be you can add a "Development" category on main page where this info might be useful.
-
Good point. On it.
Regarding the port itself, IMHO the best solutions in order of preference (not in feasibility
- Get
blktap2in the kernel as a driver (then "nothing to do" for next kernels) - Create a dedicated repo able to generate a RPM from its sources
- Publish a big RPM blob (fine for quick benchmarks, but not maintainable as you said
edit: what are the obstacles, in your opinion, to get to the solution nĀ°1?
- Get
-
As of my understanding from a previous project - a driver can be added to mainline if it has enough use cases in kernel world. Compared to the audience of kernel,
blktap2seems drop in the ocean. Upstream already has Xen whileblktap2is XenServer/XCP-ng specific, thus I think its a hard route.I was reading on DKMS and if this can be given out as DKMS.
Generally, XenServer/XCP-ng doesn't refuse to boot on upstream kernel. It boots well but xapi does not start due to dependency on tapdev. Tapdev is the blktap2 driver which is expected to be present. So if we could ship this as DKMS it can be installed once dom0 is up on any upstream kernel.
This would mean installing 2 packages - DKMS itself from epel and blktap2 driver.
From roadmap perspective, do you have plans to take a vanilla CentOS7 and do
yum install xcp-ng? or you envision that xcp-ng ISO won't go away at least for next year or so? -
We don't have the resources (nor enough community dev) to be able to get CentOS 7 "transformed" into a XCP-ng in short term. There is a lot of patches to apply on top of CentOS various packages.
So we'll focus first on a transparent/automated/testable build of XCP-ng from XS, then with this experience/know-how, we'll explore the "from pure CentOS" option.
We need to prioritize due to our size!
-
Sure. Its not a need as well. I know Citrix tried it years ago but probably left for same reason.
I'll continue to work on DKMS.
Cheers. Enjoy the Sunday!
-
You too!
-
Update -
- Success in running kernel-4.16+ in dom0 with blktap2 driver + Ceph luminous version with bluestore backend.
- Could isolate blktap2 driver from kernel to a loadable kernel module in form of
blktap2-4.16.8-modules-1.0-1.x86_64.rpm - Can install any upstream kernel in dom0 but blktap2 needs to be built against same kernel else xapi won't start.
[root@client ~]# uname -a Linux client.enlight 4.16.13 #1 SMP Wed May 30 09:22:58 GMT 2018 x86_64 x86_64 x86_64 GNU/Linux[root@client ~]# rpm -ivh blktap2-4.16.8-modules-1.0-1.x86_64.rpm
error: Failed dependencies: kernel-uname-r = 4.16.8 is needed by blktap2-4.16.8-modules-1.0-1.x86_64So essentially to make it worth - the only option is to setup a continous build system that will build latest stable kernels and host either full upstream dom0 compatible kernel or just the blktap2 module... may be both
Time to rest blktap2 and move to the actual Ceph productivity integrations.
-
We want to make this CI stuff, that we will be the goal of Sam, our new recruit coming in 20 days!
edit: @r1 you are obviously more than welcome aboard to help us on that!
