XOSTOR hyperconvergence preview

Swen

@olivierlambert did you already had the change to test your new hardware?

We did some more benchmarking with only bonded 40G interface.
We used the following fio command:
fio --name=a --direct=1 --bs=1M --iodepth=32 --ioengine=libaio --rw=write

on bare-metal (OS: Ubuntu 22 LTS) we are able to reach 3100MB/s
on a VM installed on xcp-ng we are able to reach 1200MB/s
on dom0 we are able to reach 1300MB/s when create a new linstor volume and using /dev/drbd directly
on dom0 when using the lvm without drbd we are able to reach 1500MB/s

And btw it looks like tapdisk is our bottleneck on dom0 as you suggested before, because this is a single-thread process and our CPU reached a limit.

From the numbers above the performance inside the VM does not look as bad as we thought at the beginning. The only question we have at this moment is why we are "loosing" over half of the performance between a bare-metal installation and when testing the same storage from within dom0?
Is this expected behavior?

gb.123

This post is deleted!

olivierlambert

@Swen Yes, it is expected. The Dom0 is NOT the baremetal but a VM in PV mode. Also, it doesn't have all the resources of the host (as a VM, only a fraction of CPUs/memory, but also not -for now- a recent kernel and IO uring.

ronan-a

@gb-123 said in XOSTOR hyperconvergence preview:

@ronan-a

running linstor command on both hosts gives the following error :

Traceback (most recent call last):
File "/usr/bin/linstor", line 21, in <module>
import linstor_client_main
ImportError: No module named linstor_client_main

Regarding this error, can you say me what's your XCP-ng version? And what's the output of: yum list installed | grep -i linstor? Do you have a XCP-ng 8.3? And did you run the beta installation script on it?

Also, your SSL/VLAN issue is not caused by LINSTOR, @BenjiReis do you have any idea?

gb.123

@ronan-a said in XOSTOR hyperconvergence preview:

@gb-123 said in XOSTOR hyperconvergence preview:

@ronan-a

running linstor command on both hosts gives the following error :

Traceback (most recent call last):
File "/usr/bin/linstor", line 21, in <module>
import linstor_client_main
ImportError: No module named linstor_client_main

Regarding this error, can you say me what's your XCP-ng version? And what's the output of: yum list installed | grep -i linstor?

Output of yum list installed | grep -i linstor :

drbd.x86_64                     9.22.0-1.el7               @xcp-ng-linstor      
drbd-bash-completion.x86_64     9.22.0-1.el7               @xcp-ng-linstor      
drbd-pacemaker.x86_64           9.22.0-1.el7               @xcp-ng-linstor      
drbd-reactor.x86_64             1.0.0-1                    @xcp-ng-linstor      
drbd-udev.x86_64                9.22.0-1.el7               @xcp-ng-linstor      
drbd-utils.x86_64               9.22.0-1.el7               @xcp-ng-linstor      
drbd-xen.x86_64                 9.22.0-1.el7               @xcp-ng-linstor      
kmod-drbd.x86_64                9.2.2+ptf.1_4.19.0+1-1     @xcp-ng-linstor      
linstor-client.noarch           1.18.0-1                   @xcp-ng-linstor      
linstor-common.noarch           1.21.1-1.el7               @xcp-ng-linstor      
linstor-controller.noarch       1.21.1-1.el7               @xcp-ng-linstor      
linstor-satellite.noarch        1.21.1-1.el7               @xcp-ng-linstor      
python-linstor.noarch           1.18.0-1                   @xcp-ng-linstor      
xcp-ng-linstor.noarch           1.2-1.xcpng8.3             @xcp-ng-linstor      
xcp-ng-release-linstor.noarch   1.4-1.xcpng8.3             @xcp-ng-base

Do you have a XCP-ng 8.3? And did you run the beta installation script on it?

I installed XCP-ng 8.3 beta from ISO and then applied all the patches that were available. To install linstor, I followed the 1st Post of this topic.

Also, your SSL/VLAN issue is not caused by LINSTOR, @BenjiReis do you have any idea?

Yeah, even I thought so, but nothing changed for me before I installed Linstor.

There is also a very peculiar thing I am noticing:
Backup from Thin (Ext4) SR to another Thin (Ext4) SR has a transfer of 120KiBs over network(WAN). I thought this is maybe due to a 'slow' network, but however, I started another backup at the same time while the previous one was running slowly, this time from Thick (LVM) SR to Thin SR and the speed I got was around 16 MiB/s (which seems to be OK). I did write a post regarding this but deleted it since I need to dig deeper into this before I report this as an issue to you.
I will be conducting tests for a few more days just to be sure. (I am talking about Continuous Backup btw.)

Update :
After a week of continuous testing, I am getting mixed results on the backup speed as mentioned in the above paragraph. I can now say that the issue may not be related to Linstor.

Maelstrom96

Is there a procedure on how we can update our current 8.2 XCP-ng cluster to 8.3? My undertanding is that if I update the host using the ISO, it will effectively wipe all changes that were made to DOM0, including the linstor/sm-linstor packages.

BenjiReis

@gb-123 are you using XOA or XO from sources?
If from sources the issue might come from having a different openssl when creating the sdn-controller certificates.
You can either try with an XOA or generate manually your certificates.

gb.123

@BenjiReis said in XOSTOR hyperconvergence preview:

@gb-123 are you using XOA or XO from sources?
If from sources the issue might come from having a different openssl when creating the sdn-controller certificates.
You can either try with an XOA or generate manually your certificates.

Using XO from sources. I just turned on "Over-ride certificates" and reinstalled the whole XO virtual machine. Seems to work fine now.

My only problem was that why it suddenly stopped working when I installed Linstore as installing Linstor should not impact upon this. So thats why I reported this on this thread.

BenjiReis

@gb-123 I see thanks

Just bad timing imho because Linstor doesn't touch this part of the host and the openssl issue is more probably coming from the env when you run your XO.

anyway, glad it's working now!

gb.123

@ronan-a

Any idea why I am getting the error while running the linstor command ?

Thanks

ronan-a

@gb-123 What command and error exactly?

Maelstrom96

@Maelstrom96 said in XOSTOR hyperconvergence preview:

Is there a procedure on how we can update our current 8.2 XCP-ng cluster to 8.3? My undertanding is that if I update the host using the ISO, it will effectively wipe all changes that were made to DOM0, including the linstor/sm-linstor packages.

Any input on this @ronan-a?

olivierlambert

@ronan-a is in holidays for the next 2 weeks

However, I can answer: I'm currently using XOSTOR on a test cluster running on a fresh 8.3 and it works Can't tell for the upgrade though.

gb.123

@ronan-a

@ronan-a said in XOSTOR hyperconvergence preview:

@gb-123 What command and error exactly?

I was referring to the our earlier conversation :

@gb-123 said in XOSTOR hyperconvergence preview:

@ronan-a

running linstor command on both hosts gives the following error :

Traceback (most recent call last):
File "/usr/bin/linstor", line 21, in <module>
import linstor_client_main
ImportError: No module named linstor_client_main

@ronan-a said in XOSTOR hyperconvergence preview:

Regarding this error, can you say me what's your XCP-ng version? And what's the output of: yum list installed | grep -i linstor?

Output of yum list installed | grep -i linstor :
drbd.x86_64                     9.22.0-1.el7               @xcp-ng-linstor      
drbd-bash-completion.x86_64     9.22.0-1.el7               @xcp-ng-linstor      
drbd-pacemaker.x86_64           9.22.0-1.el7               @xcp-ng-linstor      
drbd-reactor.x86_64             1.0.0-1                    @xcp-ng-linstor      
drbd-udev.x86_64                9.22.0-1.el7               @xcp-ng-linstor      
drbd-utils.x86_64               9.22.0-1.el7               @xcp-ng-linstor      
drbd-xen.x86_64                 9.22.0-1.el7               @xcp-ng-linstor      
kmod-drbd.x86_64                9.2.2+ptf.1_4.19.0+1-1     @xcp-ng-linstor      
linstor-client.noarch           1.18.0-1                   @xcp-ng-linstor      
linstor-common.noarch           1.21.1-1.el7               @xcp-ng-linstor      
linstor-controller.noarch       1.21.1-1.el7               @xcp-ng-linstor      
linstor-satellite.noarch        1.21.1-1.el7               @xcp-ng-linstor      
python-linstor.noarch           1.18.0-1                   @xcp-ng-linstor      
xcp-ng-linstor.noarch           1.2-1.xcpng8.3             @xcp-ng-linstor      
xcp-ng-release-linstor.noarch   1.4-1.xcpng8.3             @xcp-ng-base
@ronan-a said in XOSTOR hyperconvergence preview:

Do you have a XCP-ng 8.3? And did you run the beta installation script on it?

I installed XCP-ng 8.3 beta from ISO and then applied all the patches that were available. To install linstor, I followed the 1st Post of this topic.

ronan-a

@gb-123 If you run LINSTOR on 8.3 (FYI, linstor packages on this version are considered unstable), you must call the linstor command like that:

> python3 /usr/bin/linstor

By default python2 is called and is not compatible with this version of LINSTOR:

> python2 /usr/bin/linstor resource list
Traceback (most recent call last):
  File "/usr/bin/linstor", line 21, in <module>
    import linstor_client_main
ImportError: No module named linstor_client_main

ronan-a

@Maelstrom96 We must update our documentation for that, This will probably require executing commands manually during an upgrade.

gb.123

@ronan-a

Are there any other packages in xcp-ng requiring python2 ?
Would it be advisable to change default python to python3 (by adding alias in bashrc) ?
Or do you think I should create a script which in-turn runs python3 command only for linstor ?

PS: By executing :

python3 /usr/bin/linstor

Result :

Error: Unable to connect to linstor://localhost:3370: [Errno 99] Cannot assign requested address

Although through XO, I can see the VM on Linstor SR running fine,

ronan-a

@gb-123 said in XOSTOR hyperconvergence preview:

Are there any other packages in xcp-ng requiring python2 ?

Probably yes for few packages.

Would it be advisable to change default python to python3 (by adding alias in bashrc) ?

I don't recommend to do that.

Or do you think I should create a script which in-turn runs python3 command only for linstor ?

It's probably the best solution for the moment. Our ideal goal would obviously to only have Python 3:

But currently XCP-ng 8.3 is still in development.
And there are still things to check before thinking about globally changing the default Python interpreter.

gb.123

@ronan-a

Any idea why I am getting :

Error: Unable to connect to linstor://localhost:3370: [Errno 99] Cannot assign requested address

I installed linstor with replication as 2 and the VM is running fine. I am executing this command from dom0 of Pool Master.

Swen

@gb-123 pool master does not need to be linstor controller. Try this command on the other hosts. It will only work on the node which is the linstor controller.