Posts made by Swen

@gb-123 pool master does not need to be linstor controller. Try this command on the other hosts. It will only work on the node which is the linstor controller.

@olivierlambert did you already had the change to test your new hardware?

We did some more benchmarking with only bonded 40G interface.
We used the following fio command:
fio --name=a --direct=1 --bs=1M --iodepth=32 --ioengine=libaio --rw=write

1. on bare-metal (OS: Ubuntu 22 LTS) we are able to reach 3100MB/s
2. on a VM installed on xcp-ng we are able to reach 1200MB/s
3. on dom0 we are able to reach 1300MB/s when create a new linstor volume and using /dev/drbd directly
4. on dom0 when using the lvm without drbd we are able to reach 1500MB/s

And btw it looks like tapdisk is our bottleneck on dom0 as you suggested before, because this is a single-thread process and our CPU reached a limit.

From the numbers above the performance inside the VM does not look as bad as we thought at the beginning. The only question we have at this moment is why we are "loosing" over half of the performance between a bare-metal installation and when testing the same storage from within dom0?
Is this expected behavior?

@olivierlambert just to be sure: we did also use your recommended fio parameters with the exact same results. We used fio from inside a VM not from inside dom0. My comments regarding waits inside the VM and no waits in dom0 was just additional information.

I am aware of possible bottleneck like latency, SSD others, but in our case we can rule them out. Reason is that we double our speed when switching from 10G to 40G interface while the rest is the exact same configuration. As fsr as I can see this looks to me like xcp-ng is the bottleneck and limiting bandwidth of the interface in some way. Even the numbers you provided are not really good performance numbers. Did you get more bandwidth than 8 Gbits over the linstor interface?

We are going to install Ubuntu on the same servers and install linstor on it to test our infrastructure on bare-metal without any hypervisor to see if it is xcp-ng related or not.

@olivierlambert thx for the feedback! I do not get how you see that you reach 20G on your nic. Can you please explain it? I see that you reach 2600MiB/s in read, but this is more likely on local disk, isn't? What I can see in our lab environment is that for what ever reason we do not get more than around 8Gbits on pass-through via a 40G interface and 4Gbits via a 10G interface and therefore we do not get any good performance out of the storage repository. I am unable to find the root cause of this. Do you have any idea where to look? I can see high waits on the OS of the VM, but no waits inside dom0 of any node.

hi @ronan-a,
we did some performance testing with the latest version and we run into a bottleneck we are unable to identify in detail.

Here is our setup:
Dell R730
CPU: 2x Intel E5-2680v4
RAM: 384GB
Storage: 2x NVMe Samsung PM9A3 3.84TB via U.2 PCIe 3 x16 Extender Card
NICs: 2x 10G Intel, 2x 40G Intel

We have 3 servers with the same configuration and installed them as a cluster with replica count of 2.
xcp-ng 8.2 with latest patches is installed. All servers are using the same switch (2x QFX5100-24Q, configured as virtual chassis). We are using a LACP bond on the 40G interfaces.

When using the 10G interfaces (xcp-ng is using those interfaces as management interfaces) for linstor traffic we run into a cap on the nic bandwith of around 4 Gbit/s (500MB/s).
When using the bonded 40G interfaces the cap is around 8 Gbit/s (1000MB/s)

Only 1 VM is installed on the pool. We are using Ubuntu 22.04 LTS with latest updates installed from ISO using the template for Ubuntu 20.04.

Here is the fio command we are using:
fio --name=a --direct=1 --bs=1M --iodepth=32 --ioengine=libaio --rw=write --filename=/tmp/test.io --size=100G

I would expect far more because we do not hit any known bottleneck of interfaces, NVMe or PCIe slot. Do I miss something? Is this expected performance? If not, any idea what the bottleneck is? Does anybody have some data we can compare with?

regards,
Swen

@ronan-a did you test some linstor vars like:
DrbdOptions/auto-diskful': Makes a resource diskful if it was continuously diskless primary for X minutes
'DrbdOptions/auto-diskful-allow-cleanup': Allows this resource to be cleaned up after toggle-disk + resync is finished

Thx for your feedback!

@olivierlambert thx for the quick reply! Does close mean days, weeks or month?

@olivierlambert can you please provide an update or better a roadmap regarding the implementation of linstor in xcp-ng? I find it hard to understand in which status this project is at the moment. As you know we are really looking forward to use it in production with our Cloudstack installation. Thx for any news.

@TheiLLeniumStudios It should be possible, but not recommended. You can end up in a split-brain-scenario.

@andersonalipio said in XOSTOR hyperconvergence preview:

Hello all,

I got it working just fine so far on all lab tests, but one thing a couldn't find in here or other post related to using a dedicated storage network other then managemente network. Can it be modified? In this lab, we have 2 hosts with 2 network cards each, one for mgmt and vm external traffic, and the second should be exclusive for storage, since is way faster.

We are using a separate network in our lab. What we do is this:

1. get the node list from the running controller via

linstor node list

2. take a look at the node interface list via

linstor node interface list <node name>

3. modify each nodes interface via

linstor node interface modify <node name> default --ip <ip>

4. check addresses via this

linstor node list

Hope that helps!

@ronan-a If you want me to test some of your fixed, please don't hesitate.

@ronan-a said in XOSTOR hyperconvergence preview:

@Swen This tool is useful to dump all key-values, there is no interpretation during dump calls: all values are a string. And metadata is a special key with a JSON object dump, the quotes are escaped by the smapi driver to store an object.

I suppose we can probably add an option to "resolve" the values with the right type like what is done in the driver itself.

It would be a great help to add an option to create some kind of json output. With this you are able to copy&paste this into a json verifier to do troubelshooting. I find it hard so read the default output at the moment when using several volumes.

@ronan-a perfect, thank you for fixing it. Is this fix already part of the code I download to install it from scratch?

@ronan-a said in XOSTOR hyperconvergence preview:

linstor-kv-tool -u xostor-2 -g xcp-sr-linstor_group_thin_device --dump-volumes -n xcp/volume
{
"7ca7b184-ec9e-40bd-addc-082483f8e420/metadata": "{"read_only": false, "snapshot_time": "", "vdi_type": "vhd", "snapshot_of": "", "name_label": "debian 11 hub disk", "name_description": "Created by XO", "type": "user", "metadata_of_pool": "", "is_a_snapshot": false}",
"7ca7b184-ec9e-40bd-addc-082483f8e420/not-exists": "0",
"7ca7b184-ec9e-40bd-addc-082483f8e420/volume-name": "xcp-volume-12571cf9-1c3b-4ee9-8f93-f4d2f7ea6bd8"
}

One more question regarding the output of this command, if you don't mind.
Can you explain why it masks all quotation marks? It looks like it is JSON, but it is not really a JSON format. Are you open for reformating the output? MY goal is to be able to perform troubleshooting easier and faster.

@ronan-a sorry, I totally missed that info.

@ronan-a said in XOSTOR hyperconvergence preview:

The main reason is that you cannot rename a LINSTOR resource once it has been created. And we need to be able to do this to implement the snapshot feature. To workaround that, a shared dictionary is used to map XAPI UUIDs to the LINSTOR resources.

It's a non-sense for the readability to use the XAPI UUIDs for the LINSTOR resources due to VDI UUID renaming when a snapshot is created.

I don't see a good reason to add the VDB UUIDs in the dictionary. You already have the VDIs, you can use xe commands to fetch the other infos.

Ok, that makes sense. But what do you mean by "You already have the VDIs"? As far as I see the only mapping from linstor-kv-tool output to the disk on xcp-ng is the name_label, is that correct?

@ronan-a sorry, was just unsure if you need more than SMlog files.

I will send you the log files via mail, because of the size.

@ronan-a said in XOSTOR hyperconvergence preview:

There is a tool installed by our RPMs to do that
For example on my host:

linstor-kv-tool -u xostor-2 -g xcp-sr-linstor_group_thin_device --dump-volumes -n xcp/volume
{
  "7ca7b184-ec9e-40bd-addc-082483f8e420/metadata": "{\"read_only\": false, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": \"\", \"name_label\": \"debian 11 hub disk\", \"name_description\": \"Created by XO\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}", 
  "7ca7b184-ec9e-40bd-addc-082483f8e420/not-exists": "0", 
  "7ca7b184-ec9e-40bd-addc-082483f8e420/volume-name": "xcp-volume-12571cf9-1c3b-4ee9-8f93-f4d2f7ea6bd8"
}

Great to know, thx for the info. Is there a reason not to use the same uuid in xcp-ng and linstor? Does it make sense to add the vdi and/or vbd uuid to the output of the command?

@ronan-a sure, which logs exactly do you need?

@ronan-a: I am playing around with xcp-ng, linstor and Cloudstack. Sometimes when I create a new VM I run into this error: The VDI is not available
CS is trying it again after this error automatically and than it works and the new VM is starting. CS is using a template which is also on the linstor SR to create new VMs.
I attached the SMlog of the host.
SMlog.txt