XOSTOR hyperconvergence preview

Swen

@ronan-a: I am playing around with xcp-ng, linstor and Cloudstack. Sometimes when I create a new VM I run into this error: The VDI is not available
CS is trying it again after this error automatically and than it works and the new VM is starting. CS is using a template which is also on the linstor SR to create new VMs.
I attached the SMlog of the host.
SMlog.txt

ronan-a

@Swen After doing the installation from scratch with new installed xcp-ng hosts, all uptodate, I need to do a repair (via xcp-ng center) of the SR after doing the xe sr-create, because the SR is in state: Broken and the pool-master is in state: Unplugged.

I am not really sure waht xcp-ng center is doing when I click repair, but it works.

It's just a PBD plug call I suppose. Can you share your logs please?

ronan-a

@Swen said in XOSTOR hyperconvergence preview:

Is there an easy way to map the linstor resource volume to the virtual disk on xcp-ng? When doing linstor volume list I get a Resource name back from linstor like this:
xcp-volume-23d07d99-9990-4046-8e7d-020bd61c1883

The last part looks like an uuid to me, but I am unable to find this uuid when using some xe commands.

There is a tool installed by our RPMs to do that
For example on my host:

linstor-kv-tool -u xostor-2 -g xcp-sr-linstor_group_thin_device --dump-volumes -n xcp/volume
{
  "7ca7b184-ec9e-40bd-addc-082483f8e420/metadata": "{\"read_only\": false, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": \"\", \"name_label\": \"debian 11 hub disk\", \"name_description\": \"Created by XO\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}", 
  "7ca7b184-ec9e-40bd-addc-082483f8e420/not-exists": "0", 
  "7ca7b184-ec9e-40bd-addc-082483f8e420/volume-name": "xcp-volume-12571cf9-1c3b-4ee9-8f93-f4d2f7ea6bd8"
}

ronan-a

@Swen said in XOSTOR hyperconvergence preview:

@ronan-a: I am playing around with xcp-ng, linstor and Cloudstack. Sometimes when I create a new VM I run into this error: The VDI is not available
CS is trying it again after this error automatically and than it works and the new VM is starting. CS is using a template which is also on the linstor SR to create new VMs.
I attached the SMlog of the host.
SMlog.txt

Can you share the log files of the other hosts please?

Swen

@ronan-a sure, which logs exactly do you need?

Swen

@ronan-a said in XOSTOR hyperconvergence preview:

There is a tool installed by our RPMs to do that
For example on my host:

linstor-kv-tool -u xostor-2 -g xcp-sr-linstor_group_thin_device --dump-volumes -n xcp/volume
{
  "7ca7b184-ec9e-40bd-addc-082483f8e420/metadata": "{\"read_only\": false, \"snapshot_time\": \"\", \"vdi_type\": \"vhd\", \"snapshot_of\": \"\", \"name_label\": \"debian 11 hub disk\", \"name_description\": \"Created by XO\", \"type\": \"user\", \"metadata_of_pool\": \"\", \"is_a_snapshot\": false}", 
  "7ca7b184-ec9e-40bd-addc-082483f8e420/not-exists": "0", 
  "7ca7b184-ec9e-40bd-addc-082483f8e420/volume-name": "xcp-volume-12571cf9-1c3b-4ee9-8f93-f4d2f7ea6bd8"
}

Great to know, thx for the info. Is there a reason not to use the same uuid in xcp-ng and linstor? Does it make sense to add the vdi and/or vbd uuid to the output of the command?

ronan-a

@Swen said in XOSTOR hyperconvergence preview:

@ronan-a sure, which logs exactly do you need?

SMlog files of each host

ronan-a

@Swen said in XOSTOR hyperconvergence preview:

Great to know, thx for the info. Is there a reason not to use the same uuid in xcp-ng and linstor? Does it make sense to add the vdi and/or vbd uuid to the output of the command?

The main reason is that you cannot rename a LINSTOR resource once it has been created. And we need to be able to do this to implement the snapshot feature. To workaround that, a shared dictionary is used to map XAPI UUIDs to the LINSTOR resources.

It's a non-sense for the readability to use the XAPI UUIDs for the LINSTOR resources due to VDI UUID renaming when a snapshot is created.

I don't see a good reason to add the VDB UUIDs in the dictionary. You already have the VDIs, you can use xe commands to fetch the other infos.

Swen

@ronan-a sorry, was just unsure if you need more than SMlog files.

I will send you the log files via mail, because of the size.

Swen

@ronan-a said in XOSTOR hyperconvergence preview:

The main reason is that you cannot rename a LINSTOR resource once it has been created. And we need to be able to do this to implement the snapshot feature. To workaround that, a shared dictionary is used to map XAPI UUIDs to the LINSTOR resources.

It's a non-sense for the readability to use the XAPI UUIDs for the LINSTOR resources due to VDI UUID renaming when a snapshot is created.

I don't see a good reason to add the VDB UUIDs in the dictionary. You already have the VDIs, you can use xe commands to fetch the other infos.

Ok, that makes sense. But what do you mean by "You already have the VDIs"? As far as I see the only mapping from linstor-kv-tool output to the disk on xcp-ng is the name_label, is that correct?

ronan-a

@Swen said in XOSTOR hyperconvergence preview:

Ok, that makes sense. But what do you mean by "You already have the VDIs"? As far as I see the only mapping from linstor-kv-tool output to the disk on xcp-ng is the name_label, is that correct?

No you have the VDI UUIDs:

"7ca7b184-ec9e-40bd-addc-082483f8e420/volume-name": "xcp-volume-12571cf9-1c3b-4ee9-8f93-f4d2f7ea6bd8"

The first UUID here is the VDI.

Swen

@ronan-a sorry, I totally missed that info.

Swen

@ronan-a said in XOSTOR hyperconvergence preview:

linstor-kv-tool -u xostor-2 -g xcp-sr-linstor_group_thin_device --dump-volumes -n xcp/volume
{
"7ca7b184-ec9e-40bd-addc-082483f8e420/metadata": "{"read_only": false, "snapshot_time": "", "vdi_type": "vhd", "snapshot_of": "", "name_label": "debian 11 hub disk", "name_description": "Created by XO", "type": "user", "metadata_of_pool": "", "is_a_snapshot": false}",
"7ca7b184-ec9e-40bd-addc-082483f8e420/not-exists": "0",
"7ca7b184-ec9e-40bd-addc-082483f8e420/volume-name": "xcp-volume-12571cf9-1c3b-4ee9-8f93-f4d2f7ea6bd8"
}

One more question regarding the output of this command, if you don't mind.
Can you explain why it masks all quotation marks? It looks like it is JSON, but it is not really a JSON format. Are you open for reformating the output? MY goal is to be able to perform troubleshooting easier and faster.

ronan-a

@Swen This tool is useful to dump all key-values, there is no interpretation during dump calls: all values are a string. And metadata is a special key with a JSON object dump, the quotes are escaped by the smapi driver to store an object.

I suppose we can probably add an option to "resolve" the values with the right type like what is done in the driver itself.

ronan-a

@Swen said in XOSTOR hyperconvergence preview:

@olivierlambert I need to me more clear about this: When doing the sr-create for the linstor storage no error is shown, but the pbd will not be plugged at the pool-master. On every other host in the cluster it works automatically. After doing a pdb-plug for the pool-master the SR will be plugged. No error is shown at all.

Ok so I confirm that it is a problem of timing when retrieving the list of volumes (more precisely their size) using the LINSTOR API. I modified the driver to retry in case of failure. So not a big issue.

Swen

@ronan-a perfect, thank you for fixing it. Is this fix already part of the code I download to install it from scratch?

Swen

@ronan-a said in XOSTOR hyperconvergence preview:

@Swen This tool is useful to dump all key-values, there is no interpretation during dump calls: all values are a string. And metadata is a special key with a JSON object dump, the quotes are escaped by the smapi driver to store an object.

I suppose we can probably add an option to "resolve" the values with the right type like what is done in the driver itself.

It would be a great help to add an option to create some kind of json output. With this you are able to copy&paste this into a json verifier to do troubelshooting. I find it hard so read the default output at the moment when using several volumes.

ronan-a

@Swen said in XOSTOR hyperconvergence preview:

@ronan-a: I am playing around with xcp-ng, linstor and Cloudstack. Sometimes when I create a new VM I run into this error: The VDI is not available
CS is trying it again after this error automatically and than it works and the new VM is starting. CS is using a template which is also on the linstor SR to create new VMs.
I attached the SMlog of the host.
SMlog.txt

Ok I got it:

Mar 29 14:46:52 pc-xcp21 SM: [8299] ['/bin/dd', 'if=/dev/zero', 'of=/dev/drbd/by-res/xcp-volume-a44a5d25-24a8-4f83-8b74-63fe36d9ec44/0', 'bs=1', 'seek=5268045312', 'count=512']
Mar 29 14:46:52 pc-xcp21 SM: [8299] FAILED in util.pread: (rc 1) stdout: '', stderr: '/bin/dd: '/dev/drbd/by-res/xcp-volume-a44a5d25-24a8-4f83-8b74-63fe36d9ec44/0': cannot seek: Invalid argument
Mar 29 14:46:52 pc-xcp21 SM: [8299] 0+0 records in
Mar 29 14:46:52 pc-xcp21 SM: [8299] 0+0 records out
Mar 29 14:46:52 pc-xcp21 SM: [8299] 0 bytes (0 B) copied, 0.0013104 s, 0.0 kB/s
Mar 29 14:46:52 pc-xcp21 SM: [8299] '

It's related to this trace, the problem is fixed in the latest linbit packages, I haven't synced them to our own repository yet.

ronan-a

@Swen said in XOSTOR hyperconvergence preview:

perfect, thank you for fixing it. Is this fix already part of the code I download to install it from scratch?

Not yet, I will probably add other fixes before.

Swen

@ronan-a If you want me to test some of your fixed, please don't hesitate.