XOSTOR hyperconvergence preview

dumarjo

Hi,

I have been able to create a small lab with 2 xcp-ngs and would like to use the XOSTOR. First I create a first xcp-ng box (xcp-ng-01) create a new XOSTOR with all the command above. All is working. I have a new SR with 1 host only.

Now If I want to add a new host to the SR how can I do this ? I would like to simulate adding new host/disk to the SR. Is it possible ?

abufrejoval

@olivierlambert

Hi, the 8.2.1 test five image wouldn't let me install XOSTOR yet (and it wasn't recommended): so is this release variant again compatible with XOSTOR?

stormi

@abufrejoval @ronan-a has updated the test packages in the linstor test repository, so it should.

dumarjo

I tried to do the setup with 3 hosts, and followed the commands. Now I get this error when I try to create the SR.

xe sr-create type=linstor name-label=XOSTOR host-uuid=43b39fc0-002f-4347-a3e8-16e1284cfcb3 device-config:hosts=xcp-ng-01,xcp-ng-02,xcp-ng-03 device-config:group-name=linstor_group/thin_device device-config:redundancy=3 shared=true device-config:provisionning=thin

Error code: SR_BACKEND_FAILURE_5006
Error parameters: , LINSTOR SR creation error [opterr=Could not create SP `xcp-sr-linstor_group_thin_device` on node `xcp-ng-01`: (Node: 'xcp-ng-01') Expected 3 columns, but got 2],

No idea of what I need to do with this error. looks like an error when inserting into the DB.

olivierlambert

Let me ping @ronan-a

ronan-a

@dumarjo Well, check if the lvm group is activated using vgchange -a y [group].

dumarjo

@ronan-a I did this on all this server and now the SR is up.

For the sake of scalability, is it possible to add a new host or a new disk in a host easily ? Do you have a tips about it ?

Regards,

ronan-a

@dumarjo Well, for the moment we don't have a script to do that. But you can modify manually the linstor database:

You must find where is running the controller (for example with the command: linstor)
Then execute this command on the host to add:
linstor --controllers=<CONTROLLER_IP> node create --node-type combined $HOSTNAME where CONTROLLER_IP is given by the first step.
If you want to add a new disk, you must create a LVM/VG group with the name used in the other storage pools:
linstor --controllers=<CONTROLLER_IP> storage-pool list.
Than you can create the storage pool:
linstor --controllers=<CONTROLLER_IP> storage-pool create {lvm or lvmthin} $HOSTNAME <STORAGE_POOL> <POOL_NAME> (Check the two last params are the same in the DB.)
The PBDs of the current SR must be modified to use the new host, and a PBD must be created on the new host.

It's a little bit complex. So I think I will add a basic script to configure a new host or to remove an existing one. I do not recommend the usage of these commands, except in the case of tests.

dumarjo

@ronan-a said in XOSTOR hyperconvergence preview:

The PBDs of the current SR must be modified to use the new host, and a PBD must be created on the new host.

I'm relatively new with xcp-ng, and I'm a bit lost for this part. I'll give you more info on my current setup

For now I have 3 hosts that are connected and working. I can install new VMs and start them on this SR.

The new host is already added to the pool (xcp-ng-04).

It's a little bit complex. So I think I will add a basic script to configure a new host or to remove an existing one. I do not recommend the usage of these commands, except in the case of tests.

Agree. This can help a bit.

dumarjo

@ronan-a
Another situation is that I cannot use the SR on all the host that are not part of the group (linstor_group/thin_device).

for now I have my new xcp-ng-04 host joined to the pool, and I would like to connect the SR to it.

The first error I get is that the driver is not installed. So I installed all the package used on the install script (without the init disk section), and try to connect the SR to the xcp-ng-04. Now I get those error in the logfile.

[12:33 xcp-ng-01 ~]# tail -f /var/log/xensource.log | grep linstor
Mar  2 12:34:10 xcp-ng-01 xapi: [debug||2306 HTTPS 192.168.2.94->:::80|host.call_plugin R:98517b08066b|audit] Host.call_plugin host = '8ac2930f-f826-4a18-8330-06153e3e4054 (xcp-ng-02)'; plugin = 'linstor-manager'; fn = 'hasControllerRunning' args = [ 'hidden' ]
Mar  2 12:34:10 xcp-ng-01 xapi: [debug||2033 HTTPS 192.168.2.94->:::80|host.call_plugin R:e39e097fe8cb|audit] Host.call_plugin host = '81b06c7f-df55-4628-b27f-4e1e7850f900 (xcp-ng-03)'; plugin = 'linstor-manager'; fn = 'hasControllerRunning' args = [ 'hidden' ]
Mar  2 12:34:11 xcp-ng-01 xapi: [debug||2306 HTTPS 192.168.2.94->:::80|host.call_plugin R:16602b9fb9d1|audit] Host.call_plugin host = '8ac2930f-f826-4a18-8330-06153e3e4054 (xcp-ng-02)'; plugin = 'linstor-manager'; fn = 'hasControllerRunning' args = [ 'hidden' ]
Mar  2 12:34:11 xcp-ng-01 xapi: [debug||2033 HTTPS 192.168.2.94->:::80|host.call_plugin R:20c5d5ce11dc|audit] Host.call_plugin host = '81b06c7f-df55-4628-b27f-4e1e7850f900 (xcp-ng-03)'; plugin = 'linstor-manager'; fn = 'hasControllerRunning' args = [ 'hidden' ]
.....
.....
Mar  2 12:34:43 xcp-ng-01 xapi: [error||6940 ||backtrace] Async.PBD.plug R:4bbaefcdb2cb failed with exception Server_error(SR_BACKEND_FAILURE_47, [ ; The SR is not available [opterr=Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-03']];  ])
Mar  2 12:34:43 xcp-ng-01 xapi: [error||6940 ||backtrace] Raised Server_error(SR_BACKEND_FAILURE_47, [ ; The SR is not available [opterr=Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-03']];  ])

it's look like a controller need to be started on all the hosts to be able to "connect" the new host ?

Like I said, This is a new test to know if a xcp-ng that are not part of the VG can use this SR.

Hope my intention is clear...I just click on the disconnected button on UI

regards,

dumarjo

HI,

Do the vm HA is suppose to work if the VM is hosted on a SR with XOStore based ? I try to get it work, but the VM never restart if I shutdown the hosts where the VM is running.

Regards,

olivierlambert

Pinging @ronan-a

ronan-a

@dumarjo Regarding your error during the attach call, could you send me the SMlog please?

dumarjo

@ronan-a
Here some more info

Mar 10 17:05:22 xcp-ng-04 SM: [28002] lock: opening lock file /var/lock/sm/bef191f3-e976-94ec-6bb7-d87529a72dbb/sr
Mar 10 17:05:22 xcp-ng-04 SM: [28002] lock: acquired /var/lock/sm/bef191f3-e976-94ec-6bb7-d87529a72dbb/sr
Mar 10 17:05:22 xcp-ng-04 SM: [28002] sr_attach {'sr_uuid': 'bef191f3-e976-94ec-6bb7-d87529a72dbb', 'subtask_of': 'DummyRef:|79eb31a6-806c-4883-8e8d-de59cde66469|SR.at$
Mar 10 17:05:22 xcp-ng-04 SMGC: [28002] === SR bef191f3-e976-94ec-6bb7-d87529a72dbb: abort ===
Mar 10 17:05:22 xcp-ng-04 SM: [28002] lock: opening lock file /var/lock/sm/bef191f3-e976-94ec-6bb7-d87529a72dbb/running
Mar 10 17:05:22 xcp-ng-04 SM: [28002] lock: opening lock file /var/lock/sm/bef191f3-e976-94ec-6bb7-d87529a72dbb/gc_active
Mar 10 17:05:22 xcp-ng-04 SM: [28002] lock: tried lock /var/lock/sm/bef191f3-e976-94ec-6bb7-d87529a72dbb/gc_active, acquired: True (exists: True)
Mar 10 17:05:22 xcp-ng-04 SMGC: [28002] abort: releasing the process lock
Mar 10 17:05:22 xcp-ng-04 SM: [28002] lock: released /var/lock/sm/bef191f3-e976-94ec-6bb7-d87529a72dbb/gc_active
Mar 10 17:05:22 xcp-ng-04 SM: [28002] lock: acquired /var/lock/sm/bef191f3-e976-94ec-6bb7-d87529a72dbb/running
Mar 10 17:05:22 xcp-ng-04 SM: [28002] RESET for SR bef191f3-e976-94ec-6bb7-d87529a72dbb (master: False)
Mar 10 17:05:22 xcp-ng-04 SM: [28002] lock: released /var/lock/sm/bef191f3-e976-94ec-6bb7-d87529a72dbb/running
Mar 10 17:05:23 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 0
Mar 10 17:05:27 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 1
Mar 10 17:05:30 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 2
Mar 10 17:05:33 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 3
Mar 10 17:05:37 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 4
Mar 10 17:05:40 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 5
Mar 10 17:05:43 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 6
Mar 10 17:05:47 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 7
Mar 10 17:05:50 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 8
Mar 10 17:05:53 xcp-ng-04 SM: [28002] Got exception: Error: Unable to connect to any of the given controller hosts: ['linstor://xcp-ng-02']. Retry number: 9
Mar 10 17:05:54 xcp-ng-04 SM: [28002] Raising exception [47, The SR is not available [opterr=Error: Unable to connect to any of the given controller hosts: ['linstor:/$
Mar 10 17:05:54 xcp-ng-04 SM: [28002] lock: released /var/lock/sm/bef191f3-e976-94ec-6bb7-d87529a72dbb/sr
Mar 10 17:05:54 xcp-ng-04 SM: [28002] ***** generic exception: sr_attach: EXCEPTION <class 'SR.SROSError'>, The SR is not available [opterr=Error: Unable to connect to$
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     return self._run_locked(sr)
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     rv = self._run(sr, target)
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/SRCommand.py", line 352, in _run
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     return sr.attach(sr_uuid)
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/LinstorSR", line 489, in wrap
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     return load(self, *args, **kwargs)
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/LinstorSR", line 415, in load
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     raise xs_errors.XenError('SRUnavailable', opterr=str(e))
Mar 10 17:05:54 xcp-ng-04 SM: [28002]
Mar 10 17:05:54 xcp-ng-04 SM: [28002] ***** LINSTOR resources on XCP-ng: EXCEPTION <class 'SR.SROSError'>, The SR is not available [opterr=Error: Unable to connect to $
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/SRCommand.py", line 378, in run
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     ret = cmd.run(sr)
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/SRCommand.py", line 110, in run
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     return self._run_locked(sr)
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/SRCommand.py", line 159, in _run_locked
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     rv = self._run(sr, target)
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/SRCommand.py", line 352, in _run
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     return sr.attach(sr_uuid)
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/LinstorSR", line 489, in wrap
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     return load(self, *args, **kwargs)
Mar 10 17:05:54 xcp-ng-04 SM: [28002]   File "/opt/xensource/sm/LinstorSR", line 415, in load
Mar 10 17:05:54 xcp-ng-04 SM: [28002]     raise xs_errors.XenError('SRUnavailable', opterr=str(e))
Mar 10 17:05:54 xcp-ng-04 SM: [28002]

This is what I have in the log file. If you need more info, let me know.

Huxy

Hi.

I have a couple of questions I'm hoping you might be able to answer. I'm currently using Linstor with a ZFS, encrypted and compressed zpool. This solves a lot of problems for me, including high availability, data redundancy and disk wide encryption. Looking at the install script provided, this simply installs the software and sets up the LVM group.

Does this mean the sr-create command is responsible for setting up the Linstor controllers and their respective storage pools? If so, presumably at this point it only supports LVM as the storage-pool creation command will specify LVM?

If it's not possible at this time to use ZFS backing devices with the SR, is there any future plans to do so? As the Linstor controllers are responsible for managing the storage pools, I would imagine it wouldn't be too difficult to achieve. Maybe even as simple as changing the creation command to zfs i.e. linstor storage-pool create zfs node data srtank?

Cheers.

ronan-a

@dumarjo Well, the new node (xcp-ng-04) is missing in the LINSTOR database. You must add it.

ronan-a

@Huxy Yeah for the moment we support only thin and thick LVM. We also want to support ZFS for several features: ZFS snapshots and lifting the restriction of the VHD format (maximum 2TB of data). Unfortunately we don't have a planning for this now. This will probably be done in the SMapiv3, not in the V1.

Huxy

@ronan-a no worries. Thanks for the quick response. I'll consider moving to LVM or staying with Proxmox for the time being.

dumarjo

@ronan-a

Is my understanding is good ? A XCP-NG host cannot use a shared SR (based on linstor) if it's not part of the linstor nodes ?

How this new host can be part of the nodes if I don't want to add HDD/SSD to this new host ? can it be done ?

Again, I want to know the limitation before thinking using this new promising technology !

ronan-a

@dumarjo

A XCP-NG host cannot use a shared SR (based on linstor) if it's not part of the linstor nodes ?

Yeah it's required to use diskless devices.

How this new host can be part of the nodes if I don't want to add HDD/SSD to this new host ? can it be done ?

There are small changes to add to support a node without storage. But it's feasible with a modification in the driver. It should be supported because diskless volumes is a feature of DRBD to access to data using local network.