XOSTOR hyperconvergence preview

henri9813

Hello,

I got my whole xostor destroyed, i don't know how precisely.

I found some errors in sattelite

Error context:
        An error occurred while processing resource 'Node: 'host', Rsc: 'xcp-volume-e011c043-8751-45e6-be06-4ce9f8807cad''
ErrorContext:
  Details:     Command 'lvcreate --config 'devices { filter=['"'"'a|/dev/md127|'"'"','"'"'a|/dev/md126p3|'"'"','"'"'r|.*|'"'"'] }' --virtualsize 52543488k linstor_primary --thinpool thin_device --name xcp-volume-e011c043-8751-45e6-be06-4ce9f8807cad_00000' returned with exitcode 5. 

Standard out: 


Error message: 
  WARNING: Remaining free space in metadata of thin pool linstor_primary/thin_device is too low (98.06% >= 96.30%). Resize is recommended.
  Cannot create new thin volume, free space in thin pool linstor_primary/thin_device reached threshold.

of course, i checked, my SR was not full

And the controller crashed, and i couldn't make it works.

Here is the error i got

==========

Category:                           RuntimeException
Class name:                         IllegalStateException
Class canonical name:               java.lang.IllegalStateException
Generated at:                       Method 'newIllegalStateException', Source file 'DataUtils.java', Line #870

Error message:                      Reading from nio:/var/lib/linstor/linstordb.mv.db failed; file length 2293760 read length 384 at 2445540 [1.4.197/1]

So i deduce the database was fucked-up, i tried to open the file as explained in the documentation, but the linstor schema was "not found" in the file, event if using cat i see data about it.

for now, i leave xostor and i'm back to localstorage until we know what to do when this issue occured with a "solution path".

ronan-a

@henri9813 said in XOSTOR hyperconvergence preview:

of course, i checked, my SR was not full

The visual representation of used space is for informational purposes only; it's an approximation that takes into account replication, disks in use, etc. For more information: https://docs.xcp-ng.org/xostor/#how-a-linstor-sr-capacity-is-calculated

We plan to display a complete view of each physical disk space on each host someday to provide a more detailed overview. In any case, if you use "lvs"/"vgs" on each machine, you should indeed see the actual disk space used.

henri9813

Hello @ronan-a ,

but how recover from this situation ?

Thanks !

snk33

@ronan-a said in XOSTOR hyperconvergence preview:

@peter_webbird We've already had feedback on CBT and LINSTOR/DRBD, we don't necessarily recommend enabling it. We have a blocking dev card regarding a bug with LVM lvchange command that may fail on CBT volumes used by a XOSTOR SR. We also have other issues related to migration with CBT.

Is the problem still occurring on latest XCP-ng / XOSTOR version ? Not being able to use CBT on XOSTOR is a big issue for backup/replication.

snk33

Any update from someone ? Not having the ability to enable CBT on XOSTOR is really complicated to make replication / DR scenarios.

At least to know if this is work in progress or not. If it's not we'll need to pause our PoC and post-pone production schedule waiting for significant update.

ronan-a

@snk33 In fact we have limitations with CBT and XOSTOR. And this won't be resolved until we get back into SMAPIv3.

Not being able to use CBT on XOSTOR is a big issue for backup/replication.

Using it is more likely to cause problems than improve your situation. What bothers you about not using CBT here?

snk33

@ronan-a we need to have CBT to use 3rd party backup & replication solution such as Veeam.

On the other hand, XOSTOR with VHD disk might be ok using Xen Orchestra backup & replication features but it needs more testing on our end to make sure it works good enough.
One big question is about the snapshot chain. Coming from VMware world where keeping snapshots may impact I/O performances, we're not very comfortable with it.

Also, we need to find solution for low-RPO replication. On our VMware infrastructure we use Zerto DR for this but XOA being more like Veeam using snapshots, low-RPO will be tough to achieve (unless making snapshots doesn't affect performances at all ?).

Maelstrom96

Is there a chance something is broken with XOSTOR when adding a new node? I've added one and tried to ran the addHost command, but whatever I do, the SM logs are filled with

Nov  3 18:45:01 van3-pprod-xen05 SM: [7577][MainThread] Got exception: Unable to find controller uri.... Retry number: 8
Nov  3 18:45:05 van3-pprod-xen05 SM: [4767][MainThread] Got exception: Unable to find controller uri.... Retry number: 16
Nov  3 18:45:23 van3-pprod-xen05 SM: [5313][MainThread] Got exception: Unable to find controller uri.... Retry number: 15
Nov  3 18:45:23 van3-pprod-xen05 SM: [9900][MainThread] Got exception: Unable to find controller uri.... Retry number: 3

Networking seems all good, but the node is not able to be enabled because of that.

Danp

@Maelstrom96 Can you provide more context?

XOA or XO from sources? Commit / version?
Are the XCP-ng hosts fully patched?
Did you follow this procedure when adding the new host?
etc

dthenot

@Maelstrom96 Hello,

There is indeed an identified bug with addHost that we are investigating at the moment.