<![CDATA[XOSTOR hyperconvergence preview]]>XOSTOR - Tech preview

⚠ Installation script is compatible with XCP-ng 8.2 and 8.3 ⚠

⚠ UPDATE to sm-2.30.7-1.3.0.linstor.7 ⚠

Please read this: https://xcp-ng.org/forum/topic/5361/xostor-hyperconvergence-preview/224?_=1679390249707

⚠ UPDATE from an older version (before sm-2.30.7-1.3.0.linstor.3) ⚠

Please read this: https://xcp-ng.org/forum/topic/5361/xostor-hyperconvergence-preview/177?_=1667938000897


XOSTOR is a "disaggregated hyperconvergence storage solution". In plain English: you can assemble local storage of multiple hosts into one "fake" shared storage.

The key to get fast hyperconvergence is to try a different approach. We used GlusterFS for XOSAN, and it wasn't really fast for small random blocks (due to the nature of the global filesystem). But in XOSTOR, there's a catch: unlike traditional hyperconvergence, it won't create a global clustered and shared filesystem. This time, when you'll create a VM disk, it will create a "resource", that will be replicated "n" times on multiple hosts (eg twice or 3 times).

So in the end, the number of resources depends on the VM disk numbers (and snapshots).

The technology we use is not invented from scratch, we are using LINSTOR from LINBIT, based itself on DRBD. See https://linbit.com/linstor/

For you, it will be (ideally) transparent.

Ultimate goals

Our first goal here is to validate the technology at scale. If it works as we expect, then we'll add a complete automated solution and UI on top of it, and sell pro support for people who want to get a "turnkey" supported solution (a la XOA).

The manual/shell script installation as described here is meant to stay fully open/accessible with community support 🙂

Now I'm letting @ronan-a writing the rest of this message 🙂 Thanks a lot for your hard work 😉

How-it-works-Isometric-Deck-5Integrations-VersionV2-1024x554.png

⚠ Important ⚠

Despite we are doing intensive testing with this technology in the last 2 years (!), it was really HARD to integrate it easily into SMAPIv1 (legacy storage stack of XCP-ng). Especially when you have to test all potential cases.

The goal of this tech preview is to scale our testing to a LOT of users.

Right now, this version should be installed on pools with 3 or 4 hosts. We plan to release another test release in one month to remove this limitation. Also, in order to ensure data integrity, it is more than recommended to use at least 3 hosts.

How to install XOSTOR on your pool?

1. Download installation script

First, you must ensure you have at least one free disk or more on each host of your pool.
Then you can download the installation script using this command:

wget https://gist.githubusercontent.com/Wescoeur/7bb568c0e09e796710b0ea966882fcac/raw/052b3dfff9c06b1765e51d8de72c90f2f90f475b/gistfile1.txt -O install && chmod +x install

2. Install

Then, on each host you must execute the script with the disks to use, for example with one partition:

./install --disks /dev/sdb

If you have many disks you can use them, BUT for optimal use, the sum of all disks should be the same on each host:

./install --disks /dev/nvme0n1 /dev/nvme0n2 /dev/nvme0n3

By default, thick provisioning is used, you can use thin instead:

./install --disks /dev/sdb --thin

Note: You can use the --force flag if you already have a VG group or PV on your hosts to override:

./install --disks /dev/sdb --thin --force

3. Verify config

With thin option

lsblk must return on each host an output similar to:

> lsblk
NAME                                                                              MAJ:MIN  RM   SIZE RO TYPE  MOUNTPOINT
...
sdb                                                                                 8:16    0   1.8T  0 disk
└─36848f690df82210028c2364008358dd7                                               253:0     0   1.8T  0 mpath
  ├─linstor_group-thin_device_tmeta                                               253:1     0   120M  0 lvm
  │ └─linstor_group-thin_device-tpool                                             253:3     0   1.8T  0 lvm
  └─linstor_group-thin_device_tdata                                               253:2     0   1.8T  0 lvm
    └─linstor_group-thin_device-tpool                                             253:3     0   1.8T  0 lvm
...

With thick option

No LVM volume is created, only a new group must be present now using vgs command.

> vgs
  VG                                                 #PV #LV #SN Attr   VSize   VFree  
  ...
  linstor_group                                        1   0   0 wz--n- 931.51g 931.51g

And you must have linstor versions of sm and xha:

> rpm -qa | grep -E "^(sm|xha)-.*linstor.*"
sm-2.30.4-1.1.0.linstor.8.xcpng8.2.x86_64
xha-10.1.0-2.2.0.linstor.1.xcpng8.2.x86_64

4. Finally you can create the SR:

If you use thick provisioning:

xe sr-create type=linstor name-label=<SR_NAME> host-uuid=<MASTER_UUID> device-config:group-name=linstor_group device-config:redundancy=<REDUNDANCY> shared=true device-config:provisioning=thick

Otherwise with thin provisioning:

xe sr-create type=linstor name-label=<SR_NAME> host-uuid=<MASTER_UUID> device-config:group-name=linstor_group/thin_device device-config:redundancy=<REDUNDANCY> shared=true device-config:provisioning=thin

So for example if you have 4 hosts, a thin config and you want a replication of 3 for each disk:

xe sr-create type=linstor name-label=XOSTOR host-uuid=bc3cd3af-3f09-48cf-ae55-515ba21930f5 device-config:group-name=linstor_group/thin_device device-config:redundancy=3 shared=true device-config:provisioning=thin

5. Verification

After that you must have a XOSTOR SR visible in XOA with all PBDs attached.

6. Update

If you want to update your LINSTOR and other packages, you can execute on each host the install script like this:

./install --update-only

F.A.Q.

How the SR capacity is calculated? 🤔

If you can't create a VDI greater than the displayed size in the XO SR view, don't worry:

  • There are two important things to remember: the maximum size of a VDI that can be created is not necessarily equal to the capacity of the SR. The SR capacity in the XOSTOR context is the maximum size that can be used to store all VDI data.
  • Exception: if the replication count is equal to the number of hosts, the SR capacity is equal to the max VDI size, i.e. the capacity of the smallest disk in the pool.

We use this formula to compute the SR capacity:

sr_capacity = smallest_host_disk_capacity * host_count / replication_count

For example if you have a pool of 3 hosts with a replication count of 2 and a disk of 200 GiB on each host, the capacity of the SR is equal to 300 GiB using the formula. Notes:

  • You can't create a VDI greater than 200 GiB because the replication is not block based but volume based.
  • If you create a volume of 200 GiB (400 of the 600 GiB are physically used) and the remaining disk can't be used because it becomes impossible to replicate on two different disks.
  • If you create 3 volumes of 100 GiB: the SR becomes fully filled. In this case you have 300 GiB of unique data and a replication of 300 GiB.

How to destroy properly the SR after a SR.forget call?

If you used a command like SR.forget, the SR is not actually removed properly. To do that you can execute these commands:

# Create new UUID for the SR to reintroduce it.
uuidgen

# Reintroduce the SR.
xe sr-introduce uuid=<UUID_of_uuidgen> type=linstor shared=true name-label="XOSTOR" content-type=user

# Get host list to recreate PBD
xe host-list
...

# For each host, you must execute a `xe pbd-create` call.
# Don't forget to use the correct SR/host UUIDs, and device-config parameters.
xe pbd-create host-uuid=<host_uuid>  sr-uuid=uuid_of_uuidgen> device-config:provisioning=thick device-config:redundancy=<redundancy> device-config:group-name=<group_name>

# After this point you can now destroy the SR properly using xe or XOA.

Node: auto-eviction and how to restore?

If a node is no longer active for 60 minutes by default, it's automatically evicted. This behavior can be changed.
There is an advantage using auto evict, if there are enough nodes in your cluster, LINSTOR will create new replicas of your disks.

See: https://linbit.com/blog/linstors-auto-evict/

Now if you want to re-add your node, it's not automatic. You can used a linstor command to remove it: linstor node lost. Then you can recreate it. Also if there is no disk issue, and it was a network problem, whatever, just run one command linstor node restore.

How to use a specific network storage?

You can run few specific LINSTOR commands to configure new NICs to use. By default the XAPI management interface is used.

For more info: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#s-managing_network_interface_cards

In case of failure with the preferred NIC, the default interface is used.

How to replace drives?

Take a look at the official documentation: https://kb.linbit.com/how-do-i-replace-a-failed-d

XAPI plugin: linstor-manager

It's possible to perform low-level tasks using the linstor-manager plugin.

It can be executed using the following command:

xe host-call-plugin host-uuid=<HOST_UUID> plugin=linstor-manager fn=<FUNCTION> args:<ARG_NAME_1>=<VALUE_1> args:<ARG_NAME_2>=<VALUE_2> ...

Many functions are not documented here and are reserved for internal use by the smapi driver (LinstorSR).

For each command, HOST_UUID is a host of your pool, master or not.

Add a new host to an existing LINSTOR SR

xe host-call-plugin host-uuid=<HOST_UUID> plugin=linstor-manager fn=addHost args:groupName=<THIN_OR_THICK_POOL_NAME>

This command creates a new PBD on the SR and new node in the LINSTOR database. Also it starts what's is necessary for the driver.
After running this command, it's up to you to set up a new storage pool in the LINSTOR database with the same name used by the other nodes.
So again use pvcreate/vgcreate and then a basic "linstor storage-pool create"

Remove a new host from an existing LINSTOR SR

xe host-call-plugin host-uuid=<HOST_UUID> plugin=linstor-manager fn=removeHost args:groupName=<THIN_OR_THICK_POOL_NAME>

Check if the linstor controller is currently running on a specific host

xe host-call-plugin host-uuid=<HOST_UUID> plugin=linstor-manager fn=hasControllerRunning

Example:

xe host-call-plugin host-uuid=ddcd3461-7052-4f5e-932c-e1ed75c192d6 plugin=linstor-manager fn=hasControllerRunning
False

Check if a DRBD volume is currently used by a process on a specific host

xe host-call-plugin host-uuid=<HOST_UUID> plugin=linstor-manager fn=getDrbdOpeners args:resourceName=<RES_NAME> args:volume=0

Example:

xe host-call-plugin host-uuid=ddcd3461-7052-4f5e-932c-e1ed75c192d6 plugin=linstor-manager fn=getDrbdOpeners args:resourceName=xcp-volume-a10809db-bb40-43bd-9dee-22d70d781c45 args:volume=0
{}

List DRBD volumes

xe host-call-plugin host-uuid=<HOST_UUID> plugin=linstor-manager fn=listDrbdVolumes args:groupName=<THIN_OR_THICK_POOL_NAME>

Example:

xe host-call-plugin host-uuid=ddcd3461-7052-4f5e-932c-e1ed75c192d6 plugin=linstor-manager fn=listDrbdVolumes args:groupName=linstor_group/thin_device
{"linstor_group": [1000, 1005, 1001, 1007, 1006]}

Force destruction of DRBD volumes

Warning: In principle, the volumes created by the smapi driver (LinstorSR) must be destroyed using the XAPI or XOA. Only use these functions if you know what you are doing. Otherwise, forget them.

# To destroy one volume:
xe host-call-plugin host-uuid=<HOST_UUID> plugin=linstor-manager fn=destroyDrbdVolume args:minor=<MINOR>

# To destroy all volumes:
xe host-call-plugin host-uuid=<HOST_UUID> plugin=linstor-manager fn=destroyDrbdVolumes args:groupName=<THIN_OR_THICK_POOL_NAME>
]]>https://xcp-ng.org/forum/topic/5361/xostor-hyperconvergence-previewRSS for NodeFri, 15 May 2026 07:48:03 GMTMon, 20 Dec 2021 09:39:54 GMT60<![CDATA[Reply to XOSTOR hyperconvergence preview on Wed, 14 Jan 2026 10:31:45 GMT]]>@ronan-a we need to have CBT to use 3rd party backup & replication solution such as Veeam.

On the other hand, XOSTOR with VHD disk might be ok using Xen Orchestra backup & replication features but it needs more testing on our end to make sure it works good enough.
One big question is about the snapshot chain. Coming from VMware world where keeping snapshots may impact I/O performances, we're not very comfortable with it.

Also, we need to find solution for low-RPO replication. On our VMware infrastructure we use Zerto DR for this but XOA being more like Veeam using snapshots, low-RPO will be tough to achieve (unless making snapshots doesn't affect performances at all ?).

]]>https://xcp-ng.org/forum/post/101538https://xcp-ng.org/forum/post/101538Wed, 14 Jan 2026 10:31:45 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Tue, 13 Jan 2026 23:24:02 GMT]]>@snk33 In fact we have limitations with CBT and XOSTOR. And this won't be resolved until we get back into SMAPIv3.

Not being able to use CBT on XOSTOR is a big issue for backup/replication.

Using it is more likely to cause problems than improve your situation. What bothers you about not using CBT here?

]]>https://xcp-ng.org/forum/post/101532https://xcp-ng.org/forum/post/101532Tue, 13 Jan 2026 23:24:02 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Tue, 06 Jan 2026 14:57:37 GMT]]>Any update from someone ? Not having the ability to enable CBT on XOSTOR is really complicated to make replication / DR scenarios.

At least to know if this is work in progress or not. If it's not we'll need to pause our PoC and post-pone production schedule waiting for significant update.

]]>https://xcp-ng.org/forum/post/101300https://xcp-ng.org/forum/post/101300Tue, 06 Jan 2026 14:57:37 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Sat, 27 Dec 2025 20:30:23 GMT]]>@ronan-a said in XOSTOR hyperconvergence preview:

@peter_webbird We've already had feedback on CBT and LINSTOR/DRBD, we don't necessarily recommend enabling it. We have a blocking dev card regarding a bug with LVM lvchange command that may fail on CBT volumes used by a XOSTOR SR. We also have other issues related to migration with CBT.

Is the problem still occurring on latest XCP-ng / XOSTOR version ? Not being able to use CBT on XOSTOR is a big issue for backup/replication.

]]>https://xcp-ng.org/forum/post/101080https://xcp-ng.org/forum/post/101080Sat, 27 Dec 2025 20:30:23 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Tue, 18 Nov 2025 19:56:41 GMT]]>Hello @ronan-a ,

but how recover from this situation ?

Thanks !

]]>https://xcp-ng.org/forum/post/99756https://xcp-ng.org/forum/post/99756Tue, 18 Nov 2025 19:56:41 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Mon, 10 Nov 2025 09:55:48 GMT]]>@henri9813 said in XOSTOR hyperconvergence preview:

of course, i checked, my SR was not full

The visual representation of used space is for informational purposes only; it's an approximation that takes into account replication, disks in use, etc. For more information: https://docs.xcp-ng.org/xostor/#how-a-linstor-sr-capacity-is-calculated

We plan to display a complete view of each physical disk space on each host someday to provide a more detailed overview. In any case, if you use "lvs"/"vgs" on each machine, you should indeed see the actual disk space used.

]]>https://xcp-ng.org/forum/post/99428https://xcp-ng.org/forum/post/99428Mon, 10 Nov 2025 09:55:48 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Sun, 09 Nov 2025 14:02:26 GMT]]>Hello,

I got my whole xostor destroyed, i don't know how precisely.

I found some errors in sattelite

Error context:
        An error occurred while processing resource 'Node: 'host', Rsc: 'xcp-volume-e011c043-8751-45e6-be06-4ce9f8807cad''
ErrorContext:
  Details:     Command 'lvcreate --config 'devices { filter=['"'"'a|/dev/md127|'"'"','"'"'a|/dev/md126p3|'"'"','"'"'r|.*|'"'"'] }' --virtualsize 52543488k linstor_primary --thinpool thin_device --name xcp-volume-e011c043-8751-45e6-be06-4ce9f8807cad_00000' returned with exitcode 5. 

Standard out: 


Error message: 
  WARNING: Remaining free space in metadata of thin pool linstor_primary/thin_device is too low (98.06% >= 96.30%). Resize is recommended.
  Cannot create new thin volume, free space in thin pool linstor_primary/thin_device reached threshold.

of course, i checked, my SR was not full
aa2774a4-c2d4-4dd1-be52-3c6e418c9083-image.png

And the controller crashed, and i couldn't make it works.

Here is the error i got

==========

Category:                           RuntimeException
Class name:                         IllegalStateException
Class canonical name:               java.lang.IllegalStateException
Generated at:                       Method 'newIllegalStateException', Source file 'DataUtils.java', Line #870

Error message:                      Reading from nio:/var/lib/linstor/linstordb.mv.db failed; file length 2293760 read length 384 at 2445540 [1.4.197/1]

So i deduce the database was fucked-up, i tried to open the file as explained in the documentation, but the linstor schema was "not found" in the file, event if using cat i see data about it.

for now, i leave xostor and i'm back to localstorage until we know what to do when this issue occured with a "solution path".

]]>https://xcp-ng.org/forum/post/99398https://xcp-ng.org/forum/post/99398Sun, 09 Nov 2025 14:02:26 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Wed, 15 Oct 2025 22:54:17 GMT]]>I have amazing news!

After the upgrade to xcp-ng 8.3, I retested velero backup, and it all just works 😁

Completed Backup

jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml backup describe grafana-test
Name:         grafana-test
Namespace:    velero
Labels:       objectset.rio.cattle.io/hash=c2b5f500ab5d9b8ffe14f2c70bf3742291df565c
              velero.io/storage-location=default
Annotations:  objectset.rio.cattle.io/applied=H4sIAAAAAAAA/4SSQW/bPgzFvwvPtv9OajeJj/8N22HdBqxFL0MPlEQlWmTRkOhgQ5HvPsixE2yH7iji8ffIJ74CDu6ZYnIcoIMTeYpcOf7vtIICji4Y6OB/1MdxgAJ6EjQoCN0rYAgsKI5Dyk9WP0hLIqmi40qjiKfMcRlAq7pBY+py26qmbEi15a5p78vtaqe0oqbVVsO5AI+K/Ju4A6YDdKDXqrVtXaNqzU5traVVY9d6Uyt7t2nW693K2Pa+naABe4IO9hEtBiyFksClmgbUdN06a9NAOtvr5B4DDunA8uR64lGgg7u6rxMUYMji6OWZ/dhTeuIPaQ6os+gTFUA/tR8NmXd+TELxUfNA5hslHqOmBN13OF16ZwvNQShIqpZClYQj7qk6blPlGF5uzC/L3P+kvok7MB9z0OcCXPiLPLHmuLLWCfVfB4rTZ9/iaA5zHovNZz7R++k6JI50q89BXcuXYR5YT0DolkChABEPHWzW9cK+rPQx8jgsH/KQj+QT/frzXCdduc/Ca9u1Y7aaFvMu5Ang5Xz+HQAA//8X7Fu+/QIAAA
              objectset.rio.cattle.io/id=e104add0-85b4-4eb5-9456-819bcbe45cfc
              velero.io/resource-timeout=10m0s
              velero.io/source-cluster-k8s-gitversion=v1.33.4+rke2r1
              velero.io/source-cluster-k8s-major-version=1
              velero.io/source-cluster-k8s-minor-version=33

Phase:  Completed


Namespaces:
  Included:  grafana
  Excluded:  <none>

Resources:
  Included cluster-scoped:    <none>
  Excluded cluster-scoped:    volumesnapshotcontents.snapshot.storage.k8s.io
  Included namespace-scoped:  *
  Excluded namespace-scoped:  volumesnapshots.snapshot.storage.k8s.io

Label selector:  <none>

Or label selector:  <none>

Storage Location:  default

Velero-Native Snapshot PVs:  true
Snapshot Move Data:          true
Data Mover:                  velero

TTL:  720h0m0s

CSISnapshotTimeout:    30m0s
ItemOperationTimeout:  4h0m0s

Hooks:  <none>

Backup Format Version:  1.1.0

Started:    2025-10-15 15:29:52 -0700 PDT
Completed:  2025-10-15 15:31:25 -0700 PDT

Expiration:  2025-11-14 14:29:52 -0800 PST

Total items to be backed up:  35
Items backed up:              35

Backup Item Operations:  1 of 1 completed successfully, 0 failed (specify --details for more information)
Backup Volumes:
  Velero-Native Snapshots: <none included>

  CSI Snapshots:
    grafana/central-grafana:
      Data Movement: included, specify --details for more information

  Pod Volume Backups: <none included>

HooksAttempted:  0
HooksFailed:     0

Completed Restore

jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml restore describe restore-grafana-test --details
Name:         restore-grafana-test
Namespace:    velero
Labels:       objectset.rio.cattle.io/hash=252addb3ed156c52d9fa9b8c045b47a55d66c0af
Annotations:  objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW7zIBBA7zJrO5/j35gzfE2rtsomymIM45jGBgTjbKLcvaKJm6qL7kDwnt7ABdDpHfmgrQEBZxrJ25W2/85rSOCkjQIBrxTYeoIEJmJUyAjiAmiMZWRtTYhb232Q5EC88tquJDKPFEU6GlpUG5UVZdpUdZ6WZZ+niOtNWtR1SypvqC8buCYwYkfjn7oBwwAC8ipHpbqC1LqqZZWrtse228isrLqywapSdS0z7KPU4EQgwN+mSI8eezSYMgWG22lwKOl7/MgERzJmdChPs9veDL9IGfSbQRcGy+96IjszCCiyCRLQRo6zIrVd5AHEfuHhkIBmmp4d+a/3e9Dl8LPoCZ3T5hg7FvQRcR8nxt6XL7sAgv1MCZztOE+01P23cvmnPYzaxNtwuF4/AwAA//8k6OwC/QEAAA
              objectset.rio.cattle.io/id=9ad8d034-7562-44f2-aa18-3669ed27ef47

Phase:                       Completed
Total items to be restored:  33
Items restored:              33

Started:    2025-10-15 15:35:26 -0700 PDT
Completed:  2025-10-15 15:36:34 -0700 PDT

Warnings:
  Velero:     <none>
  Cluster:    <none>
  Namespaces:
    grafana-restore:  could not restore, ConfigMap:elasticsearch-es-transport-ca-internal already exists. Warning: the in-cluster version is different than the backed-up version
                      could not restore, ConfigMap:kube-root-ca.crt already exists. Warning: the in-cluster version is different than the backed-up version

Backup:  grafana-test

Namespaces:
  Included:  grafana
  Excluded:  <none>

Resources:
  Included:        *
  Excluded:        nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
  Cluster-scoped:  auto

Namespace mappings:  grafana=grafana-restore

Label selector:  <none>

Or label selector:  <none>

Restore PVs:  true

CSI Snapshot Restores:
  grafana-restore/central-grafana:
    Data Movement:
      Operation ID: dd-ffa56e1c-9fd0-44b4-a8bb-8163f40a49e9.330b82fc-ca6a-423217ee5
      Data Mover: velero
      Uploader Type: kopia

Existing Resource Policy:   <none>
ItemOperationTimeout:       4h0m0s

Preserve Service NodePorts:  auto

Restore Item Operations:
  Operation for persistentvolumeclaims grafana-restore/central-grafana:
    Restore Item Action Plugin:  velero.io/csi-pvc-restorer
    Operation ID:                dd-ffa56e1c-9fd0-44b4-a8bb-8163f40a49e9.330b82fc-ca6a-423217ee5
    Phase:                       Completed
    Progress:                    856284762 of 856284762 complete (Bytes)
    Progress description:        Completed
    Created:                     2025-10-15 15:35:28 -0700 PDT
    Started:                     2025-10-15 15:36:06 -0700 PDT
    Updated:                     2025-10-15 15:36:26 -0700 PDT

HooksAttempted:   0
HooksFailed:      0

Resource List:
  apps/v1/Deployment:
    - grafana-restore/central-grafana(created)
    - grafana-restore/grafana-debug(created)
  apps/v1/ReplicaSet:
    - grafana-restore/central-grafana-5448b9f65(created)
    - grafana-restore/central-grafana-56887c6cb6(created)
    - grafana-restore/central-grafana-56ddd4f497(created)
    - grafana-restore/central-grafana-5f4757844b(created)
    - grafana-restore/central-grafana-5f69f86c85(created)
    - grafana-restore/central-grafana-64545dcdc(created)
    - grafana-restore/central-grafana-69c66c54d9(created)
    - grafana-restore/central-grafana-6c8d6f65b8(created)
    - grafana-restore/central-grafana-7b479f79ff(created)
    - grafana-restore/central-grafana-bc7d96cdd(created)
    - grafana-restore/central-grafana-cb88bd49c(created)
    - grafana-restore/grafana-debug-556845ff7b(created)
    - grafana-restore/grafana-debug-6fb594cb5f(created)
    - grafana-restore/grafana-debug-8f66bfbf6(created)
  discovery.k8s.io/v1/EndpointSlice:
    - grafana-restore/central-grafana-hkgd5(created)
  networking.k8s.io/v1/Ingress:
    - grafana-restore/central-grafana(created)
  rbac.authorization.k8s.io/v1/Role:
    - grafana-restore/central-grafana(created)
  rbac.authorization.k8s.io/v1/RoleBinding:
    - grafana-restore/central-grafana(created)
  v1/ConfigMap:
    - grafana-restore/central-grafana(created)
    - grafana-restore/elasticsearch-es-transport-ca-internal(failed)
    - grafana-restore/kube-root-ca.crt(failed)
  v1/Endpoints:
    - grafana-restore/central-grafana(created)
  v1/PersistentVolume:
    - pvc-e3f6578f-08b2-4e79-85f0-76bbf8985b55(skipped)
  v1/PersistentVolumeClaim:
    - grafana-restore/central-grafana(created)
  v1/Pod:
    - grafana-restore/central-grafana-cb88bd49c-fc5br(created)
  v1/Secret:
    - grafana-restore/fpinfra-net-cf-cert(created)
    - grafana-restore/grafana(created)
  v1/Service:
    - grafana-restore/central-grafana(created)
  v1/ServiceAccount:
    - grafana-restore/central-grafana(created)
    - grafana-restore/default(skipped)
  velero.io/v2alpha1/DataUpload:
    - velero/grafana-test-nw7zj(skipped)

Image of working restore pod, with correct data in PV
34d87db1-19ae-4348-8d4e-6599375d7634-image.png

Velero installed from helm: https://vmware-tanzu.github.io/helm-charts
Version: velero:11.1.0
Values

---
image:
  repository: velero/velero
  tag: v1.17.0

# Whether to deploy the restic daemonset.
deployNodeAgent: true

initContainers:
   - name: velero-plugin-for-aws
     image: velero/velero-plugin-for-aws:latest
     imagePullPolicy: IfNotPresent
     volumeMounts:
       - mountPath: /target
         name: plugins

configuration:
  defaultItemOperationTimeout: 2h
  features: EnableCSI
  defaultSnapshotMoveData: true

  backupStorageLocation:
    - name: default
      provider: aws
      bucket: velero
      config:
        region: us-east-1
        s3ForcePathStyle: true
        s3Url: https://s3.location

  # Destination VSL points to LINSTOR snapshot class
  volumeSnapshotLocation:
    - name: linstor
      provider: velero.io/csi
      config:
        snapshotClass: linstor-vsc

credentials:
  useSecret: true
  existingSecret: velero-user


metrics:
  enabled: true

  serviceMonitor:
    enabled: true

  prometheusRule:
    enabled: true
    # Additional labels to add to deployed PrometheusRule
    additionalLabels: {}
    # PrometheusRule namespace. Defaults to Velero namespace.
    # namespace: ""
    # Rules to be deployed
    spec:
      - alert: VeleroBackupPartialFailures
        annotations:
          message: Velero backup {{ $labels.schedule }} has {{ $value | humanizePercentage }} partialy failed backups.
        expr: |-
          velero_backup_partial_failure_total{schedule!=""} / velero_backup_attempt_total{schedule!=""} > 0.25
        for: 15m
        labels:
          severity: warning
      - alert: VeleroBackupFailures
        annotations:
          message: Velero backup {{ $labels.schedule }} has {{ $value | humanizePercentage }} failed backups.
        expr: |-
          velero_backup_failure_total{schedule!=""} / velero_backup_attempt_total{schedule!=""} > 0.25
        for: 15m
        labels:
          severity: warning

Also create the following.

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: linstor-vsc
  labels:
    velero.io/csi-volumesnapshot-class: "true"
driver: linstor.csi.linbit.com
deletionPolicy: Delete

We are using Piraeus operator to use xostor in k8s
https://github.com/piraeusdatastore/piraeus-operator.git
Version: v2.9.1
Values:

---
operator: 
  resources:
    requests:
      cpu: 250m
      memory: 500Mi
    limits:
      memory: 1Gi
installCRDs: true
imageConfigOverride:
- base: quay.io/piraeusdatastore
  components:
    linstor-satellite:
      image: piraeus-server
      tag: v1.29.0
tls:
  certManagerIssuerRef:
    name: step-issuer
    kind: StepClusterIssuer
    group: certmanager.step.sm

Then we just connect to the xostor cluster like external linstor controller.

]]>https://xcp-ng.org/forum/post/98517https://xcp-ng.org/forum/post/98517Wed, 15 Oct 2025 22:54:17 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Wed, 02 Jul 2025 08:31:54 GMT]]>Hello,

I plan to install my XOSTOR cluster on a pool of 7 nodes with 3 replicas, but not all nodes at once because disks are in use.
consider:

  • node1
  • node2
  • node ...
  • node 5
  • node 6
  • node 7.

with 2 disks on each

  • sda: 128GB for the OS
  • sdb: 1TB for local sr ( for now 😄 )

I emptied node 6 & 7.

so, here is what i plan to do:

  • On ALL NODES: setup linstor packages

Run the install script on node 6 & 7 to add their disks
so:

node6# install.sh --disks /dev/sdb
node7# install.sh --disks /dev/sdb

Then, configure the SR and the linstor plugin manager as the following

xe sr-create \ 
    type=linstor name-label=pool-01 \
    host-uuid=XXXX \
    device-config:group-name=linstor_group/thin_device device-config:redundancy=3 shared=true device-config:provisioning=thin

Normally, i should have a linstor cluster running of 2 nodes ( 2 satellite and one controller randomly placed ) with only 2 disks and then, only 2/3 working replicas.

The cluster SHOULD be usable ( i'm right on this point ? )

The next step, would be to move VM from node 5 on it to evacuate node 5. and then add it to the cluster by the following

node5# install.sh --disks /dev/sdb
node5# xe host-call-plugin \
  host-uuid=node5-uuid \
  plugin=linstor-manager \
  fn=addHost args:groupName=linstor_group/thin_device

That should deploy satelite on node 5 and add the disk.

I normally should have 3/3 working replicas and can start to deploy others nodes progressively.

I'm right on the process ?

aS mentionned in the discord, i will post my feedbacks and results from my setup once i finalized it. ( maybe thought a blog post somewhere ).

Thanks to provide xostor in opensource, it's clearly the missing piece for this virtualization stack in opensource ( vs proxmox )

]]>https://xcp-ng.org/forum/post/94730https://xcp-ng.org/forum/post/94730Wed, 02 Jul 2025 08:31:54 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Tue, 01 Jul 2025 15:02:07 GMT]]>@dthenot said in XOSTOR hyperconvergence preview:

@JeffBerntsen That's why I meant, the way to install written in the first post still work in 8.3, the script still work as expected also, it basically only create the VG/LV needed on hosts before you create the SR.

Got it. Thanks!

]]>https://xcp-ng.org/forum/post/94689https://xcp-ng.org/forum/post/94689Tue, 01 Jul 2025 15:02:07 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Tue, 01 Jul 2025 14:52:44 GMT]]>@JeffBerntsen That's why I meant, the way to install written in the first post still work in 8.3, the script still work as expected also, it basically only create the VG/LV needed on hosts before you create the SR.

]]>https://xcp-ng.org/forum/post/94685https://xcp-ng.org/forum/post/94685Tue, 01 Jul 2025 14:52:44 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Mon, 30 Jun 2025 19:43:44 GMT]]>@dthenot said in XOSTOR hyperconvergence preview:

@gb.123 Hello,
The instruction in the first post are still the way to go 🙂

I'm curious about that as well but the first post says that the installation script is only compatible with 8.2 and doesn't mention 8.3. Is that still the case or is the installation script now compatible with 8.3 as well? If not, is there an installation script that is compatible with 8.3?

I know that using XO is the recommended method for installation but I'm interested in an installation script as I would like to try to integrate XOSTOR installation into an XCP-ng installation script I already have which runs via PXE boot.

]]>https://xcp-ng.org/forum/post/94600https://xcp-ng.org/forum/post/94600Mon, 30 Jun 2025 19:43:44 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Mon, 30 Jun 2025 09:43:28 GMT]]>@gb.123 Hello,
The instruction in the first post are still the way to go 🙂

]]>https://xcp-ng.org/forum/post/94541https://xcp-ng.org/forum/post/94541Mon, 30 Jun 2025 09:43:28 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Sun, 29 Jun 2025 20:15:38 GMT]]>@ronan-a @dthenot @Team-Storage

Guys, Can you please clarify which method to use for installing XOSTOR in XCP-ng 8.3 ?

Simple :

yum install xcp-ng-linstor
yum install xcp-ng-release-linstor
./install --disks /dev/nvme0n1 --thin

Or the script in the first post ?
Or Some other script ?

]]>https://xcp-ng.org/forum/post/94534https://xcp-ng.org/forum/post/94534Sun, 29 Jun 2025 20:15:38 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Thu, 26 Jun 2025 12:34:06 GMT]]>@peter_webbird We've already had feedback on CBT and LINSTOR/DRBD, we don't necessarily recommend enabling it. We have a blocking dev card regarding a bug with LVM lvchange command that may fail on CBT volumes used by a XOSTOR SR. We also have other issues related to migration with CBT.

]]>https://xcp-ng.org/forum/post/94436https://xcp-ng.org/forum/post/94436Thu, 26 Jun 2025 12:34:06 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Thu, 26 Jun 2025 06:02:58 GMT]]>Ping @ronan-a or @dthenot

]]>https://xcp-ng.org/forum/post/94412https://xcp-ng.org/forum/post/94412Thu, 26 Jun 2025 06:02:58 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Thu, 26 Jun 2025 00:41:25 GMT]]>Is CBT meant to be supported on XOSTOR?

I've been experimenting with XOSTOR recently, but upon testing a delta-backup, noticed this warning...
couldn't enable CBT

The error message behind this is SR_OPERATION_NOT_SUPPORTED when calling Async.VDI.enable_cbt.

Running xe sr-param-list uuid={uuid} shows the following:

[~]# xe sr-param-list uuid={...}
    uuid ( RO) : {...}
    name-label ( RW): CD6
    name-description ( RW): Array of Kioxia CD6 U.2 drives, one in each Host.
    host ( RO): <shared>
    allowed-operations (SRO): unplug; plug; PBD.create; update; PBD.destroy; VDI.resize; VDI.clone; scan; VDI.snapshot; VDI.mirror; VDI.create; VDI.destroy
    {...etc}
    type ( RO): linstor
    content-type ( RO): user
    shared ( RW): true
    introduced-by ( RO): <not in database>
    is-tools-sr ( RO): false
    other-config (MRW): auto-scan: true
    sm-config (MRO):
    {...etc}

Compared to another SR, the following allowed-operations are missing:

VDI.enable_cbt;
VDI.list_changed_blocks;
VDI.disable_cbt;
VDI.data_destroy;
VDI.set_on_boot

Is this the expected behaviour? Note that this is using XCP-ng 8.2 (I've yet to test out 8.3).

]]>https://xcp-ng.org/forum/post/94407https://xcp-ng.org/forum/post/94407Thu, 26 Jun 2025 00:41:25 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Thu, 19 Jun 2025 15:42:48 GMT]]>Ping @Team-Storage

]]>https://xcp-ng.org/forum/post/94098https://xcp-ng.org/forum/post/94098Thu, 19 Jun 2025 15:42:48 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Thu, 19 Jun 2025 15:24:05 GMT]]>@olivierlambert

Since XOSTOR is now supported on XCP-ng 8.3 LTS, should we use the same script or some other method is required ?
Can you remove the heading which states the script is only compatible with 8.2 ?

]]>https://xcp-ng.org/forum/post/94097https://xcp-ng.org/forum/post/94097Thu, 19 Jun 2025 15:24:05 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Mon, 18 Nov 2024 16:32:20 GMT]]>@Danp Success this this - thanks for the assist. 🙂 Executed with great success:

yum install xcp-ng-linstor
yum install xcp-ng-release-linstor
./install --disks /dev/nvme0n1 --thin
]]>https://xcp-ng.org/forum/post/85956https://xcp-ng.org/forum/post/85956Mon, 18 Nov 2024 16:32:20 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Mon, 18 Nov 2024 13:05:06 GMT]]>@BlueToast This should be fixed now. Please retry the XOSTOR installation.

]]>https://xcp-ng.org/forum/post/85943https://xcp-ng.org/forum/post/85943Mon, 18 Nov 2024 13:05:06 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Sun, 17 Nov 2024 23:03:32 GMT]]>Last week I stood up two XCP-ng hosts with LINSTOR without issue.

This week I am standing up a third XCP-ng host and encounter a show stopping error when I try to install LINSTOR:

Error: Package: xcp-ng-linstor-1.1-3.xcpng8.2.noarch (xcp-ng-linstor)
Requires: sm-linstor
...
Failed to install LINSTOR package: xcp-ng-linstor.

This third host is 100% identical in hardware to the first two hosts. Fresh from-scratch new install of XCP-ng. Here is an output of commands and their results on the third host:

[21:35 bol-xcp3 ~]# wget https://gist.githubusercontent.com/Wescoeur/7bb568c0e09e796710b0ea966882fcac/raw/052b3dfff9c06b1765e51d8de72c90f2f90f475b/gistfile1.txt -O install && chmod +x install
--2024-11-16 21:35:08--  https://gist.githubusercontent.com/Wescoeur/7bb568c0e09e796710b0ea966882fcac/raw/052b3dfff9c06b1765e51d8de72c90f2f90f475b/gistfile1.txt
Resolving gist.githubusercontent.com (gist.githubusercontent.com)... 185.199.111.133, 185.199.110.133, 185.199.109.133, ...
Connecting to gist.githubusercontent.com (gist.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3596 (3.5K) [text/plain]
Saving to: ‘install’

100%[=============================================================================>] 3,596       --.-K/s   in 0s      

2024-11-16 21:35:08 (22.4 MB/s) - ‘install’ saved [3596/3596]

[21:35 bol-xcp3 ~]# ./install --disks /dev/nvme0n1 --thin
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Resolving Dependencies
--> Running transaction check
---> Package xcp-ng-release-linstor.noarch 0:1.3-1.xcpng8.2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=======================================================================================================================
 Package                             Arch                Version                     Repository                   Size
=======================================================================================================================
Installing:
 xcp-ng-release-linstor              noarch              1.3-1.xcpng8.2              xcp-ng-updates              4.0 k

Transaction Summary
=======================================================================================================================
Install  1 Package

Total download size: 4.0 k
Installed size: 477  
Downloading packages:
xcp-ng-release-linstor-1.3-1.xcpng8.2.noarch.rpm                                                | 4.0 kB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : xcp-ng-release-linstor-1.3-1.xcpng8.2.noarch                                                        1/1 
  Verifying  : xcp-ng-release-linstor-1.3-1.xcpng8.2.noarch                                                        1/1 

Installed:
  xcp-ng-release-linstor.noarch 0:1.3-1.xcpng8.2                                                                       

Complete!
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Resolving Dependencies
--> Running transaction check
---> Package xcp-ng-linstor.noarch 0:1.1-3.xcpng8.2 will be installed
--> Processing Dependency: sm-linstor for package: xcp-ng-linstor-1.1-3.xcpng8.2.noarch
--> Finished Dependency Resolution
Error: Package: xcp-ng-linstor-1.1-3.xcpng8.2.noarch (xcp-ng-linstor)
           Requires: sm-linstor
           Available: sm-2.30.8-2.1.0.linstor.4.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-2.1.0.linstor.5.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-2.1.0.linstor.6.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-2.3.0.linstor.1.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-2.3.0.linstor.2.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-7.1.0.linstor.2.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-10.1.0.linstor.1.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-10.1.0.linstor.2.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-10.1.0.linstor.3.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-12.1.0.linstor.2.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-12.1.0.linstor.3.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Available: sm-2.30.8-12.1.0.linstor.4.xcpng8.2.x86_64 (xcp-ng-linstor)
               sm-linstor
           Installed: sm-2.30.8-13.1.xcpng8.2.x86_64 (@xcp-ng-updates)
               Not found
           Available: sm-2.29.1-1.2.xcpng8.2.x86_64 (xcp-ng-base)
               Not found
           Available: sm-2.30.4-1.1.xcpng8.2.x86_64 (xcp-ng-updates)
               Not found
           Available: sm-2.30.4-1.1.0.linstor.8.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.6-1.1.xcpng8.2.x86_64 (xcp-ng-updates)
               Not found
           Available: sm-2.30.6-1.1.0.linstor.1.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.6-1.2.0.linstor.1.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.7-1.1.0.linstor.1.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.7-1.2.0.linstor.1.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.7-1.3.xcpng8.2.x86_64 (xcp-ng-updates)
               Not found
           Available: sm-2.30.7-1.3.0.linstor.1.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.7-1.3.0.linstor.2.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.7-1.3.0.linstor.3.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.7-1.3.0.linstor.7.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.7-1.3.0.linstor.8.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.8-2.1.xcpng8.2.x86_64 (xcp-ng-updates)
               Not found
           Available: sm-2.30.8-2.1.0.linstor.1.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.8-2.1.0.linstor.2.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.8-2.1.0.linstor.3.xcpng8.2.x86_64 (xcp-ng-linstor)
               Not found
           Available: sm-2.30.8-2.3.xcpng8.2.x86_64 (xcp-ng-updates)
               Not found
           Available: sm-2.30.8-7.1.xcpng8.2.x86_64 (xcp-ng-updates)
               Not found
           Available: sm-2.30.8-10.1.xcpng8.2.x86_64 (xcp-ng-updates)
               Not found
           Available: sm-2.30.8-12.1.xcpng8.2.x86_64 (xcp-ng-updates)
               Not found
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest
Failed to install LINSTOR package: xcp-ng-linstor.

It seems the sm* package(s) are missing ? I don't know what to do.

Edit 2024-11-17 1702 CST: yum install xcp-ng-release-linstor indicates xcp-ng-release-linstor is already installed and latest version.

[16:14 bol-xcp3 ~]# yum install xcp-ng-release-linstor
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
Package xcp-ng-release-linstor-1.3-1.xcpng8.2.noarch already installed and latest version
Nothing to do
]]>https://xcp-ng.org/forum/post/85925https://xcp-ng.org/forum/post/85925Sun, 17 Nov 2024 23:03:32 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Fri, 01 Nov 2024 21:22:21 GMT]]>OK we have debugged and improved this process, so including it here if it helps anyone else.

How to migrate resources between XOSTOR (linstor) clusters. This also works with piraeus-operator, which we use for k8s.

Manually moving listor resource with thin_send_recv

Migration of data

Commands
# PV: pvc-6408a214-6def-44c4-8d9a-bebb67be5510
# S: pgdata-snapshot
# s: 10741612544B

#get size
lvs --noheadings --units B -o lv_size linstor_group/pvc-6408a214-6def-44c4-8d9a-bebb67be5510_00000

#prep
lvcreate -V 10741612544B --thinpool linstor_group/thin_device -n pvc-6408a214-6def-44c4-8d9a-bebb67be5510_00000 linstor_group

#create snapshot
linstor --controller original-xostor-server s create pvc-6408a214-6def-44c4-8d9a-bebb67be5510  pgdata-snapshot

#send
thin_send linstor_group/pvc-6408a214-6def-44c4-8d9a-bebb67be5510_00000_pgdata-snapshot 2>/dev/null | ssh root@new-xostor-server-01 thin_recv linstor_group/pvc-6408a214-6def-44c4-8d9a-bebb67be5510_00000 2>/dev/null
Walk-through

Prep migration

[13:29 original-xostor-server ~]# lvs --noheadings --units B -o lv_size linstor_group/pvc-12aca72c-d94a-4c09-8102-0a6646906f8d_00000
  26851934208B


[13:53 new-xostor-server-01 ~]# lvcreate -V 26851934208B --thinpool linstor_group/thin_device -n pvc-12aca72c-d94a-4c09-8102-0a6646906f8d_00000 linstor_group
  Logical volume "pvc-12aca72c-d94a-4c09-8102-0a6646906f8d_00000" created.

Create snapshot

15:35:03] jonathon@jonathon-framework:~$ linstor --controller original-xostor-server s create pvc-12aca72c-d94a-4c09-8102-0a6646906f8d s_test
SUCCESS:
Description:
    New snapshot 's_test' of resource 'pvc-12aca72c-d94a-4c09-8102-0a6646906f8d' registered.
Details:
    Snapshot 's_test' of resource 'pvc-12aca72c-d94a-4c09-8102-0a6646906f8d' UUID is: 3a07d2fd-6dc3-4994-b13f-8c3a2bb206b8
SUCCESS:
    Suspended IO of '[pvc-12aca72c-d94a-4c09-8102-0a6646906f8d]' on 'ovbh-vprod-k8s04-worker02' for snapshot
SUCCESS:
    Suspended IO of '[pvc-12aca72c-d94a-4c09-8102-0a6646906f8d]' on 'original-xostor-server' for snapshot
SUCCESS:
    Took snapshot of '[pvc-12aca72c-d94a-4c09-8102-0a6646906f8d]' on 'ovbh-vprod-k8s04-worker02'
SUCCESS:
    Took snapshot of '[pvc-12aca72c-d94a-4c09-8102-0a6646906f8d]' on 'original-xostor-server'
SUCCESS:
    Resumed IO of '[pvc-12aca72c-d94a-4c09-8102-0a6646906f8d]' on 'ovbh-vprod-k8s04-worker02' after snapshot
SUCCESS:
    Resumed IO of '[pvc-12aca72c-d94a-4c09-8102-0a6646906f8d]' on 'original-xostor-server' after snapshot

Migration

[13:53 original-xostor-server ~]# thin_send /dev/linstor_group/pvc-12aca72c-d94a-4c09-8102-0a6646906f8d_00000_s_test 2>/dev/null | ssh root@new-xostor-server-01 thin_recv linstor_group/pvc-12aca72c-d94a-4c09-8102-0a6646906f8d_00000 2>/dev/null

Need to yeet errors on both ends of command or it will fail.

This is the same setup process for replica-1 or replica-3. For replica-3 can target new-xostor-server-01 each time, for replica-1 be sure to spread them out right.

Replica-3 Setup

Explanation

thin_send to new-xostor-server-01, will need to run commands to force sync of data to replicas.

Commands
# PV: pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
# snapshot: snipeit-snapshot
# size: 21483225088B

#get size
lvs --noheadings --units B -o lv_size linstor_group/pvc-96cbebbe-f827-4a47-ae95-38b078e0d584_00000

#prep
lvcreate -V 21483225088B --thinpool linstor_group/thin_device -n pvc-96cbebbe-f827-4a47-ae95-38b078e0d584_00000 linstor_group

#create snapshot
linstor --controller original-xostor-server s create pvc-96cbebbe-f827-4a47-ae95-38b078e0d584 snipeit-snapshot
linstor --controller original-xostor-server s l | grep -e 'snipeit-snapshot'

#send
thin_send linstor_group/pvc-96cbebbe-f827-4a47-ae95-38b078e0d584_00000_snipeit-snapshot 2>/dev/null | ssh root@new-xostor-server-01 thin_recv linstor_group/pvc-96cbebbe-f827-4a47-ae95-38b078e0d584_00000 2>/dev/null

#linstor setup
linstor --controller new-xostor-server-01 resource-definition create pvc-96cbebbe-f827-4a47-ae95-38b078e0d584 --resource-group sc-74e1434b-b435-587e-9dea-fa067deec898
linstor --controller new-xostor-server-01 volume-definition create pvc-96cbebbe-f827-4a47-ae95-38b078e0d584 21483225088B --storage-pool xcp-sr-linstor_group_thin_device

linstor --controller new-xostor-server-01 resource create --storage-pool xcp-sr-linstor_group_thin_device --providers LVM_THIN new-xostor-server-01 pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
linstor --controller new-xostor-server-01 resource create --auto-place +1 pvc-96cbebbe-f827-4a47-ae95-38b078e0d584

#Run the following on the node with the data. This is the prefered command
drbdadm invalidate-remote pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
#Run the following on the node without the data. This is just for reference
drbdadm invalidate pvc-96cbebbe-f827-4a47-ae95-38b078e0d584

linstor --controller new-xostor-server-01 r l | grep -e 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584'

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
  annotations:
    pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
  finalizers:
    - external-provisioner.volume.kubernetes.io/finalizer
    - kubernetes.io/pv-protection
    - external-attacher/linstor-csi-linbit-com
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 20Gi # Ensure this matches the actual size of the LINSTOR volume
  persistentVolumeReclaimPolicy: Retain
  storageClassName: linstor-replica-three # Adjust to the storage class you want to use
  volumeMode: Filesystem
  csi:
    driver: linstor.csi.linbit.com
    fsType: ext4
    volumeHandle: pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
    volumeAttributes:
      linstor.csi.linbit.com/mount-options: ''
      linstor.csi.linbit.com/post-mount-xfs-opts: ''
      linstor.csi.linbit.com/uses-volume-context: 'true'
      linstor.csi.linbit.com/remote-access-policy: 'true'
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
    volume.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
  finalizers:
    - kubernetes.io/pvc-protection
  name: pp-snipeit-pvc
  namespace: snipe-it
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 20Gi
  storageClassName: linstor-replica-three
  volumeMode: Filesystem
  volumeName: pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
Walk-through
jonathon@jonathon-framework:~$ linstor --controller new-xostor-server-01 resource-definition create pvc-96cbebbe-f827-4a47-ae95-38b078e0d584 --resource-group sc-74e1434b-b435-587e-9dea-fa067deec898
SUCCESS:
Description:
    New resource definition 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' created.
Details:
    Resource definition 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' UUID is: 772692e2-3fca-4069-92e9-2bef22c68a6f
jonathon@jonathon-framework:~$ linstor --controller new-xostor-server-01 volume-definition create pvc-96cbebbe-f827-4a47-ae95-38b078e0d584 21483225088B --storage-pool xcp-sr-linstor_group_thin_device
SUCCESS:
    Successfully set property key(s): StorPoolName
SUCCESS:
    New volume definition with number '0' of resource definition 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' created.
jonathon@jonathon-framework:~$ linstor --controller new-xostor-server-01 resource create --storage-pool xcp-sr-linstor_group_thin_device --providers LVM_THIN new-xostor-server-01 pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
SUCCESS:
    Successfully set property key(s): StorPoolName
INFO:
    Updated pvc-96cbebbe-f827-4a47-ae95-38b078e0d584 DRBD auto verify algorithm to 'crct10dif-pclmul'
SUCCESS:
Description:
    New resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' on node 'new-xostor-server-01' registered.
Details:
    Resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' on node 'new-xostor-server-01' UUID is: 3072aaae-4a34-453e-bdc6-facb47809b3d
SUCCESS:
Description:
    Volume with number '0' on resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' on node 'new-xostor-server-01' successfully registered
Details:
    Volume UUID is: 52b11ef6-ec50-42fb-8710-1d3f8c15c657
SUCCESS:
    Created resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' on 'new-xostor-server-01'
jonathon@jonathon-framework:~$ linstor --controller new-xostor-server-01 resource create --auto-place +1 pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
SUCCESS:
    Successfully set property key(s): StorPoolName
SUCCESS:
    Successfully set property key(s): StorPoolName
SUCCESS:
Description:
    Resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' successfully autoplaced on 2 nodes
Details:
    Used nodes (storage pool name): 'new-xostor-server-02 (xcp-sr-linstor_group_thin_device)', 'new-xostor-server-03 (xcp-sr-linstor_group_thin_device)'
INFO:
    Resource-definition property 'DrbdOptions/Resource/quorum' updated from 'off' to 'majority' by auto-quorum
INFO:
    Resource-definition property 'DrbdOptions/Resource/on-no-quorum' updated from 'off' to 'suspend-io' by auto-quorum
SUCCESS:
    Added peer(s) 'new-xostor-server-02', 'new-xostor-server-03' to resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' on 'new-xostor-server-01'
SUCCESS:
    Created resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' on 'new-xostor-server-02'
SUCCESS:
    Created resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' on 'new-xostor-server-03'
SUCCESS:
Description:
    Resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' on 'new-xostor-server-03' ready
Details:
    Auto-placing resource: pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
SUCCESS:
Description:
    Resource 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584' on 'new-xostor-server-02' ready
Details:
    Auto-placing resource: pvc-96cbebbe-f827-4a47-ae95-38b078e0d584

At this point

jonathon@jonathon-framework:~$ linstor --controller new-xostor-server-01 v l | grep -e 'pvc-96cbebbe-f827-4a47-ae95-38b078e0d584'
| new-xostor-server-01                         | pvc-96cbebbe-f827-4a47-ae95-38b078e0d584        | xcp-sr-linstor_group_thin_device |     0 |    1032 | /dev/drbd1032 |   9.20 GiB | Unused | UpToDate |
| new-xostor-server-02                         | pvc-96cbebbe-f827-4a47-ae95-38b078e0d584        | xcp-sr-linstor_group_thin_device |     0 |    1032 | /dev/drbd1032 | 112.73 MiB | Unused | UpToDate |
| new-xostor-server-03                         | pvc-96cbebbe-f827-4a47-ae95-38b078e0d584        | xcp-sr-linstor_group_thin_device |     0 |    1032 | /dev/drbd1032 | 112.73 MiB | Unused | UpToDate |

To force the sync, run the following command on the node with the data

drbdadm invalidate-remote pvc-96cbebbe-f827-4a47-ae95-38b078e0d584

This will kick it to get the data re-synced.

[14:51 new-xostor-server-01 ~]# drbdadm invalidate-remote pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
[14:51 new-xostor-server-01 ~]# drbdadm status pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
pvc-96cbebbe-f827-4a47-ae95-38b078e0d584 role:Secondary
  disk:UpToDate
  new-xostor-server-02 role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:1.14
  new-xostor-server-03 role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:1.18

[14:51 new-xostor-server-01 ~]# drbdadm status pvc-96cbebbe-f827-4a47-ae95-38b078e0d584
pvc-96cbebbe-f827-4a47-ae95-38b078e0d584 role:Secondary
  disk:UpToDate
  new-xostor-server-02 role:Secondary
    peer-disk:UpToDate
  new-xostor-server-03 role:Secondary
    peer-disk:UpToDate

See: https://github.com/LINBIT/linstor-server/issues/389

Replica-1setup

# PV: pvc-6408a214-6def-44c4-8d9a-bebb67be5510
# S: pgdata-snapshot
# s: 10741612544B

#get size
lvs --noheadings --units B -o lv_size linstor_group/pvc-6408a214-6def-44c4-8d9a-bebb67be5510_00000

#prep
lvcreate -V 10741612544B --thinpool linstor_group/thin_device -n pvc-6408a214-6def-44c4-8d9a-bebb67be5510_00000 linstor_group

#create snapshot
linstor --controller original-xostor-server s create pvc-6408a214-6def-44c4-8d9a-bebb67be5510  pgdata-snapshot

#send
thin_send linstor_group/pvc-6408a214-6def-44c4-8d9a-bebb67be5510_00000_pgdata-snapshot 2>/dev/null | ssh root@new-xostor-server-01 thin_recv linstor_group/pvc-6408a214-6def-44c4-8d9a-bebb67be5510_00000 2>/dev/null

# 1
linstor --controller new-xostor-server-01 resource-definition create pvc-6408a214-6def-44c4-8d9a-bebb67be5510  --resource-group sc-b066e430-6206-5588-a490-cc91ecef53d6
linstor --controller new-xostor-server-01 volume-definition create pvc-6408a214-6def-44c4-8d9a-bebb67be5510  10741612544B --storage-pool xcp-sr-linstor_group_thin_device
linstor --controller new-xostor-server-01 resource create new-xostor-server-01 pvc-6408a214-6def-44c4-8d9a-bebb67be5510 


---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: pvc-6408a214-6def-44c4-8d9a-bebb67be5510
  annotations:
    pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
  finalizers:
    - external-provisioner.volume.kubernetes.io/finalizer
    - kubernetes.io/pv-protection
    - external-attacher/linstor-csi-linbit-com
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 10Gi # Ensure this matches the actual size of the LINSTOR volume
  persistentVolumeReclaimPolicy: Retain
  storageClassName: linstor-replica-one-local # Adjust to the storage class you want to use
  volumeMode: Filesystem
  csi:
    driver: linstor.csi.linbit.com
    fsType: ext4
    volumeHandle: pvc-6408a214-6def-44c4-8d9a-bebb67be5510
    volumeAttributes:
      linstor.csi.linbit.com/mount-options: ''
      linstor.csi.linbit.com/post-mount-xfs-opts: ''
      linstor.csi.linbit.com/uses-volume-context: 'true'
      linstor.csi.linbit.com/remote-access-policy: |
        - fromSame:
            - xcp-ng/node
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: xcp-ng/node
              operator: In
              values:
                - new-xostor-server-01
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
    volume.kubernetes.io/selected-node: ovbh-vtest-k8s01-worker01
    volume.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
  finalizers:
    - kubernetes.io/pvc-protection
  name: acid-merch-2
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: linstor-replica-one-local
  volumeMode: Filesystem
  volumeName: pvc-6408a214-6def-44c4-8d9a-bebb67be5510
0sorkon created this issue in LINBIT/linstor-server

open Incorrect replica size after resource-definition auto-place #389

]]>https://xcp-ng.org/forum/post/85280https://xcp-ng.org/forum/post/85280Fri, 01 Nov 2024 21:22:21 GMT<![CDATA[Reply to XOSTOR hyperconvergence preview on Tue, 29 Oct 2024 12:08:49 GMT]]>The typical hypervisor node was provisioned with at least 4 network ports, but most had 8 to allow trunking/resilience for every connection:

Management + live migration (Isolated)
World (Trunk)
iSCSI-A
iSCSI-B

The DotHills were 4 ports per controller, and the two controllers were configured as:

Management
iSCSI-A
iSCSI-B
Spare

The iSCSI-A and iSCSI-B were simple web-managed procurves with 24 ports each.

We had this scaled to 12 hypervisor nodes in 3 pools, all running off the same underlying dothill (dual redundant controller) hardware. We had provisioned some local SSD storage in each hypervisor for any nodes that were struggling for IOPs but ultimately never needed that.

]]>https://xcp-ng.org/forum/post/85018https://xcp-ng.org/forum/post/85018Tue, 29 Oct 2024 12:08:49 GMT