Hello, @burbilog
I think theses backup doesn't come from your backup plan.
Is there maybe snapshots on the VM ? 2022 is "very old"
Your full backups include memory which is not "classical" in a incremental backups.
Hello, @burbilog
I think theses backup doesn't come from your backup plan.
Is there maybe snapshots on the VM ? 2022 is "very old"
Your full backups include memory which is not "classical" in a incremental backups.
Hello,
I tried on a new pool.
a little different scenario since i don't create xostor for now, on my previous example, i tried to add a node as replacement of an existing one..
I just run the install script only on node 1.
When i try make node2 join the pool, i reproduce the incompatible sm error i got previously.
The things which is "bizarre", is i don't have the license issue i got on Xen-orchestra. ( maybe it was finally not related ? )
Here is the complete logs.
pool.mergeInto
{
"sources": [
"17510fe0-db23-9414-f3df-2941bd34f8dc"
],
"target": "cc91fcdc-c7a8-a44c-65b3-a76dced49252",
"force": true
}
{
"code": "POOL_JOINING_SM_FEATURES_INCOMPATIBLE",
"params": [
"OpaqueRef:090b8da1-9654-066c-84f9-7ab15cb101fd",
""
],
"call": {
"duration": 1061,
"method": "pool.join_force",
"params": [
"* session id *",
"<MASTER_IP>",
"root",
"* obfuscated *"
]
},
"message": "POOL_JOINING_SM_FEATURES_INCOMPATIBLE(OpaqueRef:090b8da1-9654-066c-84f9-7ab15cb101fd, )",
"name": "XapiError",
"stack": "XapiError: POOL_JOINING_SM_FEATURES_INCOMPATIBLE(OpaqueRef:090b8da1-9654-066c-84f9-7ab15cb101fd, )
at Function.wrap (file:///etc/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12)
at file:///etc/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:38:21
at runNextTicks (node:internal/process/task_queues:60:5)
at processImmediate (node:internal/timers:454:9)
at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}```
Hello,
we have nodes with multiple disk group.
Does someone experienced multiple xostor on different local group ?
In our case
It should works easily because we gonna register 2 xostor, but maybe should we put all disks in the same VG ? but in this case, the max vdi size will be lower ?
I purpose to edit the install script to provide group name selection to permit this use case easily.
It's ok ?
Hello @ronan-a
Just to be sure, you want logs of the node which want to join or the master ?
Bonne journée
Hello @ronan-a
I will reproduce the case, i will re-destroy one hypervisor and retrigger the case.
Thank you @ronan-a et @olivierlambert
If you need me to tests some special case don't hesit, we have a pool dedicated for this
Hello, @ronan-a
I will reinstall my hypervisor this week.
I will reproduce it and then, resend you the logs.
Bonne journée,
In this case @olivierlambert
The new node doesn't have any linstor version installed before.
I planned to install it AFTER adding it to a pool, but that's not possible. It's a "dependency" to join the pool. ( as i understand ^^ )
If my interpretation and solution is correct, can i propose a MR to add it thought the documentation ? ( for the CLI part at least, the xo side looks to be a bug for me )
Hello @olivierlambert
All my hosts were up to date ( verified many times ).
But once the linstor installed, the error disappeared.
Here is the my ssh i keeped open ( thanks Mac )
[00:55 node-3 ~]# xe pool-join master-address=XXXXX master-username=XXXX master-password="YYYYY"
The host joining the pool has an incompatible set of sm features from the pool coordinator. Make sure the sm are of the same versions and try again.
pool_sm_ref: OpaqueRef:4d762998-79ee-8e88-1f0f-5b74aac9af85
candidate_sm_ref:
[00:56 node-3 ~]# ^C
[00:58 node-3 ~]# yum install -y xcp-ng-release-linstor
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
* xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
* xcp-ng-updates: mirrors.xcp-ng.org
Resolving Dependencies
--> Running transaction check
---> Package xcp-ng-release-linstor.noarch 0:1.4-2.xcpng8.3 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
============================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================
Installing:
xcp-ng-release-linstor noarch 1.4-2.xcpng8.3 xcp-ng-updates 3.9 k
Transaction Summary
============================================================================================================================================================
Install 1 Package
Total download size: 3.9 k
Installed size: 477
Downloading packages:
xcp-ng-release-linstor-1.4-2.xcpng8.3.noarch.rpm | 3.9 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : xcp-ng-release-linstor-1.4-2.xcpng8.3.noarch 1/1
Verifying : xcp-ng-release-linstor-1.4-2.xcpng8.3.noarch 1/1
Installed:
xcp-ng-release-linstor.noarch 0:1.4-2.xcpng8.3
Complete!
[00:58 node-3 ~]# yum install -y xcp-ng-linstor
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
* xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
* xcp-ng-updates: mirrors.xcp-ng.org
xcp-ng-linstor/signature | 473 B 00:00:00
Retrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-xcpng
Importing GPG key 0x3FD3AC9E:
Userid : "XCP-ng Key (XCP-ng Official Signing Key) <security@xcp-ng.org>"
Fingerprint: 34ac 2eb6 8248 fed6 d076 838a cd75 783a 3fd3 ac9e
Package : xcp-ng-release-8.3.0-32.x86_64 (@xcp:main/$releasever)
From : /etc/pki/rpm-gpg/RPM-GPG-KEY-xcpng
xcp-ng-linstor/signature | 3.0 kB 00:00:00 !!!
xcp-ng-linstor/primary_db | 26 kB 00:00:00
Resolving Dependencies
--> Running transaction check
---> Package xcp-ng-linstor.noarch 0:1.2-2.xcpng8.3 will be installed
--> Processing Dependency: drbd for package: xcp-ng-linstor-1.2-2.xcpng8.3.noarch
--> Processing Dependency: drbd-reactor for package: xcp-ng-linstor-1.2-2.xcpng8.3.noarch
--> Processing Dependency: http-nbd-transfer for package: xcp-ng-linstor-1.2-2.xcpng8.3.noarch
--> Processing Dependency: kmod-drbd for package: xcp-ng-linstor-1.2-2.xcpng8.3.noarch
--> Processing Dependency: linstor-client for package: xcp-ng-linstor-1.2-2.xcpng8.3.noarch
--> Processing Dependency: linstor-controller for package: xcp-ng-linstor-1.2-2.xcpng8.3.noarch
--> Processing Dependency: linstor-satellite for package: xcp-ng-linstor-1.2-2.xcpng8.3.noarch
--> Running transaction check
---> Package drbd.x86_64 0:9.28.0-1.el7 will be installed
--> Processing Dependency: drbd-bash-completion = 9.28.0-1.el7 for package: drbd-9.28.0-1.el7.x86_64
--> Processing Dependency: drbd-pacemaker = 9.28.0-1.el7 for package: drbd-9.28.0-1.el7.x86_64
--> Processing Dependency: drbd-udev = 9.28.0-1.el7 for package: drbd-9.28.0-1.el7.x86_64
--> Processing Dependency: drbd-utils = 9.28.0-1.el7 for package: drbd-9.28.0-1.el7.x86_64
--> Processing Dependency: drbd-xen = 9.28.0-1.el7 for package: drbd-9.28.0-1.el7.x86_64
---> Package drbd-reactor.x86_64 0:1.4.1-1 will be installed
---> Package http-nbd-transfer.x86_64 0:1.6.0-1.xcpng8.3 will be installed
--> Processing Dependency: nbdkit for package: http-nbd-transfer-1.6.0-1.xcpng8.3.x86_64
---> Package kmod-drbd.x86_64 0:9.2.11-1.1.xcpng8.3 will be installed
---> Package linstor-client.noarch 0:1.23.0-1.xcpng8.3 will be installed
--> Processing Dependency: python-linstor >= 1.23.0 for package: linstor-client-1.23.0-1.xcpng8.3.noarch
---> Package linstor-controller.noarch 0:1.29.0-1.el7_9 will be installed
--> Processing Dependency: linstor-common = 1.29.0 for package: linstor-controller-1.29.0-1.el7_9.noarch
--> Processing Dependency: jre-11-headless for package: linstor-controller-1.29.0-1.el7_9.noarch
---> Package linstor-satellite.noarch 0:1.29.0-1.el7_9 will be installed
--> Running transaction check
---> Package drbd-bash-completion.x86_64 0:9.28.0-1.el7 will be installed
---> Package drbd-pacemaker.x86_64 0:9.28.0-1.el7 will be installed
---> Package drbd-udev.x86_64 0:9.28.0-1.el7 will be installed
---> Package drbd-utils.x86_64 0:9.28.0-1.el7 will be installed
---> Package drbd-xen.x86_64 0:9.28.0-1.el7 will be installed
---> Package java-11-openjdk-headless.x86_64 1:11.0.23.0.9-2.el7_9 will be installed
--> Processing Dependency: copy-jdk-configs >= 3.3 for package: 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64
--> Processing Dependency: tzdata-java >= 2023d for package: 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64
--> Processing Dependency: javapackages-tools for package: 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64
--> Processing Dependency: libasound.so.2(ALSA_0.9)(64bit) for package: 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64
--> Processing Dependency: libasound.so.2(ALSA_0.9.0rc4)(64bit) for package: 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64
--> Processing Dependency: lksctp-tools(x86-64) for package: 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64
--> Processing Dependency: pcsc-lite-libs(x86-64) for package: 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64
--> Processing Dependency: libasound.so.2()(64bit) for package: 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64
--> Processing Dependency: libharfbuzz.so.0()(64bit) for package: 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64
---> Package linstor-common.noarch 0:1.29.0-1.el7_9 will be installed
---> Package nbdkit.x86_64 0:1.8.0-4.el7 will be installed
---> Package python-linstor.noarch 0:1.23.0-1.xcpng8.3 will be installed
--> Running transaction check
---> Package alsa-lib.x86_64 0:1.1.4.1-2.el7 will be installed
---> Package copy-jdk-configs.noarch 0:3.3-10.el7_5 will be installed
---> Package harfbuzz.x86_64 0:1.3.2-1.el7 will be installed
--> Processing Dependency: libgraphite2.so.3()(64bit) for package: harfbuzz-1.3.2-1.el7.x86_64
---> Package javapackages-tools.noarch 0:3.4.1-11.el7 will be installed
--> Processing Dependency: python-javapackages = 3.4.1-11.el7 for package: javapackages-tools-3.4.1-11.el7.noarch
--> Processing Dependency: libxslt for package: javapackages-tools-3.4.1-11.el7.noarch
---> Package lksctp-tools.x86_64 0:1.0.17-2.el7 will be installed
---> Package pcsc-lite-libs.x86_64 0:1.8.8-8.el7 will be installed
---> Package tzdata-java.noarch 0:2024a-1.el7 will be installed
--> Running transaction check
---> Package graphite2.x86_64 0:1.3.10-1.el7_3 will be installed
---> Package libxslt.x86_64 0:1.1.28-5.el7 will be installed
---> Package python-javapackages.noarch 0:3.4.1-11.el7 will be installed
--> Processing Dependency: python-lxml for package: python-javapackages-3.4.1-11.el7.noarch
--> Running transaction check
---> Package python-lxml.x86_64 0:3.2.1-4.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
============================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================
Installing:
xcp-ng-linstor noarch 1.2-2.xcpng8.3 xcp-ng-linstor 3.6 k
Installing for dependencies:
alsa-lib x86_64 1.1.4.1-2.el7 xcp-ng-base 421 k
copy-jdk-configs noarch 3.3-10.el7_5 xcp-ng-base 21 k
drbd x86_64 9.28.0-1.el7 xcp-ng-linstor 22 k
drbd-bash-completion x86_64 9.28.0-1.el7 xcp-ng-linstor 8.5 k
drbd-pacemaker x86_64 9.28.0-1.el7 xcp-ng-linstor 47 k
drbd-reactor x86_64 1.4.1-1 xcp-ng-linstor 4.3 M
drbd-udev x86_64 9.28.0-1.el7 xcp-ng-linstor 7.0 k
drbd-utils x86_64 9.28.0-1.el7 xcp-ng-linstor 657 k
drbd-xen x86_64 9.28.0-1.el7 xcp-ng-linstor 9.8 k
graphite2 x86_64 1.3.10-1.el7_3 xcp-ng-base 115 k
harfbuzz x86_64 1.3.2-1.el7 xcp-ng-base 176 k
http-nbd-transfer x86_64 1.6.0-1.xcpng8.3 xcp-ng-updates 30 k
java-11-openjdk-headless x86_64 1:11.0.23.0.9-2.el7_9 xcp-ng-base 39 M
javapackages-tools noarch 3.4.1-11.el7 xcp-ng-base 72 k
kmod-drbd x86_64 9.2.11-1.1.xcpng8.3 xcp-ng-linstor 2.7 M
libxslt x86_64 1.1.28-5.el7 xcp-ng-base 241 k
linstor-client noarch 1.23.0-1.xcpng8.3 xcp-ng-linstor 266 k
linstor-common noarch 1.29.0-1.el7_9 xcp-ng-linstor 54 M
linstor-controller noarch 1.29.0-1.el7_9 xcp-ng-linstor 14 M
linstor-satellite noarch 1.29.0-1.el7_9 xcp-ng-linstor 880 k
lksctp-tools x86_64 1.0.17-2.el7 xcp-ng-base 87 k
nbdkit x86_64 1.8.0-4.el7 xcp-ng-base 57 k
pcsc-lite-libs x86_64 1.8.8-8.el7 xcp-ng-base 34 k
python-javapackages noarch 3.4.1-11.el7 xcp-ng-base 31 k
python-linstor noarch 1.23.0-1.xcpng8.3 xcp-ng-linstor 163 k
python-lxml x86_64 3.2.1-4.el7 xcp-ng-base 758 k
tzdata-java noarch 2024a-1.el7 xcp-ng-base 186 k
Transaction Summary
============================================================================================================================================================
Install 1 Package (+27 Dependent packages)
Total download size: 118 M
Installed size: 286 M
Downloading packages:
(1/28): drbd-9.28.0-1.el7.x86_64.rpm | 22 kB 00:00:00
(2/28): drbd-bash-completion-9.28.0-1.el7.x86_64.rpm | 8.5 kB 00:00:00
(3/28): drbd-pacemaker-9.28.0-1.el7.x86_64.rpm | 47 kB 00:00:00
(4/28): drbd-udev-9.28.0-1.el7.x86_64.rpm | 7.0 kB 00:00:00
(5/28): copy-jdk-configs-3.3-10.el7_5.noarch.rpm | 21 kB 00:00:00
(6/28): drbd-utils-9.28.0-1.el7.x86_64.rpm | 657 kB 00:00:00
(7/28): drbd-xen-9.28.0-1.el7.x86_64.rpm | 9.8 kB 00:00:00
(8/28): drbd-reactor-1.4.1-1.x86_64.rpm | 4.3 MB 00:00:00
(9/28): alsa-lib-1.1.4.1-2.el7.x86_64.rpm | 421 kB 00:00:00
(10/28): graphite2-1.3.10-1.el7_3.x86_64.rpm | 115 kB 00:00:00
(11/28): harfbuzz-1.3.2-1.el7.x86_64.rpm | 176 kB 00:00:00
(12/28): kmod-drbd-9.2.11-1.1.xcpng8.3.x86_64.rpm | 2.7 MB 00:00:00
(13/28): javapackages-tools-3.4.1-11.el7.noarch.rpm | 72 kB 00:00:00
(14/28): linstor-client-1.23.0-1.xcpng8.3.noarch.rpm | 266 kB 00:00:00
(15/28): http-nbd-transfer-1.6.0-1.xcpng8.3.x86_64.rpm | 30 kB 00:00:00
(16/28): libxslt-1.1.28-5.el7.x86_64.rpm | 241 kB 00:00:00
(17/28): linstor-controller-1.29.0-1.el7_9.noarch.rpm | 14 MB 00:00:00
(18/28): java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64.rpm | 39 MB 00:00:00
(19/28): linstor-satellite-1.29.0-1.el7_9.noarch.rpm | 880 kB 00:00:00
(20/28): lksctp-tools-1.0.17-2.el7.x86_64.rpm | 87 kB 00:00:00
(21/28): pcsc-lite-libs-1.8.8-8.el7.x86_64.rpm | 34 kB 00:00:00
(22/28): python-linstor-1.23.0-1.xcpng8.3.noarch.rpm | 163 kB 00:00:00
(23/28): nbdkit-1.8.0-4.el7.x86_64.rpm | 57 kB 00:00:00
(24/28): python-javapackages-3.4.1-11.el7.noarch.rpm | 31 kB 00:00:00
(25/28): xcp-ng-linstor-1.2-2.xcpng8.3.noarch.rpm | 3.6 kB 00:00:00
(26/28): python-lxml-3.2.1-4.el7.x86_64.rpm | 758 kB 00:00:00
(27/28): linstor-common-1.29.0-1.el7_9.noarch.rpm | 54 MB 00:00:00
(28/28): tzdata-java-2024a-1.el7.noarch.rpm | 186 kB 00:00:00
------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 88 MB/s | 118 MB 00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : drbd-utils-9.28.0-1.el7.x86_64 1/28
Installing : libxslt-1.1.28-5.el7.x86_64 2/28
Installing : tzdata-java-2024a-1.el7.noarch 3/28
Installing : python-lxml-3.2.1-4.el7.x86_64 4/28
Installing : python-javapackages-3.4.1-11.el7.noarch 5/28
Installing : javapackages-tools-3.4.1-11.el7.noarch 6/28
Installing : kmod-drbd-9.2.11-1.1.xcpng8.3.x86_64 7/28
Installing : drbd-pacemaker-9.28.0-1.el7.x86_64 8/28
Installing : drbd-xen-9.28.0-1.el7.x86_64 9/28
Installing : drbd-udev-9.28.0-1.el7.x86_64 10/28
Installing : drbd-bash-completion-9.28.0-1.el7.x86_64 11/28
Installing : drbd-9.28.0-1.el7.x86_64 12/28
Installing : drbd-reactor-1.4.1-1.x86_64 13/28
Installing : alsa-lib-1.1.4.1-2.el7.x86_64 14/28
Installing : nbdkit-1.8.0-4.el7.x86_64 15/28
Installing : http-nbd-transfer-1.6.0-1.xcpng8.3.x86_64 16/28
Installing : pcsc-lite-libs-1.8.8-8.el7.x86_64 17/28
Installing : lksctp-tools-1.0.17-2.el7.x86_64 18/28
Installing : python-linstor-1.23.0-1.xcpng8.3.noarch 19/28
Installing : linstor-client-1.23.0-1.xcpng8.3.noarch 20/28
Installing : copy-jdk-configs-3.3-10.el7_5.noarch 21/28
Installing : graphite2-1.3.10-1.el7_3.x86_64 22/28
Installing : harfbuzz-1.3.2-1.el7.x86_64 23/28
Installing : 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64 24/28
Installing : linstor-common-1.29.0-1.el7_9.noarch 25/28
Installing : linstor-satellite-1.29.0-1.el7_9.noarch 26/28
Installing : linstor-controller-1.29.0-1.el7_9.noarch 27/28
Installing : xcp-ng-linstor-1.2-2.xcpng8.3.noarch 28/28
Verifying : drbd-utils-9.28.0-1.el7.x86_64 1/28
Verifying : graphite2-1.3.10-1.el7_3.x86_64 2/28
Verifying : python-lxml-3.2.1-4.el7.x86_64 3/28
Verifying : harfbuzz-1.3.2-1.el7.x86_64 4/28
Verifying : tzdata-java-2024a-1.el7.noarch 5/28
Verifying : copy-jdk-configs-3.3-10.el7_5.noarch 6/28
Verifying : python-javapackages-3.4.1-11.el7.noarch 7/28
Verifying : linstor-common-1.29.0-1.el7_9.noarch 8/28
Verifying : kmod-drbd-9.2.11-1.1.xcpng8.3.x86_64 9/28
Verifying : drbd-pacemaker-9.28.0-1.el7.x86_64 10/28
Verifying : python-linstor-1.23.0-1.xcpng8.3.noarch 11/28
Verifying : 1:java-11-openjdk-headless-11.0.23.0.9-2.el7_9.x86_64 12/28
Verifying : lksctp-tools-1.0.17-2.el7.x86_64 13/28
Verifying : http-nbd-transfer-1.6.0-1.xcpng8.3.x86_64 14/28
Verifying : linstor-client-1.23.0-1.xcpng8.3.noarch 15/28
Verifying : libxslt-1.1.28-5.el7.x86_64 16/28
Verifying : linstor-satellite-1.29.0-1.el7_9.noarch 17/28
Verifying : pcsc-lite-libs-1.8.8-8.el7.x86_64 18/28
Verifying : javapackages-tools-3.4.1-11.el7.noarch 19/28
Verifying : drbd-xen-9.28.0-1.el7.x86_64 20/28
Verifying : nbdkit-1.8.0-4.el7.x86_64 21/28
Verifying : drbd-9.28.0-1.el7.x86_64 22/28
Verifying : drbd-udev-9.28.0-1.el7.x86_64 23/28
Verifying : drbd-bash-completion-9.28.0-1.el7.x86_64 24/28
Verifying : linstor-controller-1.29.0-1.el7_9.noarch 25/28
Verifying : drbd-reactor-1.4.1-1.x86_64 26/28
Verifying : xcp-ng-linstor-1.2-2.xcpng8.3.noarch 27/28
Verifying : alsa-lib-1.1.4.1-2.el7.x86_64 28/28
Installed:
xcp-ng-linstor.noarch 0:1.2-2.xcpng8.3
Dependency Installed:
alsa-lib.x86_64 0:1.1.4.1-2.el7 copy-jdk-configs.noarch 0:3.3-10.el7_5 drbd.x86_64 0:9.28.0-1.el7
drbd-bash-completion.x86_64 0:9.28.0-1.el7 drbd-pacemaker.x86_64 0:9.28.0-1.el7 drbd-reactor.x86_64 0:1.4.1-1
drbd-udev.x86_64 0:9.28.0-1.el7 drbd-utils.x86_64 0:9.28.0-1.el7 drbd-xen.x86_64 0:9.28.0-1.el7
graphite2.x86_64 0:1.3.10-1.el7_3 harfbuzz.x86_64 0:1.3.2-1.el7 http-nbd-transfer.x86_64 0:1.6.0-1.xcpng8.3
java-11-openjdk-headless.x86_64 1:11.0.23.0.9-2.el7_9 javapackages-tools.noarch 0:3.4.1-11.el7 kmod-drbd.x86_64 0:9.2.11-1.1.xcpng8.3
libxslt.x86_64 0:1.1.28-5.el7 linstor-client.noarch 0:1.23.0-1.xcpng8.3 linstor-common.noarch 0:1.29.0-1.el7_9
linstor-controller.noarch 0:1.29.0-1.el7_9 linstor-satellite.noarch 0:1.29.0-1.el7_9 lksctp-tools.x86_64 0:1.0.17-2.el7
nbdkit.x86_64 0:1.8.0-4.el7 pcsc-lite-libs.x86_64 0:1.8.8-8.el7 python-javapackages.noarch 0:3.4.1-11.el7
python-linstor.noarch 0:1.23.0-1.xcpng8.3 python-lxml.x86_64 0:3.2.1-4.el7 tzdata-java.noarch 0:2024a-1.el7
Complete!
[00:59 node-3 ~]# xe-toolstack-restart
Executing xe-toolstack-restart
done.
[00:59 node-3 ~]# xe pool-join master-address=XXXXX master-username=XXXX master-password="YYYYY"
Host agent will restart and attempt to join pool in 10.000 seconds...
Hello,
My post title may be missleading.
I built a pool with 3 nodes and 3 replicas.
When i remove node 3 ( linstor forget + xo forget node ).
The snapshot of disks are not possible anymore.
SR_BACKEND_FAILURE_1200(, local variable 'e' referenced before assignment, )
However, i have 2 healthy node on 3. it should work no ?
️ I don't need support, it's just an observation in our test pool, maybe a bug, i report
Best regards
Hello,
When you try to add a fresh node to a pool using XOSTOR
If you run the following error from CLI :
[00:54 node-3 ~]# xe pool-join master-address=XXXX master-username=XXX master-password="YYYY"
The host joining the pool has an incompatible set of sm features from the pool coordinator. Make sure the sm are of the same versions and try again.
pool_sm_ref: OpaqueRef:4d762998-79ee-8e88-1f0f-5b74aac9af85
candidate_sm_ref:
OR this one thought XO
app.getLicenses is not a function
It's maybe because you didn't installed Linstor package on it.
Run
yum install -y xcp-ng-release-linstor
yum install -y xcp-ng-linstor
xe-toolstack-restart
Or simply use the install.sh script provided in the first post of this thread: https://xcp-ng.org/forum/topic/5361/xostor-hyperconvergence-preview
Then, you can add your node without issue
[00:59 node-3 ~]# xe pool-join master-address=XXXX master-username=XXXX master-password="YYYYY"
Host agent will restart and attempt to join pool in 10.000 seconds...
Hey @Danp ,
I can keep my cluster without this node up to Monday, do you have an idea / do you want to investigate on this case ?
Hello, @Danp
I just reproduced it.
All my nodes are up to date
( my xo have 7 commits behind master, i read the commit's log of my missing commit, the commits are not related to my issue. ).
In my scenario, i had a pool with 3 nodes
I reinstall node3 after a disaster ( i force forgot node 3).
Now, i can't add the host back to the pool.
I also tried to update the host after installation ( usually, i do this after ).
But it doesn't works more.
I will not add my node thought xcpng center to permit you further investigation
Here is the detailed operation json
{
"id": "0mcwehjzy",
"properties": {
"method": "pool.mergeInto",
"params": {
"sources": [
"64365465-fd4e-25b6-3db2-2cdcd9edba5e"
],
"target": "a92ca4ca-caac-83b9-fa00-4bb75cb48f6c",
"force": true
},
"name": "API call: pool.mergeInto",
"userId": "63a0dbaf-ba2d-4028-b80f-fe49f56686b2",
"type": "api.call"
},
"start": 1752092249470,
"status": "failure",
"updatedAt": 1752092249473,
"end": 1752092249472,
"result": {
"message": "app.getLicenses is not a function",
"name": "TypeError",
"stack": "TypeError: app.getLicenses is not a function\n at enforceHostsHaveLicense (file:///etc/xen-orchestra/packages/xo-server/src/xo-mixins/pool.mjs:15:30)\n at Pools.apply (file:///etc/xen-orchestra/packages/xo-server/src/xo-mixins/pool.mjs:80:13)\n at Pools.mergeInto (/etc/xen-orchestra/node_modules/golike-defer/src/index.js:85:19)\n at Xo.mergeInto (file:///etc/xen-orchestra/packages/xo-server/src/api/pool.mjs:311:15)\n at Task.runInside (/etc/xen-orchestra/@vates/task/index.js:175:22)\n at Task.run (/etc/xen-orchestra/@vates/task/index.js:159:20)\n at Api.#callApiMethod (file:///etc/xen-orchestra/packages/xo-server/src/xo-mixins/api.mjs:469:18)"
}
}```
@olivierlambert No,
For once, i followed the installation step carefully ^^'
Hello,
I have a XCP-NG 8.3 pool running 3 hosts with XOSTOR in a 3 replicas with HA enabled.
This setup should permit to lose up to 2 nodes without dataloss
Initial informations:
I was able to migrate VDI on XOSTOR successfuly ( even if when i start a transfert into xostor, i need to wait ~1 minute before the transfert really start ( i see that in XO ).
In my first tests, i will shut node 3 (which is neither master, not linstor controller )
For my first test, i didn't want to kill the linster controller host / pool master immediately, it should be my second test / third test )
I stopped node 3 ( poweroff from IPMI ).
However, then entire pool was dead.
In xensource.log
of all remaining nodes ( node 1, and node 2 ), i can see:
Jul 5 15:32:20 node2 xapi: [debug||0 |Checking HA configuration D:9b97e277d80e|helpers] /usr/libexec/xapi/cluster-stack/xhad/ha_start_daemon exited with code 8 [stdout = ''; stderr = 'Sat Jul 5 15:32:20 CEST 2025 ha_start_daemon: the HA daemon stopped without forming a liveset (8)\x0A']
Jul 5 15:32:20 node2 xapi: [ warn||0 |Checking HA configuration D:9b97e277d80e|xapi_ha] /usr/libexec/xapi/cluster-stack/xhad/ha_start_daemon returned MTC_EXIT_CAN_NOT_ACCESS_STATEFILE (State-File is inaccessible)
Jul 5 15:32:20 gco-002-rbx-002 xapi: [ warn||0 |Checking HA configuration D:9b97e277d80e|xapi_ha] ha_start_daemon failed with MTC_EXIT_CAN_NOT_ACCESS_STATEFILE: will contact existing master and check if HA is still enabled
However, the storage layer was ok
[15:33 node1 linstor-controller]# linstor node list
╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node ┊ NodeType ┊ Addresses ┊ State ┊
╞════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ h1 ┊ COMBINED ┊ 192.168.1.1:3366 (PLAIN) ┊ Online ┊
┊ h2 ┊ COMBINED ┊ 192.168.1.2:3366 (PLAIN) ┊ Online ┊
┊ h3 ┊ COMBINED ┊ 192.168.1.3:3366 (PLAIN) ┊ OFFLINE (Auto-eviction: 2025-07-05 16:33:42) ┊
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Volumes was also OK using linstor volume list
[15:33 r1 linstor-controller]# linstor volume list
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
┊ Node ┊ Resource ┊ StoragePool ┊ VolNr ┊ MinorNr ┊ DeviceName ┊ Allocated ┊ InUse ┊ State ┊
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
┊ r1 ┊ xcp-persistent-database ┊ xcp-sr-linstor_group_thin_device ┊ 0 ┊ 1000 ┊ /dev/drbd1000 ┊ 52.74 MiB ┊ InUse ┊ UpToDate ┊
┊ r2 ┊ xcp-persistent-database ┊ xcp-sr-linstor_group_thin_device ┊ 0 ┊ 1000 ┊ /dev/drbd1000 ┊ 6.99 MiB ┊ Unused ┊ UpToDate ┊
┊ r3 ┊ xcp-persistent-database ┊ xcp-sr-linstor_group_thin_device ┊ 0 ┊ 1000 ┊ /dev/drbd1000 ┊ 6.99 MiB ┊ ┊ Unknown ┊
( i didn't put the entire list of volumes, in writing my post, i'm feel a bit stupid to don't save the entire output ).
I finally solved my issue by re-upping node3 which promote itself as master, but i need to perform this test again because the result is not the expected one.
Did i do something wrong ?
Hello, @nikade
Yeah i think like this already, but i'm surprised by the result of the calcul.
I should i have either: 3 nodes which can die, or "1", but i should not have the same number right ?
in my case, i run the command in a 3 hosts cluster and it return me 3
as value. which disturb me.
Best regards,
Hello @fred974
Did you find your answer ?
I don't understand clearly how a 4 nodes pool can tolerate 4 nodes dead.
Hello,
I plan to install my XOSTOR cluster on a pool of 7 nodes with 3 replicas, but not all nodes at once because disks are in use.
consider:
with 2 disks on each
I emptied node 6 & 7.
so, here is what i plan to do:
Run the install script on node 6 & 7 to add their disks
so:
node6# install.sh --disks /dev/sdb
node7# install.sh --disks /dev/sdb
Then, configure the SR and the linstor plugin manager as the following
xe sr-create \
type=linstor name-label=pool-01 \
host-uuid=XXXX \
device-config:group-name=linstor_group/thin_device device-config:redundancy=3 shared=true device-config:provisioning=thin
Normally, i should have a linstor cluster running of 2 nodes ( 2 satellite and one controller randomly placed ) with only 2 disks and then, only 2/3 working replicas.
The cluster SHOULD be usable ( i'm right on this point ? )
The next step, would be to move VM from node 5 on it to evacuate node 5. and then add it to the cluster by the following
node5# install.sh --disks /dev/sdb
node5# xe host-call-plugin \
host-uuid=node5-uuid \
plugin=linstor-manager \
fn=addHost args:groupName=linstor_group/thin_device
That should deploy satelite on node 5 and add the disk.
I normally should have 3/3 working replicas and can start to deploy others nodes progressively.
I'm right on the process ?
aS mentionned in the discord, i will post my feedbacks and results from my setup once i finalized it. ( maybe thought a blog post somewhere ).
Thanks to provide xostor in opensource, it's clearly the missing piece for this virtualization stack in opensource ( vs proxmox )
Hello, @TheNorthernLight
XOStor is now available in 8.3 since it's LTS