So, controller failover works. I used instructions here to test drbd-reactor failover: https://linbit.com/blog/drbd-reactor-promoter/
I'm seeing an error in linstor error-reports list that has to do with how linstor queries free space on thin provisioned LVM storage. It traces back to this ticket. https://github.com/LINBIT/linstor-server/issues/80
ERROR REPORT 65558791-33400-000000
============================================================
Application: LINBIT® LINSTOR
Module: Satellite
Version: 1.24.2
Build ID: adb19ca96a07039401023410c1ea116f09929295
Build time: 2023-08-30T05:15:08+00:00
Error time: 2023-11-15 22:08:11
Node: node0
============================================================
Reported error:
===============
Description:
Expected 3 columns, but got 2
Cause:
Failed to parse line: thin_device;23044370202624;
Additional information:
External command: vgs --config devices { filter=['a|/dev/sdn|','a|/dev/sdk|','a|/dev/sdj|','a|/dev/sdm|','a|/dev/sdl|','a|/dev/sdg|','a|/dev/sdf|','a|/dev/sdi|','a|/dev/sdh|','a|/dev/sdc|','a|/dev/sde|','a|/dev/sdd|','r|.*|'] } -o lv_name,lv_size,data_percent --units b --separator ; --noheadings --nosuffix linstor_group/thin_device
Category: LinStorException
Class name: StorageException
Class canonical name: com.linbit.linstor.storage.StorageException
Generated at: Method 'getThinFreeSize', Source file 'LvmUtils.java', Line #399
Error message: Unable to parse free thin sizes
ErrorContext: Description: Expected 3 columns, but got 2
Cause: Failed to parse line: thin_device;23044370202624;
Details: External command: vgs --config devices { filter=['a|/dev/sdn|','a|/dev/sdk|','a|/dev/sdj|','a|/dev/sdm|','a|/dev/sdl|','a|/dev/sdg|','a|/dev/sdf|','a|/dev/sdi|','a|/dev/sdh|','a|/dev/sdc|','a|/dev/sde|','a|/dev/sdd|','r|.*|'] } -o lv_name,lv_size,data_percent --units b --separator ; --noheadings --nosuffix linstor_group/thin_device
Call backtrace:
Method Native Class:Line number
getThinFreeSize N com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:399
getSpaceInfo N com.linbit.linstor.layer.storage.lvm.LvmThinProvider:406
getStoragePoolSpaceInfo N com.linbit.linstor.layer.storage.StorageLayer:441
getSpaceInfo N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1116
getSpaceInfo N com.linbit.linstor.core.devmgr.DeviceManagerImpl:1816
getStoragePoolSpaceInfo N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:325
applyChanges N com.linbit.linstor.core.apicallhandler.StltStorPoolApiCallHandler:274
applyFullSync N com.linbit.linstor.core.apicallhandler.StltApiCallHandler:330
execute N com.linbit.linstor.api.protobuf.FullSync:113
executeNonReactive N com.linbit.linstor.proto.CommonMessageProcessor:534
lambda$execute$14 N com.linbit.linstor.proto.CommonMessageProcessor:509
doInScope N com.linbit.linstor.core.apicallhandler.ScopeRunner:149
lambda$fluxInScope$0 N com.linbit.linstor.core.apicallhandler.ScopeRunner:76
call N reactor.core.publisher.MonoCallable:72
trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:127
subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49
subscribe N reactor.core.publisher.Flux:8759
onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:195
request N reactor.core.publisher.Operators$ScalarSubscription:2545
onSubscribe N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141
subscribe N reactor.core.publisher.MonoJust:55
subscribe N reactor.core.publisher.MonoDeferContextual:55
subscribe N reactor.core.publisher.Flux:8773
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:427
slowPath N reactor.core.publisher.FluxArray$ArraySubscription:127
request N reactor.core.publisher.FluxArray$ArraySubscription:100
onSubscribe N reactor.core.publisher.FluxFlatMap$FlatMapMain:371
subscribe N reactor.core.publisher.FluxMerge:70
subscribe N reactor.core.publisher.Flux:8773
onComplete N reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:258
subscribe N reactor.core.publisher.FluxConcatArray:78
subscribe N reactor.core.publisher.InternalFluxOperator:62
subscribe N reactor.core.publisher.FluxDefer:54
subscribe N reactor.core.publisher.Flux:8773
onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:427
drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453
drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724
onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256
drainFused N reactor.core.publisher.SinkManyUnicast:319
drain N reactor.core.publisher.SinkManyUnicast:362
tryEmitNext N reactor.core.publisher.SinkManyUnicast:237
tryEmitNext N reactor.core.publisher.SinkManySerialized:100
processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:392
doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:227
lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:164
onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:185
runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:440
run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:527
call N reactor.core.scheduler.WorkerTask:84
call N reactor.core.scheduler.WorkerTask:37
run N java.util.concurrent.FutureTask:264
run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304
runWorker N java.util.concurrent.ThreadPoolExecutor:1128
run N java.util.concurrent.ThreadPoolExecutor$Worker:628
run N java.lang.Thread:829
END OF ERROR REPORT.
I think the vgs query is improperly formatted for this version of device-mapper-persistent-data
[12:31 node0 ~]# vgs -o lv_name,lv_size,data_percent --units b --noheadings --separator ;
vgs: option '--separator' requires an argument
Error during parsing of command line.
but it works if formatted like this:
[12:32 node0 ~]# vgs -o lv_name,lv_size,data_percent --units b --noheadings --separator=";"
MGT;4194304B;
VHD-d959f7a9-2bd1-4ac5-83af-1724336a73d0;532676608B;
thin_device;23044370202624B;6.96
xcp-persistent-database_00000;1077936128B;13.85
xcp-volume-fda3d913-47cc-4a8d-8a54-3364c8ae722a_00000;86083895296B;25.20
xcp-volume-8ddb8f7e-a549-4c53-a9d5-9b2e40d3810e_00000;215197155328B;2.35
xcp-volume-43467341-30c8-4fec-b807-81334d0dd309_00000;215197155328B;2.52
xcp-volume-5283a6e0-4e95-4aca-b5e1-7eb3fea7fcd3_00000;2194921226240B;69.30
xcp-volume-907e72d1-4389-4425-8e1e-e53a4718cb92_00000;86088089600B;0.60
xcp-volume-4c368a33-d0af-4f1d-9f7d-486a1df1d028_00000;86088089600B;0.06
xcp-volume-2bd88964-3feb-401a-afc1-c88c790cc206_00000;86092283904B;24.81
xcp-volume-833eba2a-a70b-4787-b78a-afef8cc0e14d_00000;86092283904B;0.04
xcp-volume-81809c66-5763-4558-919a-591b864d3f22_00000;215197155328B;4.66
xcp-volume-9fa2ec95-9bea-45ae-a583-6f1941a614e7_00000;86096478208B;0.04
xcp-volume-5dbfaef0-cc83-43a8-bba1-469d65bc3460_00000;215205543936B;6.12
xcp-volume-1e2dd480-a505-46fc-a6e8-ac8d4341a213_00000;215209738240B;0.02
xcp-volume-603ac344-edf1-43d7-8c27-eecfd7e6d627_00000;215209738240B;2.19
In fact, vgs --separator accepts pretty much any character except semicolon. Maybe it's a problem with this version of LVM2?
[12:37 node0 ~]# yum info device-mapper-persistent-data.x86_64
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
* xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
* xcp-ng-updates: mirrors.xcp-ng.org
Installed Packages
Name : device-mapper-persistent-data
Arch : x86_64
Version : 0.7.3
Release : 3.el7
Size : 1.2 M
Repo : installed
From repo : install
