So, controller failover works. I used instructions here to test drbd-reactor failover: https://linbit.com/blog/drbd-reactor-promoter/

I'm seeing an error in linstor error-reports list that has to do with how linstor queries free space on thin provisioned LVM storage. It traces back to this ticket. https://github.com/LINBIT/linstor-server/issues/80

ERROR REPORT 65558791-33400-000000 ============================================================ Application: LINBIT® LINSTOR Module: Satellite Version: 1.24.2 Build ID: adb19ca96a07039401023410c1ea116f09929295 Build time: 2023-08-30T05:15:08+00:00 Error time: 2023-11-15 22:08:11 Node: node0 ============================================================ Reported error: =============== Description: Expected 3 columns, but got 2 Cause: Failed to parse line: thin_device;23044370202624; Additional information: External command: vgs --config devices { filter=['a|/dev/sdn|','a|/dev/sdk|','a|/dev/sdj|','a|/dev/sdm|','a|/dev/sdl|','a|/dev/sdg|','a|/dev/sdf|','a|/dev/sdi|','a|/dev/sdh|','a|/dev/sdc|','a|/dev/sde|','a|/dev/sdd|','r|.*|'] } -o lv_name,lv_size,data_percent --units b --separator ; --noheadings --nosuffix linstor_group/thin_device Category: LinStorException Class name: StorageException Class canonical name: com.linbit.linstor.storage.StorageException Generated at: Method 'getThinFreeSize', Source file 'LvmUtils.java', Line #399 Error message: Unable to parse free thin sizes ErrorContext: Description: Expected 3 columns, but got 2 Cause: Failed to parse line: thin_device;23044370202624; Details: External command: vgs --config devices { filter=['a|/dev/sdn|','a|/dev/sdk|','a|/dev/sdj|','a|/dev/sdm|','a|/dev/sdl|','a|/dev/sdg|','a|/dev/sdf|','a|/dev/sdi|','a|/dev/sdh|','a|/dev/sdc|','a|/dev/sde|','a|/dev/sdd|','r|.*|'] } -o lv_name,lv_size,data_percent --units b --separator ; --noheadings --nosuffix linstor_group/thin_device Call backtrace: Method Native Class:Line number getThinFreeSize N com.linbit.linstor.layer.storage.lvm.utils.LvmUtils:399 getSpaceInfo N com.linbit.linstor.layer.storage.lvm.LvmThinProvider:406 getStoragePoolSpaceInfo N com.linbit.linstor.layer.storage.StorageLayer:441 getSpaceInfo N com.linbit.linstor.core.devmgr.DeviceHandlerImpl:1116 getSpaceInfo N com.linbit.linstor.core.devmgr.DeviceManagerImpl:1816 getStoragePoolSpaceInfo N com.linbit.linstor.core.apicallhandler.StltApiCallHandlerUtils:325 applyChanges N com.linbit.linstor.core.apicallhandler.StltStorPoolApiCallHandler:274 applyFullSync N com.linbit.linstor.core.apicallhandler.StltApiCallHandler:330 execute N com.linbit.linstor.api.protobuf.FullSync:113 executeNonReactive N com.linbit.linstor.proto.CommonMessageProcessor:534 lambda$execute$14 N com.linbit.linstor.proto.CommonMessageProcessor:509 doInScope N com.linbit.linstor.core.apicallhandler.ScopeRunner:149 lambda$fluxInScope$0 N com.linbit.linstor.core.apicallhandler.ScopeRunner:76 call N reactor.core.publisher.MonoCallable:72 trySubscribeScalarMap N reactor.core.publisher.FluxFlatMap:127 subscribeOrReturn N reactor.core.publisher.MonoFlatMapMany:49 subscribe N reactor.core.publisher.Flux:8759 onNext N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:195 request N reactor.core.publisher.Operators$ScalarSubscription:2545 onSubscribe N reactor.core.publisher.MonoFlatMapMany$FlatMapManyMain:141 subscribe N reactor.core.publisher.MonoJust:55 subscribe N reactor.core.publisher.MonoDeferContextual:55 subscribe N reactor.core.publisher.Flux:8773 onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:427 slowPath N reactor.core.publisher.FluxArray$ArraySubscription:127 request N reactor.core.publisher.FluxArray$ArraySubscription:100 onSubscribe N reactor.core.publisher.FluxFlatMap$FlatMapMain:371 subscribe N reactor.core.publisher.FluxMerge:70 subscribe N reactor.core.publisher.Flux:8773 onComplete N reactor.core.publisher.FluxConcatArray$ConcatArraySubscriber:258 subscribe N reactor.core.publisher.FluxConcatArray:78 subscribe N reactor.core.publisher.InternalFluxOperator:62 subscribe N reactor.core.publisher.FluxDefer:54 subscribe N reactor.core.publisher.Flux:8773 onNext N reactor.core.publisher.FluxFlatMap$FlatMapMain:427 drainAsync N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:453 drain N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:724 onNext N reactor.core.publisher.FluxFlattenIterable$FlattenIterableSubscriber:256 drainFused N reactor.core.publisher.SinkManyUnicast:319 drain N reactor.core.publisher.SinkManyUnicast:362 tryEmitNext N reactor.core.publisher.SinkManyUnicast:237 tryEmitNext N reactor.core.publisher.SinkManySerialized:100 processInOrder N com.linbit.linstor.netcom.TcpConnectorPeer:392 doProcessMessage N com.linbit.linstor.proto.CommonMessageProcessor:227 lambda$processMessage$2 N com.linbit.linstor.proto.CommonMessageProcessor:164 onNext N reactor.core.publisher.FluxPeek$PeekSubscriber:185 runAsync N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:440 run N reactor.core.publisher.FluxPublishOn$PublishOnSubscriber:527 call N reactor.core.scheduler.WorkerTask:84 call N reactor.core.scheduler.WorkerTask:37 run N java.util.concurrent.FutureTask:264 run N java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask:304 runWorker N java.util.concurrent.ThreadPoolExecutor:1128 run N java.util.concurrent.ThreadPoolExecutor$Worker:628 run N java.lang.Thread:829 END OF ERROR REPORT.

I think the vgs query is improperly formatted for this version of device-mapper-persistent-data

[12:31 node0 ~]# vgs -o lv_name,lv_size,data_percent --units b --noheadings --separator ; vgs: option '--separator' requires an argument Error during parsing of command line.

but it works if formatted like this:

[12:32 node0 ~]# vgs -o lv_name,lv_size,data_percent --units b --noheadings --separator=";" MGT;4194304B; VHD-d959f7a9-2bd1-4ac5-83af-1724336a73d0;532676608B; thin_device;23044370202624B;6.96 xcp-persistent-database_00000;1077936128B;13.85 xcp-volume-fda3d913-47cc-4a8d-8a54-3364c8ae722a_00000;86083895296B;25.20 xcp-volume-8ddb8f7e-a549-4c53-a9d5-9b2e40d3810e_00000;215197155328B;2.35 xcp-volume-43467341-30c8-4fec-b807-81334d0dd309_00000;215197155328B;2.52 xcp-volume-5283a6e0-4e95-4aca-b5e1-7eb3fea7fcd3_00000;2194921226240B;69.30 xcp-volume-907e72d1-4389-4425-8e1e-e53a4718cb92_00000;86088089600B;0.60 xcp-volume-4c368a33-d0af-4f1d-9f7d-486a1df1d028_00000;86088089600B;0.06 xcp-volume-2bd88964-3feb-401a-afc1-c88c790cc206_00000;86092283904B;24.81 xcp-volume-833eba2a-a70b-4787-b78a-afef8cc0e14d_00000;86092283904B;0.04 xcp-volume-81809c66-5763-4558-919a-591b864d3f22_00000;215197155328B;4.66 xcp-volume-9fa2ec95-9bea-45ae-a583-6f1941a614e7_00000;86096478208B;0.04 xcp-volume-5dbfaef0-cc83-43a8-bba1-469d65bc3460_00000;215205543936B;6.12 xcp-volume-1e2dd480-a505-46fc-a6e8-ac8d4341a213_00000;215209738240B;0.02 xcp-volume-603ac344-edf1-43d7-8c27-eecfd7e6d627_00000;215209738240B;2.19

In fact, vgs --separator accepts pretty much any character except semicolon. Maybe it's a problem with this version of LVM2?

[12:37 node0 ~]# yum info device-mapper-persistent-data.x86_64 Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile Excluding mirror: updates.xcp-ng.org * xcp-ng-base: mirrors.xcp-ng.org Excluding mirror: updates.xcp-ng.org * xcp-ng-updates: mirrors.xcp-ng.org Installed Packages Name : device-mapper-persistent-data Arch : x86_64 Version : 0.7.3 Release : 3.el7 Size : 1.2 M Repo : installed From repo : install solohin created this issue in LINBIT/linstor-server closed Unable to parse free thin sizes error on Satellite #80