Wonder if these are just issues that will be fixed and not come back by manually cleaning it up. Held off at first cause I did not want to manually clean up, and then have it happen again.
Will post in a bit when it finishes and will monitor this one for a few.
[18:45 ovbh-pprod-xen02 ~]# drbdadm status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
drbdsetup status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 node-id:1 role:Primary suspended:no force-io-failures:no
write-ordering:flush
volume:0 minor:1013 disk:UpToDate backing_dev:/dev/linstor_group/xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1_00000 quorum:yes open:yes
size:315195168 read:296465702 written:1319196709 al-writes:790653 bm-writes:361 upper-pending:1 lower-pending:1 al-suspended:no blocked:no
ovbh-pprod-xen01 node-id:0 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0 rs-in-flight:0
volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
received:0 sent:1319197101 out-of-sync:0 pending:1 unacked:0
ovbh-pprod-xen05 node-id:2 connection:Connected role:Secondary tls:no congested:no ap-in-flight:8 rs-in-flight:0
volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
received:0 sent:1292149931 out-of-sync:0 pending:1 unacked:0
[18:45 ovbh-pprod-xen02 ~]# drbdadm invalidate-remote xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
[18:45 ovbh-pprod-xen02 ~]# drbdadm status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
drbdsetup status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 node-id:1 role:Primary suspended:no force-io-failures:no
write-ordering:flush
volume:0 minor:1013 disk:UpToDate backing_dev:/dev/linstor_group/xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1_00000 quorum:yes open:yes
size:315195168 read:300587646 written:1319298558 al-writes:790664 bm-writes:15389 upper-pending:0 lower-pending:1 al-suspended:no blocked:no
ovbh-pprod-xen01 node-id:0 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0 rs-in-flight:0
volume:0 replication:SyncSource peer-disk:Inconsistent done:0.69 resync-suspended:no
received:0 sent:1321387882 out-of-sync:313023380 pending:0 unacked:1 dbdt1:35.66 eta:8573
ovbh-pprod-xen05 node-id:2 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0 rs-in-flight:0
volume:0 replication:SyncSource peer-disk:Inconsistent done:0.65 resync-suspended:no
received:0 sent:1294222956 out-of-sync:313141144 pending:0 unacked:0 dbdt1:30.42 eta:10052
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ DRBD,STORAGE │ Unused │ Ok │ UpToDate │ Yes │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ DRBD,STORAGE │ InUse │ Ok │ UpToDate │ Yes │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ DRBD,STORAGE │ Unused │ Ok │ UpToDate │ Yes │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ DRBD,STORAGE │ Unused │ Ok │ SyncTarget(0.04%) │ Yes │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ DRBD,STORAGE │ InUse │ Ok │ UpToDate │ Yes │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ DRBD,STORAGE │ Unused │ Ok │ Inconsistent │ Yes │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ DRBD,STORAGE │ Unused │ Ok │ SyncTarget(0.64%) │ Yes │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ DRBD,STORAGE │ InUse │ Ok │ UpToDate │ Yes │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ DRBD,STORAGE │ Unused │ Ok │ SyncTarget(0.61%) │ Yes │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01 │ DRBD,STORAGE │ Unused │ Ok │ SyncTarget(1.00%) │ Yes │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02 │ DRBD,STORAGE │ InUse │ Ok │ UpToDate │ Yes │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05 │ DRBD,STORAGE │ Unused │ Ok │ SyncTarget(0.96%) │ Yes │