@poddingue Turns out it is all secondary-to-secondary out of sync. So that makes it less intense, though if primary dies I wonder how it will resolve this, or if it will become split brained. Still unsure of how it happened, but the ones I manually cleaned up have not come back. Going to continue to manually clear them up. If it happens again I will have an alert setup to notify me, and I have all the xcp-ng logs and everything to be able to see what happened.

If that happens I will post here with details and logs for the resource so we can see how it occurs.

Wonder if these are just issues that will be fixed and not come back by manually cleaning it up. Held off at first cause I did not want to manually clean up, and then have it happen again.

Will post in a bit when it finishes and will monitor this one for a few.

[18:45 ovbh-pprod-xen02 ~]# drbdadm status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
drbdsetup status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 node-id:1 role:Primary suspended:no force-io-failures:no
    write-ordering:flush
  volume:0 minor:1013 disk:UpToDate backing_dev:/dev/linstor_group/xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1_00000 quorum:yes open:yes
      size:315195168 read:296465702 written:1319196709 al-writes:790653 bm-writes:361 upper-pending:1 lower-pending:1 al-suspended:no blocked:no
  ovbh-pprod-xen01 node-id:0 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0 rs-in-flight:0
    volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
        received:0 sent:1319197101 out-of-sync:0 pending:1 unacked:0
  ovbh-pprod-xen05 node-id:2 connection:Connected role:Secondary tls:no congested:no ap-in-flight:8 rs-in-flight:0
    volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
        received:0 sent:1292149931 out-of-sync:0 pending:1 unacked:0

[18:45 ovbh-pprod-xen02 ~]# drbdadm invalidate-remote xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
[18:45 ovbh-pprod-xen02 ~]# drbdadm status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
drbdsetup status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 node-id:1 role:Primary suspended:no force-io-failures:no
    write-ordering:flush
  volume:0 minor:1013 disk:UpToDate backing_dev:/dev/linstor_group/xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1_00000 quorum:yes open:yes
      size:315195168 read:300587646 written:1319298558 al-writes:790664 bm-writes:15389 upper-pending:0 lower-pending:1 al-suspended:no blocked:no
  ovbh-pprod-xen01 node-id:0 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0 rs-in-flight:0
    volume:0 replication:SyncSource peer-disk:Inconsistent done:0.69 resync-suspended:no
        received:0 sent:1321387882 out-of-sync:313023380 pending:0 unacked:1 dbdt1:35.66 eta:8573
  ovbh-pprod-xen05 node-id:2 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0 rs-in-flight:0
    volume:0 replication:SyncSource peer-disk:Inconsistent done:0.65 resync-suspended:no
        received:0 sent:1294222956 out-of-sync:313141144 pending:0 unacked:0 dbdt1:30.42 eta:10052

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01                         │ DRBD,STORAGE │ Unused │ Ok    │ UpToDate │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02                         │ DRBD,STORAGE │ InUse  │ Ok    │ UpToDate │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05                         │ DRBD,STORAGE │ Unused │ Ok    │ UpToDate │ Yes  │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01                         │ DRBD,STORAGE │ Unused │ Ok    │ SyncTarget(0.04%) │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02                         │ DRBD,STORAGE │ InUse  │ Ok    │          UpToDate │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05                         │ DRBD,STORAGE │ Unused │ Ok    │      Inconsistent │ Yes  │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01                         │ DRBD,STORAGE │ Unused │ Ok    │ SyncTarget(0.64%) │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02                         │ DRBD,STORAGE │ InUse  │ Ok    │          UpToDate │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05                         │ DRBD,STORAGE │ Unused │ Ok    │ SyncTarget(0.61%) │ Yes  │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01                         │ DRBD,STORAGE │ Unused │ Ok    │ SyncTarget(1.00%) │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02                         │ DRBD,STORAGE │ InUse  │ Ok    │          UpToDate │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05                         │ DRBD,STORAGE │ Unused │ Ok    │ SyncTarget(0.96%) │ Yes  │

Got a response

Yes, sorry, this was not properly pointed out in the release message. This comes from using https://github.com/piraeusdatastore/linstor-csi/releases/tag/v1.13.0

So you can still use the new operator, but you have to downgrade the LINSTOR CSI version until you upgrade the cluster.

So doing that for now with the following helm values

---
operator: 
  resources:
    requests:
      cpu: 250m
      memory: 500Mi
    limits:
      memory: 1Gi
installCRDs: true
imageConfigOverride:
- base: quay.io/piraeusdatastore
  components:
    linstor-satellite:
      image: piraeus-server
      tag: v1.33.1
    linstor-csi:
      image: piraeus-csi
      tag: v1.12.0
tls:
  certManagerIssuerRef:
    name: pp-step-issuer
    kind: StepClusterIssuer
    group: certmanager.step.sm

Created issue on their github, cause the wording makes it seem like only latest is supported now, which was not mentioned in their release notes

https://github.com/piraeusdatastore/piraeus-operator/issues/1066

Hello! Got this error when upgrade piraeus operator from 2.11.0 to 2.12.0

Curious to know when a new version of linstor will be supported?

time="2026-09-23T17:46:01Z" level=fatal msg="LINSTOR 1.33.1 (REST API 1.27.0) is not supported: the driver requires REST API 1.29 or newer in the 1.x series, shipped with LINSTOR 1.35"

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 controller v
linstor controller 1.33.1; GIT-hash: 95da7940d6efb6a39ea303c5f37b03478a6fab0b

hmm

primary, xen02

[13:30 ovbh-pprod-xen02 ~]# drbdsetup status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 node-id:1 role:Primary suspended:no force-io-failures:no
    write-ordering:flush
  volume:0 minor:1013 disk:UpToDate backing_dev:/dev/linstor_group/xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1_00000 quorum:yes open:yes
      size:315195168 read:233813750 written:1173974506 al-writes:758189 bm-writes:361 upper-pending:0 lower-pending:0 al-suspended:no blocked:no
  ovbh-pprod-xen01 node-id:0 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0 rs-in-flight:0
    volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
        received:0 sent:1173974894 out-of-sync:0 pending:1 unacked:0
  ovbh-pprod-xen05 node-id:2 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0 rs-in-flight:0
    volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
        received:0 sent:1146927724 out-of-sync:0 pending:1 unacked:0

xen01

[13:31 ovbh-pprod-xen01 ~]# drbdsetup status xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 --verbose --statistics
xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 node-id:0 role:Secondary suspended:no force-io-failures:no
    write-ordering:flush
  volume:0 minor:1013 disk:UpToDate backing_dev:/dev/linstor_group/xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1_00000 quorum:yes open:no
      size:315195168 read:1242 written:1181073258 al-writes:819172 bm-writes:349703 upper-pending:0 lower-pending:0 al-suspended:no blocked:no
  ovbh-pprod-xen02 node-id:1 connection:Connected role:Primary tls:no congested:no ap-in-flight:0 rs-in-flight:0
    volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
        received:1174048714 sent:0 out-of-sync:0 pending:0 unacked:0
  ovbh-pprod-xen05 node-id:2 connection:Connected role:Secondary tls:no congested:no ap-in-flight:0 rs-in-flight:0
    volume:0 replication:Established peer-disk:UpToDate resync-suspended:no
        received:0 sent:0 out-of-sync:111291800 pending:0 unacked:0

Now for my question that I would love input on. This DRBD dashboard states that there are some things out of sync.

Which is fed from this metric

However linstor says differently.

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 r l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01                         │ DRBD,STORAGE │ Unused │ Ok    │ UpToDate │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02                         │ DRBD,STORAGE │ InUse  │ Ok    │ UpToDate │ Yes  │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05                         │ DRBD,STORAGE │ Unused │ Ok    │ UpToDate │ Yes  │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 v l | grep xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen01                         │ xcp-sr-linstor_group_thin_device │     0 │    1013 │ /dev/drbd1013 │ 300.26 GiB │ Unused │ UpToDate │ Established(2) │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen02                         │ xcp-sr-linstor_group_thin_device │     0 │    1013 │ /dev/drbd1013 │ 300.26 GiB │ InUse  │ UpToDate │ Established(2) │
│ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1 │ ovbh-pprod-xen05                         │ xcp-sr-linstor_group_thin_device │     0 │    1013 │ /dev/drbd1013 │ 300.26 GiB │ Unused │ UpToDate │ Established(2) │

What is this pv? K8s worker vdi. Cool

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv s xcp-sr-linstor_group_thin_device | grep d07a41d6-0aa2-49a5-a241-b755c6daa3d8
│ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/metadata    │ {"name_label": "OVBH-VPROD-K8S01-WORKER02", "name_description": "", "is_a_snapshot": false, "snapshot_of": "", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}                                         │
│ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/not-exists  │ 0                                                                                                                                                                                                                                                          │
│ xcp/volume/d07a41d6-0aa2-49a5-a241-b755c6daa3d8/volume-name │ xcp-volume-6a8544e8-f17f-49c8-b725-54c5e3213ae1                                                                                                                                                                                                            │

I notice it only states that some xcp-volumes-* are out of sync, and none of the pvs created by piraeus operator.

Curious to know if anyone has any thoughts on this?

Hello! I set this up and thought I would share, as it gave me a question I wanted to ask.

First the setup.
For each of our xcp-ng hosts (5 for this cluster) I setup the following.

cat <<EOF > /etc/drbd-reactor.d/prometheus.toml
[[prometheus]]
enums = true
address = "10.2.0.1[0-4]:9942"
EOF

systemctl restart drbd-reactor

nano /etc/sysconfig/iptables 

#add 
-A RH-Firewall-1-INPUT -s 10.1.8.0/22 -p tcp -m conntrack --ctstate NEW -m tcp --dport 9942 -j ACCEPT
-A RH-Firewall-1-INPUT -s 10.1.8.0/22 -p tcp -m conntrack --ctstate NEW -m tcp --dport 9100 -j ACCEPT

systemctl restart iptables



wget https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz
tar xvfz node_exporter-*.linux-amd64.tar.gz
mv node_exporter-*.linux-amd64/node_exporter /usr/local/bin/


cat <<EOF > /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=root
ExecStart=/usr/local/bin/node_exporter --web.listen-address=10.2.0.1[0-4]:9100

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now node_exporter

Then because I wanted to reuse my existing setup, I deployed the following in k8s in a new namespace for organization.

---
apiVersion: v1
kind: Endpoints
metadata:
  name: xcp-ng-metrics
  namespace: xcp-ng-monitoring
subsets:
  - addresses:
      - ip: 10.2.0.10
      - ip: 10.2.0.11
      - ip: 10.2.0.12
      - ip: 10.2.0.13
      - ip: 10.2.0.14
    ports:
      - name: drbd
        port: 9942
        protocol: TCP
      - name: node
        port: 9100
        protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: xcp-ng-metrics
  namespace: xcp-ng-monitoring
  labels:
    app: xcp-ng-metrics
spec:
  clusterIP: None
  ports:
    - name: drbd
      port: 9942
      targetPort: 9942
    - name: node
      port: 9100
      targetPort: 9100
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: xcp-ng
  namespace: xcp-ng-monitoring
spec:
  selector:
    matchLabels:
      app: xcp-ng-metrics
  endpoints:
    - port: drbd
      interval: 30s
    - port: node
      interval: 30s

Then I went and nabbed these two dashboards linked in the GUI

Node-exporter is as expected

I have already built out the DRBD reactor dashboard, cause I want to spec out some prometheus rules.

Possible prometheus rules for alerting, gotten by taking https://github.com/LINBIT/drbd-reactor/blob/master/doc/prometheus.md and feeding it into an llm. Then going over each one and created a panel for it and tinkered to ensure it functioned as desired. A few generated were nonsense but that is how it goes lol.

---
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: xcp-ng-metrics-rules
  namespace: xcp-ng-monitoring
spec:
  groups:
  - name: drbd_process_health
    rules:
    - alert: DRBDReactorDaemonDown
      expr: drbdreactor_up == 0 or up{job="drbd-reactor"} == 0
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "DRBD Reactor telemetry agent is unreachable on {{ $labels.instance }}"
        description: "The DRBD Prometheus endpoint has failed. Metrics for this node are stale, blinding the cluster manager to storage faults."

    - alert: DRBDResourceCountAnomaly
      expr: changes(drbd_resource_resources[15m]) > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "DRBD resource count fluctuated on {{ $labels.instance }}"
        description: "The number of active DRBD resources has changed unexpectedly. Verify that no resources were erroneously deleted or unconfigured."


  - name: drbd_connection_integrity
    rules:
    - alert: DRBDConnectionSplitBrain
      expr: drbd_connection_state{drbd_connection_state="StandAlone"} == 1
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "DRBD Split-Brain detected on {{ $labels.instance }} (StandAlone State)"
        description: "The DRBD resource has entered the StandAlone state, refusing to reconnect. This signifies failed authentication, manual disconnection, or an active split-brain partition."

    - alert: DRBDNetworkFailure
      expr: drbd_connection_state{drbd_connection_state="NetworkFailure"} == 1 or drbd_connection_state{drbd_connection_state="BrokenPipe"} == 1
      for: 2m
      labels:
        severity: critical
      annotations:
        summary: "DRBD peer connection lost on {{ $labels.instance }}"
        description: "The TCP/IP connection to the DRBD peer has been severed. Replication is currently halted."

    - alert: DRBDNetworkCongestion
      expr: drbd_connection_congested == 1
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "TCP Send Buffer saturated on {{ $labels.instance }}"
        description: "The DRBD connection's TCP send buffer has remained over 80% full for 10 minutes, indicating systemic network bandwidth exhaustion."

    - alert: DRBDApplicationInFlightHigh
      expr: drbd_connection_apinflight_bytes > 536870912 # 500 MB
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Excessive unacknowledged application I/O on {{ $labels.instance }}"
        description: "Over 500MB of application data is in flight to the peer. The peer storage or network link is severely bottlenecking synchronous replication."


  - name: drbd_device_diagnostics
    rules:
    - alert: DRBDDisklessCatastrophe
      expr: drbd_device_unintentionaldiskless == 1 and drbd_device_client == 0
      for: 30s
      labels:
        severity: critical
      annotations:
        summary: "DRBD physical backing device failed on {{ $labels.instance }}"
        description: "The underlying storage device has suffered an I/O fault and detached. The node is operating in a failed diskless state."

    - alert: DRBDDeviceStateDegraded
      expr: drbd_device_state{drbd_device_state="Failed"} == 1 or drbd_device_state{drbd_device_state="Inconsistent"} == 1
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "DRBD local disk state is degraded on {{ $labels.instance }}"
        description: "The local block device is marked as {{ $labels.drbd_device_state }}. Data redundancy is compromised."

    - alert: DRBDActivityLogFrozen
      expr: drbd_device_alsuspended == 1
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "DRBD metadata Activity Log suspended on {{ $labels.instance }}"
        description: "The metadata sector is unwritable, suspending the Activity Log. All application writes are blocked to prevent corruption."

    - alert: DRBDLocalDiskLatency
      expr: drbd_device_lowerpending > 1000 and drbd_device_upperpending > 1000
      for: 3m
      labels:
        severity: warning
      annotations:
        summary: "Severe local disk latency on {{ $labels.instance }}"
        description: "Over 1000 requests are queued from DRBD to the local block layer, causing upstream application I/O to stall."

  - name: drbd_cluster_consensus
    rules:
    - alert: DRBDQuorumLost
      expr: drbd_device_quorum == 0
      for: 15s
      labels:
        severity: critical
      annotations:
        summary: "DRBD Quorum lost for resource on {{ $labels.instance }}"
        description: "The node has lost contact with the cluster majority. To prevent split-brain data corruption, the volume has likely suspended I/O."

    - alert: DRBDResourceSuspended
      expr: drbd_resource_suspended == 1
      for: 1m
      labels:
        severity: critical
      annotations:
        summary: "DRBD Resource is fully suspended on {{ $labels.instance }}"
        description: "All read and write system calls are blocked. Applications utilizing this block device will hang in D-state."

    - alert: DRBDStuckSynchronization
      expr: drbd_peerdevice_outofsync_bytes > 0 and rate(drbd_connection_rsinflight_bytes[5m]) == 0
      for: 15m
      labels:
        severity: critical
      annotations:
        summary: "DRBD background synchronization is stalled on {{ $labels.instance }}"
        description: "Data divergence exists between peers ({{ $value }} bytes), but no resynchronization traffic has been detected for 5 minutes."

After getting back decided to run it.

[16:34 ovbh-pprod-xen01 ~]# python3 snapshot-fixer.py rewrite
INFO:root:Check HA...
INFO:root:Shutting down xapi...
INFO:root:Regenerating database...
INFO:root:The VM b532b6e3-df2a-42e2-ade1-ac424c782938 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9707615f-334e-43bf-9946-95205bf8869f has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 1555cf78-7546-2b86-771d-ee66a306faf3 has Ref:941 as its "snapshot_of" value, changing to null.
INFO:root:The VM 99532d6b-a2e3-4133-8737-c416e8ef5bda has Ref:14 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9b7d927c-b70d-27f3-5d08-100d0744780f has Ref:50 as its "snapshot_of" value, changing to null.
INFO:root:The VM a04e5d4a-1067-4f91-89d7-8b5addb205f2 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 2b712337-b1e6-1b0f-d39e-5f3803bd8643 has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5b5c6ae6-2c9c-d76f-d9ae-7eff962287f0 has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM d155c631-b18f-c0d2-b425-49d53ed065fb has Ref:90 as its "snapshot_of" value, changing to null.
INFO:root:The VM 8c0a4696-4ea7-34b2-ca14-0252b4d6e357 has Ref:243 as its "snapshot_of" value, changing to null.
INFO:root:The VM c23a131e-e724-7f02-adba-e199526b401d has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5a2311cd-bbb7-6444-0119-9bb8c8a9e593 has Ref:265 as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5a6c334e-040e-4825-9ced-55ee2b17aac6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI f887e9da-cda5-496e-9040-57aab22e510a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a3406a09-928a-4bf1-b558-53ab744dc41b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ea9d191b-0a2d-49d5-a046-7ebe483993fa has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI eca38833-219e-439a-91ac-d3321772f71a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI dc740184-28b4-4db5-938b-837234f08290 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1ee50d61-1426-4322-892b-e270cfd4c97e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7bc9add5-cf5d-49c6-8022-78fb386a1e3b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 91f43c21-07ba-426f-a146-2d678e7de83b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a8dd3a48-8035-4a49-90f6-d91acdbbb032 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7f128b35-5e6f-4e32-b0c8-15059878c10c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5235fea0-4f20-4f19-859a-044b99ebce3c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 52e7b2de-84cb-4e1b-9a1f-4b898399a650 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2418d9e7-c3a7-424a-af36-fd983d438163 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7fc59e4-8d90-4157-b526-bc5ecc549b66 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI fd507e6c-c3e2-4305-81c8-f92da2703db9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 70081652-9efd-4881-85b6-c39b0de6f90b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5bf279c2-cff6-4eb6-8e70-376467d649f6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b37f9db8-b2bc-42d1-88b5-10207f9aeeb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 25937e64-4344-4772-8d8b-b5e9fee7cd81 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd0511c9-b803-48f2-afae-35fa51225c9e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7063926-088b-4cf0-8004-d75ea15653c5 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aff3122f-1693-4371-8d6f-5a2975aec85d has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 17200bad-cc3e-4b5d-9844-d1e56604d2a3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 99ad1143-0b88-46f0-ac0e-e5a72b69f327 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55c5e0a1-0af7-4b36-b5a6-57cff1185208 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2c5dec71-cd9a-4208-9e96-922b44544d09 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 27ccf472-3fa9-4d23-bdb1-6ca70f570230 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI e95206e0-4940-48fc-8564-238f2f0f6440 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5544ccc6-dec4-4528-91b4-302a6552146b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d1f154d5-9bdf-4c01-897c-367863efe114 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2476d6b3-1b25-47ca-bc8d-869125eed779 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a4fc211d-9e68-4f7c-99e5-170f3786ad55 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2ce4926e-c8dc-4185-bbb4-ce887ef78b42 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09ee300e-ac03-48f8-a6cc-bc33398fd75f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 78a67732-4889-4053-8a36-fb5d3cf1c0d3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ad9b1572-a29d-42bb-842d-448e49bdbbb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 60ad964b-b5aa-4077-9ca0-6acb84cbea15 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6cfe58b0-3dd7-422d-924c-284d811afa97 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bde8b13b-2f8c-454a-8bae-6f07fa907b9f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a9c40305-9f7c-4af9-be56-b1dd4f26458c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bc31eab4-7720-45fb-b6e8-23d529381221 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6a1ff504-f448-420a-ba4b-8a05d8a3f2da has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cbaf2499-489f-4b48-a017-e48fb73b4987 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bd2b4490-3464-4650-8847-11a376393ae9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 8874451d-249f-4913-ad9c-fb8da214f663 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a05390d4-5ef7-458d-b090-799215c49c6e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1e949f7b-3f04-4fcc-bdc9-2052974aef34 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 156877f1-0701-484e-9c60-659841110d38 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ab4c71e6-ac41-4fc6-8435-546ab827c728 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bb36c707-d6a9-4811-b3dc-33c694b2faf8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5df27f53-4a98-41ba-b577-7468e3eb9557 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd158763-7f28-42ca-bddb-30ffcc89e18f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b741152c-ce78-4d00-8227-2031ba3778bc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 450bddc9-946f-43c1-b9a3-5f50678ac249 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d07a41d6-0aa2-49a5-a241-b755c6daa3d8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 415196a5-de15-4a2a-912a-675f804af98e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 31ecf300-cc63-4845-8e7c-d191f4042135 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 357c608a-a805-48ab-8a96-d93fe1157304 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5111fe82-17b7-467c-851c-82ccb18d7bb1 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 653f53d3-1e8b-41e7-a997-41025bc268ef has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 832e9bfd-899a-41e3-baa6-373175b4b0ec has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55300191-34ec-4252-aa9f-3b2bdd5f98c7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d55536f8-906e-419e-a17a-318095312563 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7884662d-967e-43b1-a789-e92c1c149403 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09d26ac2-82a5-4208-8ebc-9d0df97665fc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 848ff799-092f-428b-90de-3a85517c7315 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:Writing database to /var/lib/xcp/state.db
INFO:root:Starting up xapi...

Everything is looking good!

Dry-run output

[14:09 ovbh-pprod-xen03 ~]# python3 snapshot-fixer.py dry-run
INFO:root:Regenerating database...
INFO:root:The VM b532b6e3-df2a-42e2-ade1-ac424c782938 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9707615f-334e-43bf-9946-95205bf8869f has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 1555cf78-7546-2b86-771d-ee66a306faf3 has Ref:941 as its "snapshot_of" value, changing to null.
INFO:root:The VM 99532d6b-a2e3-4133-8737-c416e8ef5bda has Ref:14 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9b7d927c-b70d-27f3-5d08-100d0744780f has Ref:50 as its "snapshot_of" value, changing to null.
INFO:root:The VM a04e5d4a-1067-4f91-89d7-8b5addb205f2 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 2b712337-b1e6-1b0f-d39e-5f3803bd8643 has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5b5c6ae6-2c9c-d76f-d9ae-7eff962287f0 has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM d155c631-b18f-c0d2-b425-49d53ed065fb has Ref:90 as its "snapshot_of" value, changing to null.
INFO:root:The VM 8c0a4696-4ea7-34b2-ca14-0252b4d6e357 has Ref:243 as its "snapshot_of" value, changing to null.
INFO:root:The VM c23a131e-e724-7f02-adba-e199526b401d has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5a2311cd-bbb7-6444-0119-9bb8c8a9e593 has Ref:265 as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5a6c334e-040e-4825-9ced-55ee2b17aac6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI f887e9da-cda5-496e-9040-57aab22e510a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a3406a09-928a-4bf1-b558-53ab744dc41b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ea9d191b-0a2d-49d5-a046-7ebe483993fa has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI eca38833-219e-439a-91ac-d3321772f71a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI dc740184-28b4-4db5-938b-837234f08290 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1ee50d61-1426-4322-892b-e270cfd4c97e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7bc9add5-cf5d-49c6-8022-78fb386a1e3b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 91f43c21-07ba-426f-a146-2d678e7de83b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a8dd3a48-8035-4a49-90f6-d91acdbbb032 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7f128b35-5e6f-4e32-b0c8-15059878c10c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5235fea0-4f20-4f19-859a-044b99ebce3c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 52e7b2de-84cb-4e1b-9a1f-4b898399a650 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2418d9e7-c3a7-424a-af36-fd983d438163 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7fc59e4-8d90-4157-b526-bc5ecc549b66 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI fd507e6c-c3e2-4305-81c8-f92da2703db9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 70081652-9efd-4881-85b6-c39b0de6f90b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5bf279c2-cff6-4eb6-8e70-376467d649f6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b37f9db8-b2bc-42d1-88b5-10207f9aeeb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 25937e64-4344-4772-8d8b-b5e9fee7cd81 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd0511c9-b803-48f2-afae-35fa51225c9e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7063926-088b-4cf0-8004-d75ea15653c5 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aff3122f-1693-4371-8d6f-5a2975aec85d has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 17200bad-cc3e-4b5d-9844-d1e56604d2a3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 99ad1143-0b88-46f0-ac0e-e5a72b69f327 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55c5e0a1-0af7-4b36-b5a6-57cff1185208 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2c5dec71-cd9a-4208-9e96-922b44544d09 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 27ccf472-3fa9-4d23-bdb1-6ca70f570230 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI e95206e0-4940-48fc-8564-238f2f0f6440 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5544ccc6-dec4-4528-91b4-302a6552146b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d1f154d5-9bdf-4c01-897c-367863efe114 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2476d6b3-1b25-47ca-bc8d-869125eed779 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a4fc211d-9e68-4f7c-99e5-170f3786ad55 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2ce4926e-c8dc-4185-bbb4-ce887ef78b42 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09ee300e-ac03-48f8-a6cc-bc33398fd75f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 78a67732-4889-4053-8a36-fb5d3cf1c0d3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ad9b1572-a29d-42bb-842d-448e49bdbbb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 60ad964b-b5aa-4077-9ca0-6acb84cbea15 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6cfe58b0-3dd7-422d-924c-284d811afa97 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bde8b13b-2f8c-454a-8bae-6f07fa907b9f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a9c40305-9f7c-4af9-be56-b1dd4f26458c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bc31eab4-7720-45fb-b6e8-23d529381221 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6a1ff504-f448-420a-ba4b-8a05d8a3f2da has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cbaf2499-489f-4b48-a017-e48fb73b4987 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bd2b4490-3464-4650-8847-11a376393ae9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 8874451d-249f-4913-ad9c-fb8da214f663 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a05390d4-5ef7-458d-b090-799215c49c6e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1e949f7b-3f04-4fcc-bdc9-2052974aef34 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 156877f1-0701-484e-9c60-659841110d38 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ab4c71e6-ac41-4fc6-8435-546ab827c728 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bb36c707-d6a9-4811-b3dc-33c694b2faf8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5df27f53-4a98-41ba-b577-7468e3eb9557 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd158763-7f28-42ca-bddb-30ffcc89e18f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b741152c-ce78-4d00-8227-2031ba3778bc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 450bddc9-946f-43c1-b9a3-5f50678ac249 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d07a41d6-0aa2-49a5-a241-b755c6daa3d8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 415196a5-de15-4a2a-912a-675f804af98e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 31ecf300-cc63-4845-8e7c-d191f4042135 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 357c608a-a805-48ab-8a96-d93fe1157304 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5111fe82-17b7-467c-851c-82ccb18d7bb1 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 653f53d3-1e8b-41e7-a997-41025bc268ef has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 832e9bfd-899a-41e3-baa6-373175b4b0ec has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55300191-34ec-4252-aa9f-3b2bdd5f98c7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d55536f8-906e-419e-a17a-318095312563 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7884662d-967e-43b1-a789-e92c1c149403 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09d26ac2-82a5-4208-8ebc-9d0df97665fc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 848ff799-092f-428b-90de-3a85517c7315 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.

@poddingue Appreciate it! I will check that out. Been digging to see wth is going on.

Hoped on xen03 to see what it had to say

[13:47 ovbh-pprod-xen03 ~]# ls -l /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0
lrwxrwxrwx 1 root root 17 Jul 30 13:15 /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0 -> ../../../drbd1227
[13:47 ovbh-pprod-xen03 ~]# vhd-util query -n /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0 -p
/dev/drbd/by-res/xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0
[13:47 ovbh-pprod-xen03 ~]# vhd-util read -p -n /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0
VHD Footer Summary:
-------------------
Cookie              : conectix
Features            : (0x00000002) <RESV>
File format version : Major: 1, Minor: 0
Data offset         : 512
Timestamp           : Sun Jul 26 03:46:10 2026
Creator Application : 'tap'
Creator version     : Major: 1, Minor: 3
Creator OS          : Unknown!
Original disk size  : 102400 MB (107374182400 Bytes)
Current disk size   : 102400 MB (107374182400 Bytes)
Geometry            : Cyl: 51400, Hds: 16, Sctrs: 255
                    : = 102398 MB (107372544000 Bytes)
Disk type           : Differencing hard disk
Checksum            : 0xffffee7e|0xffffee7e (Good!)
UUID                : c55dc0bf-77b9-431d-b16a-891509da7e54
Saved state         : No
Hidden              : 0

VHD Header Summary:
-------------------
Cookie              : cxsparse
Data offset (unusd) : 18446744073709551615
Table offset        : 1536
Header version      : 0x00010000
Max BAT size        : 1048576
Block size          : 2097152 (2 MB)
Parent name         : 0
Parent UUID         : efc16e1a-8384-444a-ac04-00f88c852cb6
Parent timestamp    : Thu Jul 30 13:14:52 2026
Checksum            : 0xffffe512|0xffffe512 (Good!)

VHD Parent Locators:
--------------------
locator:            : 0
       code         : PLAT_CODE_MACX
       data_space   : 512
       data_length  : 59
       data_offset  : 4327424
       decoded name : ../xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0

locator:            : 1
       code         : PLAT_CODE_W2KU
       data_space   : 512
       data_length  : 108
       data_offset  : 4327936
       decoded name : ./../xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0

locator:            : 2
       code         : PLAT_CODE_W2RU
       data_space   : 512
       data_length  : 108
       data_offset  : 4328448
       decoded name : ./../xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0

VHD Batmap Summary:
-------------------
Batmap offset       : 4196352
Batmap size (secs)  : 256
Batmap version      : 0x00010002
Checksum            : 0xfffd78cf|0xfffd78cf (Good!)

Which points to a base copy

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/volume-name │ xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d                                                                                                                                                                                                            │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep c7fc59e4-8d90-4157-b526-bc5ecc549b66
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/metadata    │ {"name_label": "base copy", "name_description": "", "is_a_snapshot": false, "snapshot_of": "", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": true, "metadata_of_pool": ""}                                                          │
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/not-exists  │ 0                                                                                                                                                                                                                                                          │
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/volume-name │ xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d                                                                                                                                                                                                            │

[13:47 ovbh-pprod-xen03 ~]# xe vdi-param-list uuid=c7fc59e4-8d90-4157-b526-bc5ecc549b66
uuid ( RO)                    : c7fc59e4-8d90-4157-b526-bc5ecc549b66
              name-label ( RW): base copy
        name-description ( RW): 
           is-a-snapshot ( RO): false
             snapshot-of ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
               snapshots ( RO): 
           snapshot-time ( RO): 19700101T00:00:00Z
      allowed-operations (SRO): copy; generate_config; clone; forget; update; resize; snapshot; destroy
      current-operations (SRO): 
                 sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
           sr-name-label ( RO): linstor-sr
               vbd-uuids (SRO): 
         crashdump-uuids (SRO): 
            virtual-size ( RO): 107374182400
    physical-utilisation ( RO): 106782908416
                location ( RO): c7fc59e4-8d90-4157-b526-bc5ecc549b66
                    type ( RO): User
                sharable ( RO): false
               read-only ( RO): true
            storage-lock ( RO): false
                 managed ( RO): false
     parent ( RO) [DEPRECATED]: <not in database>
                 missing ( RO): false
            is-tools-iso ( RO): false
            other-config (MRW): 
           xenstore-data (MRO): 
               sm-config (MRO): vhd-blocks: eJzt2T0KwCAMgNFAh44eu9fyKIVeQ/yJixVBLBQC8r0hyWKSIZspTQUNtzTvWs2fAwAAAAC2dlkvAPzptF4A+4mH3ezHbjQwcBq8SM2lcF8bcNBYE7r/iwziv7hc; image-format: vhd
                 on-boot ( RW): persist
           allow-caching ( RW): false
         metadata-latest ( RO): false
        metadata-of-pool ( RO): <not in database>
                    tags (SRW): 
             cbt-enabled ( RO): false

which also points back to 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0

Thought this was weird. Something happened and now most of the VDIs in linstor show up as snapshots now.
Looking at just these two, to compare.

LB VDI: 150cd8e3-633d-42af-91b2-faa0d1f7b658

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep 150cd8e3-633d-42af-91b2-faa0d1f7b658
│ xcp/volume/150cd8e3-633d-42af-91b2-faa0d1f7b658/metadata     │ {"name_label": "OVBH-VPROD-RANCHERLB", "name_description": "", "is_a_snapshot": false, "snapshot_of": null, "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}                                            │
│ xcp/volume/150cd8e3-633d-42af-91b2-faa0d1f7b658/not-exists   │ 0                                                                                                                                                                                                                                                          │
│ xcp/volume/150cd8e3-633d-42af-91b2-faa0d1f7b658/volume-name  │ xcp-volume-7fea7c97-3783-46b6-bcc6-cfd234b1f4a7                                                                                                                                                                                                            │
│ xcp/volume/ebdd16b6-5293-4c7d-ac37-b427fc3b368d/metadata     │ {"name_label": "OVBH-VPROD-RANCHERLB", "name_description": "", "is_a_snapshot": true, "snapshot_of": "150cd8e3-633d-42af-91b2-faa0d1f7b658", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}           │

[13:12 ovbh-pprod-xen01 ~]# xe vdi-param-list uuid=150cd8e3-633d-42af-91b2-faa0d1f7b658
uuid ( RO)                    : 150cd8e3-633d-42af-91b2-faa0d1f7b658
              name-label ( RW): OVBH-VPROD-RANCHERLB
        name-description ( RW): 
           is-a-snapshot ( RO): false
             snapshot-of ( RO): <not in database>
               snapshots ( RO): ebdd16b6-5293-4c7d-ac37-b427fc3b368d
           snapshot-time ( RO): 19700101T00:00:00Z
      allowed-operations (SRO): clone; snapshot
      current-operations (SRO): 
                 sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
           sr-name-label ( RO): linstor-sr
               vbd-uuids (SRO): 07b60986-8d19-aa23-f2ba-8d8ab96f93b4
         crashdump-uuids (SRO): 
            virtual-size ( RO): 53687091200
    physical-utilisation ( RO): 1108372480
                location ( RO): 150cd8e3-633d-42af-91b2-faa0d1f7b658
                    type ( RO): User
                sharable ( RO): false
               read-only ( RO): false
            storage-lock ( RO): false
                 managed ( RO): true
     parent ( RO) [DEPRECATED]: <not in database>
                 missing ( RO): false
            is-tools-iso ( RO): false
            other-config (MRW): 
           xenstore-data (MRO): 
               sm-config (MRO): host_OpaqueRef:b3c659f5-efb3-4202-9268-b824e3eed523: RW; read-caching-reason-85838113-e4b8-4520-9f6d-8f3cf554c8f1: SR_NOT_SUPPORTED; read-caching-enabled-on-85838113-e4b8-4520-9f6d-8f3cf554c8f1: false; image-format: vhd; vdi_type: vhd; vhd-parent: 28d1e0e1-29e6-4734-90cd-5e394fe25af8
                 on-boot ( RW): persist
           allow-caching ( RW): false
         metadata-latest ( RO): false
        metadata-of-pool ( RO): <not in database>
                    tags (SRW): 
             cbt-enabled ( RO): false

03 VDI: 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
│ xcp/volume/07f430b2-c7f1-41d8-bd9f-2fe67f4f458b/metadata     │ {"name_label": "OVBH-VPROD-RANCHER03", "name_description": "", "is_a_snapshot": false, "snapshot_of": null, "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}                                            │
│ xcp/volume/07f430b2-c7f1-41d8-bd9f-2fe67f4f458b/not-exists   │ 0                                                                                                                                                                                                                                                          │
│ xcp/volume/07f430b2-c7f1-41d8-bd9f-2fe67f4f458b/volume-name  │ xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5                                                                                                                                                                                                            │
│ xcp/volume/098cc02d-0d62-4f36-b86e-e85e4a2802e0/metadata     │ {"name_label": "OVBH-VPROD-RANCHER03", "name_description": "", "is_a_snapshot": true, "snapshot_of": "07f430b2-c7f1-41d8-bd9f-2fe67f4f458b", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}           │

[13:10 ovbh-pprod-xen01 ~]# xe vdi-param-list uuid=07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
uuid ( RO)                    : 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
              name-label ( RW): OVBH-VPROD-RANCHER03
        name-description ( RW): 
           is-a-snapshot ( RO): false
             snapshot-of ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
               snapshots ( RO): 098cc02d-0d62-4f36-b86e-e85e4a2802e0
           snapshot-time ( RO): 19700101T00:00:00Z
      allowed-operations (SRO): clone; snapshot
      current-operations (SRO): 
                 sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
           sr-name-label ( RO): linstor-sr
               vbd-uuids (SRO): bdfc6a23-a89b-eb7e-7509-97e910e7ca79
         crashdump-uuids (SRO): 
            virtual-size ( RO): 107374182400
    physical-utilisation ( RO): 24027230208
                location ( RO): 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
                    type ( RO): User
                sharable ( RO): false
               read-only ( RO): false
            storage-lock ( RO): false
                 managed ( RO): true
     parent ( RO) [DEPRECATED]: <not in database>
                 missing ( RO): false
            is-tools-iso ( RO): false
            other-config (MRW): 
           xenstore-data (MRO): 
               sm-config (MRO): host_OpaqueRef:7f23da94-1933-41ad-8d9a-27dac34bc1a9: RW; read-caching-reason-c03e1a45-4c4c-46f5-a2a1-d8de2e22a866: SR_NOT_SUPPORTED; read-caching-enabled-on-c03e1a45-4c4c-46f5-a2a1-d8de2e22a866: false; vhd-parent: 59c33f05-2601-46dd-af09-9e86b3e166e3; image-format: vhd; vdi_type: vhd
                 on-boot ( RW): persist
           allow-caching ( RW): false
         metadata-latest ( RO): false
        metadata-of-pool ( RO): <not in database>
                    tags (SRW): 
             cbt-enabled ( RO): false

HUH?

[13:14 ovbh-pprod-xen01 ~]# xe vdi-param-list uuid=1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
uuid ( RO)                    : 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
              name-label ( RW): OVBH-VPROD-RANCHER01
        name-description ( RW): 
           is-a-snapshot ( RO): false
             snapshot-of ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
               snapshots ( RO): 848ff799-092f-428b-90de-3a85517c7315; 09d26ac2-82a5-4208-8ebc-9d0df97665fc; 7884662d-967e-43b1-a789-e92c1c149403; d55536f8-906e-419e-a17a-318095312563; 55300191-34ec-4252-aa9f-3b2bdd5f98c7; 832e9bfd-899a-41e3-baa6-373175b4b0ec; 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67; aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775; 653f53d3-1e8b-41e7-a997-41025bc268ef; 5111fe82-17b7-467c-851c-82ccb18d7bb1; 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0; 357c608a-a805-48ab-8a96-d93fe1157304; 31ecf300-cc63-4845-8e7c-d191f4042135; 415196a5-de15-4a2a-912a-675f804af98e; d07a41d6-0aa2-49a5-a241-b755c6daa3d8; 450bddc9-946f-43c1-b9a3-5f50678ac249; c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae; b741152c-ce78-4d00-8227-2031ba3778bc; cd158763-7f28-42ca-bddb-30ffcc89e18f; 5df27f53-4a98-41ba-b577-7468e3eb9557; bb36c707-d6a9-4811-b3dc-33c694b2faf8; ab4c71e6-ac41-4fc6-8435-546ab827c728; 156877f1-0701-484e-9c60-659841110d38; 1e949f7b-3f04-4fcc-bdc9-2052974aef34; a05390d4-5ef7-458d-b090-799215c49c6e; 8874451d-249f-4913-ad9c-fb8da214f663; 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6; bd2b4490-3464-4650-8847-11a376393ae9; cbaf2499-489f-4b48-a017-e48fb73b4987; 6a1ff504-f448-420a-ba4b-8a05d8a3f2da; bc31eab4-7720-45fb-b6e8-23d529381221; 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271; a9c40305-9f7c-4af9-be56-b1dd4f26458c; bde8b13b-2f8c-454a-8bae-6f07fa907b9f; 6cfe58b0-3dd7-422d-924c-284d811afa97; 60ad964b-b5aa-4077-9ca0-6acb84cbea15; 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7; c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4; ad9b1572-a29d-42bb-842d-448e49bdbbb2; 78a67732-4889-4053-8a36-fb5d3cf1c0d3; 6ac57186-d928-4f34-8ce8-ae5f8b916107; 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0; 09ee300e-ac03-48f8-a6cc-bc33398fd75f; 2ce4926e-c8dc-4185-bbb4-ce887ef78b42; a4fc211d-9e68-4f7c-99e5-170f3786ad55; 2476d6b3-1b25-47ca-bc8d-869125eed779; d1f154d5-9bdf-4c01-897c-367863efe114; 5544ccc6-dec4-4528-91b4-302a6552146b; e95206e0-4940-48fc-8564-238f2f0f6440; 27ccf472-3fa9-4d23-bdb1-6ca70f570230; 2c5dec71-cd9a-4208-9e96-922b44544d09; 55c5e0a1-0af7-4b36-b5a6-57cff1185208; 99ad1143-0b88-46f0-ac0e-e5a72b69f327; 17200bad-cc3e-4b5d-9844-d1e56604d2a3; aff3122f-1693-4371-8d6f-5a2975aec85d; c7063926-088b-4cf0-8004-d75ea15653c5; cd0511c9-b803-48f2-afae-35fa51225c9e; 25937e64-4344-4772-8d8b-b5e9fee7cd81; 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b; b37f9db8-b2bc-42d1-88b5-10207f9aeeb2; 5bf279c2-cff6-4eb6-8e70-376467d649f6; 70081652-9efd-4881-85b6-c39b0de6f90b; fd507e6c-c3e2-4305-81c8-f92da2703db9; c7fc59e4-8d90-4157-b526-bc5ecc549b66; 2418d9e7-c3a7-424a-af36-fd983d438163; 52e7b2de-84cb-4e1b-9a1f-4b898399a650; 5235fea0-4f20-4f19-859a-044b99ebce3c; 7f128b35-5e6f-4e32-b0c8-15059878c10c; a8dd3a48-8035-4a49-90f6-d91acdbbb032; 91f43c21-07ba-426f-a146-2d678e7de83b; 7bc9add5-cf5d-49c6-8022-78fb386a1e3b; 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6; 1ee50d61-1426-4322-892b-e270cfd4c97e; dc740184-28b4-4db5-938b-837234f08290; eca38833-219e-439a-91ac-d3321772f71a; ea9d191b-0a2d-49d5-a046-7ebe483993fa; a3406a09-928a-4bf1-b558-53ab744dc41b; f887e9da-cda5-496e-9040-57aab22e510a; 5a6c334e-040e-4825-9ced-55ee2b17aac6
           snapshot-time ( RO): 20260716T19:53:31Z
      allowed-operations (SRO): copy; generate_config; clone; forget; update; resize; snapshot; destroy
      current-operations (SRO): 
                 sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
           sr-name-label ( RO): linstor-sr
               vbd-uuids (SRO): 
         crashdump-uuids (SRO): 
            virtual-size ( RO): 107374182400
    physical-utilisation ( RO): 21519360
                location ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
                    type ( RO): User
                sharable ( RO): false
               read-only ( RO): false
            storage-lock ( RO): false
                 managed ( RO): true
     parent ( RO) [DEPRECATED]: <not in database>
                 missing ( RO): false
            is-tools-iso ( RO): false
            other-config (MRW): 
           xenstore-data (MRO): 
               sm-config (MRO): vhd-parent: 848ff799-092f-428b-90de-3a85517c7315; image-format: vhd; vdi_type: vhd
                 on-boot ( RW): persist
           allow-caching ( RW): false
         metadata-latest ( RO): false
        metadata-of-pool ( RO): <not in database>
                    tags (SRW): 
             cbt-enabled ( RO): false

wth

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
│ xcp/volume/1516113c-9dbd-4e23-b76d-e1ec8cbed0c0/metadata    │ {"name_label": "OVBH-VPROD-RANCHER01", "name_description": "", "is_a_snapshot": true, "snapshot_of": "752611f2-bd1a-4009-9e6d-ce76b82144c4", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}           │
│ xcp/volume/1516113c-9dbd-4e23-b76d-e1ec8cbed0c0/not-exists  │ 0                                                                                                                                                                                                                                                          │
│ xcp/volume/1516113c-9dbd-4e23-b76d-e1ec8cbed0c0/volume-name │ xcp-volume-cc57261d-6309-4a88-a672-f5eed4b0bc8d                                                                                                                                                                                                            │

All hosts are restarted with 9.2.18 loaded now. Will see what happens next.

[15:56 ovbh-pprod-xen05 ~]# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090212
DRBD_KERNEL_VERSION=9.2.18
DRBDADM_VERSION_CODE=0x092100
DRBDADM_VERSION=9.33.0

New crash, this time xen02. The time is different because I updated the velero backup schedules to not all be in the middle of the night. Also still happened with the openvas schedule removed, which makes sense.
Xen02 now has 9.2.18 loaded. Will restart other hosts now.
I also uploaded logs if those are desired.

@MajorP93 My mistake. No I have not rebooted the hosts yet. https://xcp-ng.org/forum/post/107164 I am aware that the reboot is needed for it to be picked up, I went ahead and installed from test so that if host crashes again it is there.

@Mathieu-L Yes, the hosts had been restarted after the updates were installed which included 9.2.16

@Mathieu-L https://xcp-ng.org/forum/post/106873 I had increased it to 16G and it still crashed.

For now I have upgraded all hosts to use --enablerepo=xcp-ng-linstor-testing to get kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3
That way it is there to be picked up if the host crashes again.
https://github.com/xcp-ng-rpms/drbd/blob/master/SPECS/kmod-drbd.spec
https://koji.xcp-ng.org/buildinfo?buildID=5747

[12:15 ovbh-pprod-xen04 ~]# yum upgrade --enablerepo=xcp-ng-linstor-testing
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
grafana/signature                                                                                                                                                                                                                      |  810 B  00:00:00     
grafana/signature                                                                                                                                                                                                                      | 3.0 kB  00:00:00 !!! 
xcp-ng-linstor-testing/signature                                                                                                                                                                                                       |  473 B  00:00:00     
xcp-ng-linstor-testing/signature                                                                                                                                                                                                       | 3.0 kB  00:00:00 !!! 
zabbix                                                                                                                                                                                                                                 | 3.0 kB  00:00:00     
zabbix-non-supported                                                                                                                                                                                                                   | 2.9 kB  00:00:00     
(1/3): grafana/primary_db                                                                                                                                                                                                              | 879 kB  00:00:00     
(2/3): xcp-ng-linstor-testing/primary_db                                                                                                                                                                                               | 2.4 kB  00:00:00     
(3/3): zabbix/x86_64/primary_db                                                                                                                                                                                                        | 114 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package kmod-drbd.x86_64 0:9.2.16-1.0.xcpng8.3 will be updated
---> Package kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3 will be an update
---> Package zabbix-agent.x86_64 0:7.0.27-release1.el7 will be updated
---> Package zabbix-agent.x86_64 0:7.0.28-release1.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================
 Package                                                    Arch                                                 Version                                                           Repository                                                            Size
==============================================================================================================================================================================================================================================================
Updating:
 kmod-drbd                                                  x86_64                                               9.2.18-2.0.xcpng8.3                                               xcp-ng-linstor-testing                                               2.8 M
 zabbix-agent                                               x86_64                                               7.0.28-release1.el7                                               zabbix                                                               660 k

Transaction Summary
==============================================================================================================================================================================================================================================================
Upgrade  2 Packages

Total download size: 3.4 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/2): zabbix-agent-7.0.28-release1.el7.x86_64.rpm                                                                                                                                                                                     | 660 kB  00:00:00     
(2/2): kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64.rpm                                                                                                                                                                                        | 2.8 MB  00:00:01     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                         2.8 MB/s | 3.4 MB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64                                                                                                                                                                                                       1/4 
  Updating   : zabbix-agent-7.0.28-release1.el7.x86_64                                                                                                                                                                                                    2/4 
  Cleanup    : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64                                                                                                                                                                                                       3/4 
  Cleanup    : zabbix-agent-7.0.27-release1.el7.x86_64                                                                                                                                                                                                    4/4 
  Verifying  : zabbix-agent-7.0.28-release1.el7.x86_64                                                                                                                                                                                                    1/4 
  Verifying  : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64                                                                                                                                                                                                       2/4 
  Verifying  : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64                                                                                                                                                                                                       3/4 
  Verifying  : zabbix-agent-7.0.27-release1.el7.x86_64                                                                                                                                                                                                    4/4 

Updated:
  kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3                                                                                       zabbix-agent.x86_64 0:7.0.28-release1.el7                                                                                      

Complete!

@poddingue I'm fully up to date, but only at version 9.2.16

[14:57 ovbh-pprod-xen02 ~]# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090210
DRBD_KERNEL_VERSION=9.2.16
DRBDADM_VERSION_CODE=0x092100
DRBDADM_VERSION=9.33.0
[14:57 ovbh-pprod-xen02 ~]# drbd-reactor --version
drbd-reactor 1.9.0
[14:57 ovbh-pprod-xen02 ~]# 

There are no updates that I can install

[14:08 ovbh-pprod-xen02 ~]# yum check-update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
zabbix                                                                                                                                                                                                                                 | 3.0 kB  00:00:00     
zabbix-non-supported                                                                                                                                                                                                                   | 2.9 kB  00:00:00     
zabbix/x86_64/primary_db                                                                                                                                                                                                               | 114 kB  00:00:00     

zabbix-agent.x86_64                                                                                                         7.0.28-release1.el7                                                                                                         zabbix

@MajorP93 No we do not have pro support at the moment

hmmm. Really seems like a build with 9.2.19 could fix all of this.
How can I request for a build @poddingue ?

https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L61
9.2.17

• Fix a kernel crash triggered by a crafted/invalid netlink message

https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L24
9.2.19

• Fix an AB-BA deadlock between online resize and activity-log transactions
• Fix several races during connection teardown that could crash or
  hang (ack_sender requeue, pending ping work, in-progress lb-tcp
  connect)