After getting back decided to run it.

[16:34 ovbh-pprod-xen01 ~]# python3 snapshot-fixer.py rewrite
INFO:root:Check HA...
INFO:root:Shutting down xapi...
INFO:root:Regenerating database...
INFO:root:The VM b532b6e3-df2a-42e2-ade1-ac424c782938 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9707615f-334e-43bf-9946-95205bf8869f has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 1555cf78-7546-2b86-771d-ee66a306faf3 has Ref:941 as its "snapshot_of" value, changing to null.
INFO:root:The VM 99532d6b-a2e3-4133-8737-c416e8ef5bda has Ref:14 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9b7d927c-b70d-27f3-5d08-100d0744780f has Ref:50 as its "snapshot_of" value, changing to null.
INFO:root:The VM a04e5d4a-1067-4f91-89d7-8b5addb205f2 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 2b712337-b1e6-1b0f-d39e-5f3803bd8643 has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5b5c6ae6-2c9c-d76f-d9ae-7eff962287f0 has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM d155c631-b18f-c0d2-b425-49d53ed065fb has Ref:90 as its "snapshot_of" value, changing to null.
INFO:root:The VM 8c0a4696-4ea7-34b2-ca14-0252b4d6e357 has Ref:243 as its "snapshot_of" value, changing to null.
INFO:root:The VM c23a131e-e724-7f02-adba-e199526b401d has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5a2311cd-bbb7-6444-0119-9bb8c8a9e593 has Ref:265 as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5a6c334e-040e-4825-9ced-55ee2b17aac6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI f887e9da-cda5-496e-9040-57aab22e510a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a3406a09-928a-4bf1-b558-53ab744dc41b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ea9d191b-0a2d-49d5-a046-7ebe483993fa has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI eca38833-219e-439a-91ac-d3321772f71a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI dc740184-28b4-4db5-938b-837234f08290 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1ee50d61-1426-4322-892b-e270cfd4c97e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7bc9add5-cf5d-49c6-8022-78fb386a1e3b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 91f43c21-07ba-426f-a146-2d678e7de83b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a8dd3a48-8035-4a49-90f6-d91acdbbb032 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7f128b35-5e6f-4e32-b0c8-15059878c10c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5235fea0-4f20-4f19-859a-044b99ebce3c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 52e7b2de-84cb-4e1b-9a1f-4b898399a650 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2418d9e7-c3a7-424a-af36-fd983d438163 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7fc59e4-8d90-4157-b526-bc5ecc549b66 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI fd507e6c-c3e2-4305-81c8-f92da2703db9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 70081652-9efd-4881-85b6-c39b0de6f90b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5bf279c2-cff6-4eb6-8e70-376467d649f6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b37f9db8-b2bc-42d1-88b5-10207f9aeeb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 25937e64-4344-4772-8d8b-b5e9fee7cd81 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd0511c9-b803-48f2-afae-35fa51225c9e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7063926-088b-4cf0-8004-d75ea15653c5 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aff3122f-1693-4371-8d6f-5a2975aec85d has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 17200bad-cc3e-4b5d-9844-d1e56604d2a3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 99ad1143-0b88-46f0-ac0e-e5a72b69f327 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55c5e0a1-0af7-4b36-b5a6-57cff1185208 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2c5dec71-cd9a-4208-9e96-922b44544d09 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 27ccf472-3fa9-4d23-bdb1-6ca70f570230 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI e95206e0-4940-48fc-8564-238f2f0f6440 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5544ccc6-dec4-4528-91b4-302a6552146b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d1f154d5-9bdf-4c01-897c-367863efe114 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2476d6b3-1b25-47ca-bc8d-869125eed779 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a4fc211d-9e68-4f7c-99e5-170f3786ad55 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2ce4926e-c8dc-4185-bbb4-ce887ef78b42 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09ee300e-ac03-48f8-a6cc-bc33398fd75f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 78a67732-4889-4053-8a36-fb5d3cf1c0d3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ad9b1572-a29d-42bb-842d-448e49bdbbb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 60ad964b-b5aa-4077-9ca0-6acb84cbea15 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6cfe58b0-3dd7-422d-924c-284d811afa97 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bde8b13b-2f8c-454a-8bae-6f07fa907b9f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a9c40305-9f7c-4af9-be56-b1dd4f26458c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bc31eab4-7720-45fb-b6e8-23d529381221 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6a1ff504-f448-420a-ba4b-8a05d8a3f2da has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cbaf2499-489f-4b48-a017-e48fb73b4987 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bd2b4490-3464-4650-8847-11a376393ae9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 8874451d-249f-4913-ad9c-fb8da214f663 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a05390d4-5ef7-458d-b090-799215c49c6e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1e949f7b-3f04-4fcc-bdc9-2052974aef34 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 156877f1-0701-484e-9c60-659841110d38 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ab4c71e6-ac41-4fc6-8435-546ab827c728 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bb36c707-d6a9-4811-b3dc-33c694b2faf8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5df27f53-4a98-41ba-b577-7468e3eb9557 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd158763-7f28-42ca-bddb-30ffcc89e18f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b741152c-ce78-4d00-8227-2031ba3778bc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 450bddc9-946f-43c1-b9a3-5f50678ac249 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d07a41d6-0aa2-49a5-a241-b755c6daa3d8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 415196a5-de15-4a2a-912a-675f804af98e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 31ecf300-cc63-4845-8e7c-d191f4042135 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 357c608a-a805-48ab-8a96-d93fe1157304 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5111fe82-17b7-467c-851c-82ccb18d7bb1 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 653f53d3-1e8b-41e7-a997-41025bc268ef has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 832e9bfd-899a-41e3-baa6-373175b4b0ec has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55300191-34ec-4252-aa9f-3b2bdd5f98c7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d55536f8-906e-419e-a17a-318095312563 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7884662d-967e-43b1-a789-e92c1c149403 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09d26ac2-82a5-4208-8ebc-9d0df97665fc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 848ff799-092f-428b-90de-3a85517c7315 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:Writing database to /var/lib/xcp/state.db
INFO:root:Starting up xapi...

Everything is looking good!

Dry-run output

[14:09 ovbh-pprod-xen03 ~]# python3 snapshot-fixer.py dry-run
INFO:root:Regenerating database...
INFO:root:The VM b532b6e3-df2a-42e2-ade1-ac424c782938 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9707615f-334e-43bf-9946-95205bf8869f has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 1555cf78-7546-2b86-771d-ee66a306faf3 has Ref:941 as its "snapshot_of" value, changing to null.
INFO:root:The VM 99532d6b-a2e3-4133-8737-c416e8ef5bda has Ref:14 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9b7d927c-b70d-27f3-5d08-100d0744780f has Ref:50 as its "snapshot_of" value, changing to null.
INFO:root:The VM a04e5d4a-1067-4f91-89d7-8b5addb205f2 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 2b712337-b1e6-1b0f-d39e-5f3803bd8643 has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5b5c6ae6-2c9c-d76f-d9ae-7eff962287f0 has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM d155c631-b18f-c0d2-b425-49d53ed065fb has Ref:90 as its "snapshot_of" value, changing to null.
INFO:root:The VM 8c0a4696-4ea7-34b2-ca14-0252b4d6e357 has Ref:243 as its "snapshot_of" value, changing to null.
INFO:root:The VM c23a131e-e724-7f02-adba-e199526b401d has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5a2311cd-bbb7-6444-0119-9bb8c8a9e593 has Ref:265 as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5a6c334e-040e-4825-9ced-55ee2b17aac6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI f887e9da-cda5-496e-9040-57aab22e510a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a3406a09-928a-4bf1-b558-53ab744dc41b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ea9d191b-0a2d-49d5-a046-7ebe483993fa has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI eca38833-219e-439a-91ac-d3321772f71a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI dc740184-28b4-4db5-938b-837234f08290 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1ee50d61-1426-4322-892b-e270cfd4c97e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7bc9add5-cf5d-49c6-8022-78fb386a1e3b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 91f43c21-07ba-426f-a146-2d678e7de83b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a8dd3a48-8035-4a49-90f6-d91acdbbb032 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7f128b35-5e6f-4e32-b0c8-15059878c10c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5235fea0-4f20-4f19-859a-044b99ebce3c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 52e7b2de-84cb-4e1b-9a1f-4b898399a650 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2418d9e7-c3a7-424a-af36-fd983d438163 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7fc59e4-8d90-4157-b526-bc5ecc549b66 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI fd507e6c-c3e2-4305-81c8-f92da2703db9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 70081652-9efd-4881-85b6-c39b0de6f90b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5bf279c2-cff6-4eb6-8e70-376467d649f6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b37f9db8-b2bc-42d1-88b5-10207f9aeeb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 25937e64-4344-4772-8d8b-b5e9fee7cd81 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd0511c9-b803-48f2-afae-35fa51225c9e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7063926-088b-4cf0-8004-d75ea15653c5 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aff3122f-1693-4371-8d6f-5a2975aec85d has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 17200bad-cc3e-4b5d-9844-d1e56604d2a3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 99ad1143-0b88-46f0-ac0e-e5a72b69f327 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55c5e0a1-0af7-4b36-b5a6-57cff1185208 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2c5dec71-cd9a-4208-9e96-922b44544d09 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 27ccf472-3fa9-4d23-bdb1-6ca70f570230 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI e95206e0-4940-48fc-8564-238f2f0f6440 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5544ccc6-dec4-4528-91b4-302a6552146b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d1f154d5-9bdf-4c01-897c-367863efe114 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2476d6b3-1b25-47ca-bc8d-869125eed779 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a4fc211d-9e68-4f7c-99e5-170f3786ad55 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2ce4926e-c8dc-4185-bbb4-ce887ef78b42 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09ee300e-ac03-48f8-a6cc-bc33398fd75f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 78a67732-4889-4053-8a36-fb5d3cf1c0d3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ad9b1572-a29d-42bb-842d-448e49bdbbb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 60ad964b-b5aa-4077-9ca0-6acb84cbea15 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6cfe58b0-3dd7-422d-924c-284d811afa97 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bde8b13b-2f8c-454a-8bae-6f07fa907b9f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a9c40305-9f7c-4af9-be56-b1dd4f26458c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bc31eab4-7720-45fb-b6e8-23d529381221 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6a1ff504-f448-420a-ba4b-8a05d8a3f2da has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cbaf2499-489f-4b48-a017-e48fb73b4987 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bd2b4490-3464-4650-8847-11a376393ae9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 8874451d-249f-4913-ad9c-fb8da214f663 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a05390d4-5ef7-458d-b090-799215c49c6e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1e949f7b-3f04-4fcc-bdc9-2052974aef34 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 156877f1-0701-484e-9c60-659841110d38 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ab4c71e6-ac41-4fc6-8435-546ab827c728 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bb36c707-d6a9-4811-b3dc-33c694b2faf8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5df27f53-4a98-41ba-b577-7468e3eb9557 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd158763-7f28-42ca-bddb-30ffcc89e18f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b741152c-ce78-4d00-8227-2031ba3778bc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 450bddc9-946f-43c1-b9a3-5f50678ac249 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d07a41d6-0aa2-49a5-a241-b755c6daa3d8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 415196a5-de15-4a2a-912a-675f804af98e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 31ecf300-cc63-4845-8e7c-d191f4042135 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 357c608a-a805-48ab-8a96-d93fe1157304 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5111fe82-17b7-467c-851c-82ccb18d7bb1 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 653f53d3-1e8b-41e7-a997-41025bc268ef has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 832e9bfd-899a-41e3-baa6-373175b4b0ec has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55300191-34ec-4252-aa9f-3b2bdd5f98c7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d55536f8-906e-419e-a17a-318095312563 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7884662d-967e-43b1-a789-e92c1c149403 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09d26ac2-82a5-4208-8ebc-9d0df97665fc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 848ff799-092f-428b-90de-3a85517c7315 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.

@poddingue Appreciate it! I will check that out. Been digging to see wth is going on.

Hoped on xen03 to see what it had to say

[13:47 ovbh-pprod-xen03 ~]# ls -l /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0
lrwxrwxrwx 1 root root 17 Jul 30 13:15 /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0 -> ../../../drbd1227
[13:47 ovbh-pprod-xen03 ~]# vhd-util query -n /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0 -p
/dev/drbd/by-res/xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0
[13:47 ovbh-pprod-xen03 ~]# vhd-util read -p -n /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0
VHD Footer Summary:
-------------------
Cookie              : conectix
Features            : (0x00000002) <RESV>
File format version : Major: 1, Minor: 0
Data offset         : 512
Timestamp           : Sun Jul 26 03:46:10 2026
Creator Application : 'tap'
Creator version     : Major: 1, Minor: 3
Creator OS          : Unknown!
Original disk size  : 102400 MB (107374182400 Bytes)
Current disk size   : 102400 MB (107374182400 Bytes)
Geometry            : Cyl: 51400, Hds: 16, Sctrs: 255
                    : = 102398 MB (107372544000 Bytes)
Disk type           : Differencing hard disk
Checksum            : 0xffffee7e|0xffffee7e (Good!)
UUID                : c55dc0bf-77b9-431d-b16a-891509da7e54
Saved state         : No
Hidden              : 0

VHD Header Summary:
-------------------
Cookie              : cxsparse
Data offset (unusd) : 18446744073709551615
Table offset        : 1536
Header version      : 0x00010000
Max BAT size        : 1048576
Block size          : 2097152 (2 MB)
Parent name         : 0
Parent UUID         : efc16e1a-8384-444a-ac04-00f88c852cb6
Parent timestamp    : Thu Jul 30 13:14:52 2026
Checksum            : 0xffffe512|0xffffe512 (Good!)

VHD Parent Locators:
--------------------
locator:            : 0
       code         : PLAT_CODE_MACX
       data_space   : 512
       data_length  : 59
       data_offset  : 4327424
       decoded name : ../xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0

locator:            : 1
       code         : PLAT_CODE_W2KU
       data_space   : 512
       data_length  : 108
       data_offset  : 4327936
       decoded name : ./../xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0

locator:            : 2
       code         : PLAT_CODE_W2RU
       data_space   : 512
       data_length  : 108
       data_offset  : 4328448
       decoded name : ./../xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0

VHD Batmap Summary:
-------------------
Batmap offset       : 4196352
Batmap size (secs)  : 256
Batmap version      : 0x00010002
Checksum            : 0xfffd78cf|0xfffd78cf (Good!)

Which points to a base copy

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/volume-name │ xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d                                                                                                                                                                                                            │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep c7fc59e4-8d90-4157-b526-bc5ecc549b66
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/metadata    │ {"name_label": "base copy", "name_description": "", "is_a_snapshot": false, "snapshot_of": "", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": true, "metadata_of_pool": ""}                                                          │
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/not-exists  │ 0                                                                                                                                                                                                                                                          │
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/volume-name │ xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d                                                                                                                                                                                                            │

[13:47 ovbh-pprod-xen03 ~]# xe vdi-param-list uuid=c7fc59e4-8d90-4157-b526-bc5ecc549b66
uuid ( RO)                    : c7fc59e4-8d90-4157-b526-bc5ecc549b66
              name-label ( RW): base copy
        name-description ( RW): 
           is-a-snapshot ( RO): false
             snapshot-of ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
               snapshots ( RO): 
           snapshot-time ( RO): 19700101T00:00:00Z
      allowed-operations (SRO): copy; generate_config; clone; forget; update; resize; snapshot; destroy
      current-operations (SRO): 
                 sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
           sr-name-label ( RO): linstor-sr
               vbd-uuids (SRO): 
         crashdump-uuids (SRO): 
            virtual-size ( RO): 107374182400
    physical-utilisation ( RO): 106782908416
                location ( RO): c7fc59e4-8d90-4157-b526-bc5ecc549b66
                    type ( RO): User
                sharable ( RO): false
               read-only ( RO): true
            storage-lock ( RO): false
                 managed ( RO): false
     parent ( RO) [DEPRECATED]: <not in database>
                 missing ( RO): false
            is-tools-iso ( RO): false
            other-config (MRW): 
           xenstore-data (MRO): 
               sm-config (MRO): vhd-blocks: eJzt2T0KwCAMgNFAh44eu9fyKIVeQ/yJixVBLBQC8r0hyWKSIZspTQUNtzTvWs2fAwAAAAC2dlkvAPzptF4A+4mH3ezHbjQwcBq8SM2lcF8bcNBYE7r/iwziv7hc; image-format: vhd
                 on-boot ( RW): persist
           allow-caching ( RW): false
         metadata-latest ( RO): false
        metadata-of-pool ( RO): <not in database>
                    tags (SRW): 
             cbt-enabled ( RO): false

which also points back to 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0

Thought this was weird. Something happened and now most of the VDIs in linstor show up as snapshots now.
Looking at just these two, to compare.

LB VDI: 150cd8e3-633d-42af-91b2-faa0d1f7b658

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep 150cd8e3-633d-42af-91b2-faa0d1f7b658
│ xcp/volume/150cd8e3-633d-42af-91b2-faa0d1f7b658/metadata     │ {"name_label": "OVBH-VPROD-RANCHERLB", "name_description": "", "is_a_snapshot": false, "snapshot_of": null, "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}                                            │
│ xcp/volume/150cd8e3-633d-42af-91b2-faa0d1f7b658/not-exists   │ 0                                                                                                                                                                                                                                                          │
│ xcp/volume/150cd8e3-633d-42af-91b2-faa0d1f7b658/volume-name  │ xcp-volume-7fea7c97-3783-46b6-bcc6-cfd234b1f4a7                                                                                                                                                                                                            │
│ xcp/volume/ebdd16b6-5293-4c7d-ac37-b427fc3b368d/metadata     │ {"name_label": "OVBH-VPROD-RANCHERLB", "name_description": "", "is_a_snapshot": true, "snapshot_of": "150cd8e3-633d-42af-91b2-faa0d1f7b658", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}           │

[13:12 ovbh-pprod-xen01 ~]# xe vdi-param-list uuid=150cd8e3-633d-42af-91b2-faa0d1f7b658
uuid ( RO)                    : 150cd8e3-633d-42af-91b2-faa0d1f7b658
              name-label ( RW): OVBH-VPROD-RANCHERLB
        name-description ( RW): 
           is-a-snapshot ( RO): false
             snapshot-of ( RO): <not in database>
               snapshots ( RO): ebdd16b6-5293-4c7d-ac37-b427fc3b368d
           snapshot-time ( RO): 19700101T00:00:00Z
      allowed-operations (SRO): clone; snapshot
      current-operations (SRO): 
                 sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
           sr-name-label ( RO): linstor-sr
               vbd-uuids (SRO): 07b60986-8d19-aa23-f2ba-8d8ab96f93b4
         crashdump-uuids (SRO): 
            virtual-size ( RO): 53687091200
    physical-utilisation ( RO): 1108372480
                location ( RO): 150cd8e3-633d-42af-91b2-faa0d1f7b658
                    type ( RO): User
                sharable ( RO): false
               read-only ( RO): false
            storage-lock ( RO): false
                 managed ( RO): true
     parent ( RO) [DEPRECATED]: <not in database>
                 missing ( RO): false
            is-tools-iso ( RO): false
            other-config (MRW): 
           xenstore-data (MRO): 
               sm-config (MRO): host_OpaqueRef:b3c659f5-efb3-4202-9268-b824e3eed523: RW; read-caching-reason-85838113-e4b8-4520-9f6d-8f3cf554c8f1: SR_NOT_SUPPORTED; read-caching-enabled-on-85838113-e4b8-4520-9f6d-8f3cf554c8f1: false; image-format: vhd; vdi_type: vhd; vhd-parent: 28d1e0e1-29e6-4734-90cd-5e394fe25af8
                 on-boot ( RW): persist
           allow-caching ( RW): false
         metadata-latest ( RO): false
        metadata-of-pool ( RO): <not in database>
                    tags (SRW): 
             cbt-enabled ( RO): false

03 VDI: 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
│ xcp/volume/07f430b2-c7f1-41d8-bd9f-2fe67f4f458b/metadata     │ {"name_label": "OVBH-VPROD-RANCHER03", "name_description": "", "is_a_snapshot": false, "snapshot_of": null, "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}                                            │
│ xcp/volume/07f430b2-c7f1-41d8-bd9f-2fe67f4f458b/not-exists   │ 0                                                                                                                                                                                                                                                          │
│ xcp/volume/07f430b2-c7f1-41d8-bd9f-2fe67f4f458b/volume-name  │ xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5                                                                                                                                                                                                            │
│ xcp/volume/098cc02d-0d62-4f36-b86e-e85e4a2802e0/metadata     │ {"name_label": "OVBH-VPROD-RANCHER03", "name_description": "", "is_a_snapshot": true, "snapshot_of": "07f430b2-c7f1-41d8-bd9f-2fe67f4f458b", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}           │

[13:10 ovbh-pprod-xen01 ~]# xe vdi-param-list uuid=07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
uuid ( RO)                    : 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
              name-label ( RW): OVBH-VPROD-RANCHER03
        name-description ( RW): 
           is-a-snapshot ( RO): false
             snapshot-of ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
               snapshots ( RO): 098cc02d-0d62-4f36-b86e-e85e4a2802e0
           snapshot-time ( RO): 19700101T00:00:00Z
      allowed-operations (SRO): clone; snapshot
      current-operations (SRO): 
                 sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
           sr-name-label ( RO): linstor-sr
               vbd-uuids (SRO): bdfc6a23-a89b-eb7e-7509-97e910e7ca79
         crashdump-uuids (SRO): 
            virtual-size ( RO): 107374182400
    physical-utilisation ( RO): 24027230208
                location ( RO): 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
                    type ( RO): User
                sharable ( RO): false
               read-only ( RO): false
            storage-lock ( RO): false
                 managed ( RO): true
     parent ( RO) [DEPRECATED]: <not in database>
                 missing ( RO): false
            is-tools-iso ( RO): false
            other-config (MRW): 
           xenstore-data (MRO): 
               sm-config (MRO): host_OpaqueRef:7f23da94-1933-41ad-8d9a-27dac34bc1a9: RW; read-caching-reason-c03e1a45-4c4c-46f5-a2a1-d8de2e22a866: SR_NOT_SUPPORTED; read-caching-enabled-on-c03e1a45-4c4c-46f5-a2a1-d8de2e22a866: false; vhd-parent: 59c33f05-2601-46dd-af09-9e86b3e166e3; image-format: vhd; vdi_type: vhd
                 on-boot ( RW): persist
           allow-caching ( RW): false
         metadata-latest ( RO): false
        metadata-of-pool ( RO): <not in database>
                    tags (SRW): 
             cbt-enabled ( RO): false

HUH?

[13:14 ovbh-pprod-xen01 ~]# xe vdi-param-list uuid=1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
uuid ( RO)                    : 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
              name-label ( RW): OVBH-VPROD-RANCHER01
        name-description ( RW): 
           is-a-snapshot ( RO): false
             snapshot-of ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
               snapshots ( RO): 848ff799-092f-428b-90de-3a85517c7315; 09d26ac2-82a5-4208-8ebc-9d0df97665fc; 7884662d-967e-43b1-a789-e92c1c149403; d55536f8-906e-419e-a17a-318095312563; 55300191-34ec-4252-aa9f-3b2bdd5f98c7; 832e9bfd-899a-41e3-baa6-373175b4b0ec; 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67; aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775; 653f53d3-1e8b-41e7-a997-41025bc268ef; 5111fe82-17b7-467c-851c-82ccb18d7bb1; 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0; 357c608a-a805-48ab-8a96-d93fe1157304; 31ecf300-cc63-4845-8e7c-d191f4042135; 415196a5-de15-4a2a-912a-675f804af98e; d07a41d6-0aa2-49a5-a241-b755c6daa3d8; 450bddc9-946f-43c1-b9a3-5f50678ac249; c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae; b741152c-ce78-4d00-8227-2031ba3778bc; cd158763-7f28-42ca-bddb-30ffcc89e18f; 5df27f53-4a98-41ba-b577-7468e3eb9557; bb36c707-d6a9-4811-b3dc-33c694b2faf8; ab4c71e6-ac41-4fc6-8435-546ab827c728; 156877f1-0701-484e-9c60-659841110d38; 1e949f7b-3f04-4fcc-bdc9-2052974aef34; a05390d4-5ef7-458d-b090-799215c49c6e; 8874451d-249f-4913-ad9c-fb8da214f663; 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6; bd2b4490-3464-4650-8847-11a376393ae9; cbaf2499-489f-4b48-a017-e48fb73b4987; 6a1ff504-f448-420a-ba4b-8a05d8a3f2da; bc31eab4-7720-45fb-b6e8-23d529381221; 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271; a9c40305-9f7c-4af9-be56-b1dd4f26458c; bde8b13b-2f8c-454a-8bae-6f07fa907b9f; 6cfe58b0-3dd7-422d-924c-284d811afa97; 60ad964b-b5aa-4077-9ca0-6acb84cbea15; 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7; c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4; ad9b1572-a29d-42bb-842d-448e49bdbbb2; 78a67732-4889-4053-8a36-fb5d3cf1c0d3; 6ac57186-d928-4f34-8ce8-ae5f8b916107; 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0; 09ee300e-ac03-48f8-a6cc-bc33398fd75f; 2ce4926e-c8dc-4185-bbb4-ce887ef78b42; a4fc211d-9e68-4f7c-99e5-170f3786ad55; 2476d6b3-1b25-47ca-bc8d-869125eed779; d1f154d5-9bdf-4c01-897c-367863efe114; 5544ccc6-dec4-4528-91b4-302a6552146b; e95206e0-4940-48fc-8564-238f2f0f6440; 27ccf472-3fa9-4d23-bdb1-6ca70f570230; 2c5dec71-cd9a-4208-9e96-922b44544d09; 55c5e0a1-0af7-4b36-b5a6-57cff1185208; 99ad1143-0b88-46f0-ac0e-e5a72b69f327; 17200bad-cc3e-4b5d-9844-d1e56604d2a3; aff3122f-1693-4371-8d6f-5a2975aec85d; c7063926-088b-4cf0-8004-d75ea15653c5; cd0511c9-b803-48f2-afae-35fa51225c9e; 25937e64-4344-4772-8d8b-b5e9fee7cd81; 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b; b37f9db8-b2bc-42d1-88b5-10207f9aeeb2; 5bf279c2-cff6-4eb6-8e70-376467d649f6; 70081652-9efd-4881-85b6-c39b0de6f90b; fd507e6c-c3e2-4305-81c8-f92da2703db9; c7fc59e4-8d90-4157-b526-bc5ecc549b66; 2418d9e7-c3a7-424a-af36-fd983d438163; 52e7b2de-84cb-4e1b-9a1f-4b898399a650; 5235fea0-4f20-4f19-859a-044b99ebce3c; 7f128b35-5e6f-4e32-b0c8-15059878c10c; a8dd3a48-8035-4a49-90f6-d91acdbbb032; 91f43c21-07ba-426f-a146-2d678e7de83b; 7bc9add5-cf5d-49c6-8022-78fb386a1e3b; 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6; 1ee50d61-1426-4322-892b-e270cfd4c97e; dc740184-28b4-4db5-938b-837234f08290; eca38833-219e-439a-91ac-d3321772f71a; ea9d191b-0a2d-49d5-a046-7ebe483993fa; a3406a09-928a-4bf1-b558-53ab744dc41b; f887e9da-cda5-496e-9040-57aab22e510a; 5a6c334e-040e-4825-9ced-55ee2b17aac6
           snapshot-time ( RO): 20260716T19:53:31Z
      allowed-operations (SRO): copy; generate_config; clone; forget; update; resize; snapshot; destroy
      current-operations (SRO): 
                 sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
           sr-name-label ( RO): linstor-sr
               vbd-uuids (SRO): 
         crashdump-uuids (SRO): 
            virtual-size ( RO): 107374182400
    physical-utilisation ( RO): 21519360
                location ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
                    type ( RO): User
                sharable ( RO): false
               read-only ( RO): false
            storage-lock ( RO): false
                 managed ( RO): true
     parent ( RO) [DEPRECATED]: <not in database>
                 missing ( RO): false
            is-tools-iso ( RO): false
            other-config (MRW): 
           xenstore-data (MRO): 
               sm-config (MRO): vhd-parent: 848ff799-092f-428b-90de-3a85517c7315; image-format: vhd; vdi_type: vhd
                 on-boot ( RW): persist
           allow-caching ( RW): false
         metadata-latest ( RO): false
        metadata-of-pool ( RO): <not in database>
                    tags (SRW): 
             cbt-enabled ( RO): false

wth

jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
│ xcp/volume/1516113c-9dbd-4e23-b76d-e1ec8cbed0c0/metadata    │ {"name_label": "OVBH-VPROD-RANCHER01", "name_description": "", "is_a_snapshot": true, "snapshot_of": "752611f2-bd1a-4009-9e6d-ce76b82144c4", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""}           │
│ xcp/volume/1516113c-9dbd-4e23-b76d-e1ec8cbed0c0/not-exists  │ 0                                                                                                                                                                                                                                                          │
│ xcp/volume/1516113c-9dbd-4e23-b76d-e1ec8cbed0c0/volume-name │ xcp-volume-cc57261d-6309-4a88-a672-f5eed4b0bc8d                                                                                                                                                                                                            │

All hosts are restarted with 9.2.18 loaded now. Will see what happens next.

[15:56 ovbh-pprod-xen05 ~]# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090212
DRBD_KERNEL_VERSION=9.2.18
DRBDADM_VERSION_CODE=0x092100
DRBDADM_VERSION=9.33.0

New crash, this time xen02. The time is different because I updated the velero backup schedules to not all be in the middle of the night. Also still happened with the openvas schedule removed, which makes sense.
Xen02 now has 9.2.18 loaded. Will restart other hosts now.
I also uploaded logs if those are desired.

@MajorP93 My mistake. No I have not rebooted the hosts yet. https://xcp-ng.org/forum/post/107164 I am aware that the reboot is needed for it to be picked up, I went ahead and installed from test so that if host crashes again it is there.

@Mathieu-L Yes, the hosts had been restarted after the updates were installed which included 9.2.16

@Mathieu-L https://xcp-ng.org/forum/post/106873 I had increased it to 16G and it still crashed.

For now I have upgraded all hosts to use --enablerepo=xcp-ng-linstor-testing to get kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3
That way it is there to be picked up if the host crashes again.
https://github.com/xcp-ng-rpms/drbd/blob/master/SPECS/kmod-drbd.spec
https://koji.xcp-ng.org/buildinfo?buildID=5747

[12:15 ovbh-pprod-xen04 ~]# yum upgrade --enablerepo=xcp-ng-linstor-testing
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
grafana/signature                                                                                                                                                                                                                      |  810 B  00:00:00     
grafana/signature                                                                                                                                                                                                                      | 3.0 kB  00:00:00 !!! 
xcp-ng-linstor-testing/signature                                                                                                                                                                                                       |  473 B  00:00:00     
xcp-ng-linstor-testing/signature                                                                                                                                                                                                       | 3.0 kB  00:00:00 !!! 
zabbix                                                                                                                                                                                                                                 | 3.0 kB  00:00:00     
zabbix-non-supported                                                                                                                                                                                                                   | 2.9 kB  00:00:00     
(1/3): grafana/primary_db                                                                                                                                                                                                              | 879 kB  00:00:00     
(2/3): xcp-ng-linstor-testing/primary_db                                                                                                                                                                                               | 2.4 kB  00:00:00     
(3/3): zabbix/x86_64/primary_db                                                                                                                                                                                                        | 114 kB  00:00:00     
Resolving Dependencies
--> Running transaction check
---> Package kmod-drbd.x86_64 0:9.2.16-1.0.xcpng8.3 will be updated
---> Package kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3 will be an update
---> Package zabbix-agent.x86_64 0:7.0.27-release1.el7 will be updated
---> Package zabbix-agent.x86_64 0:7.0.28-release1.el7 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================================================================================================================================================================================================
 Package                                                    Arch                                                 Version                                                           Repository                                                            Size
==============================================================================================================================================================================================================================================================
Updating:
 kmod-drbd                                                  x86_64                                               9.2.18-2.0.xcpng8.3                                               xcp-ng-linstor-testing                                               2.8 M
 zabbix-agent                                               x86_64                                               7.0.28-release1.el7                                               zabbix                                                               660 k

Transaction Summary
==============================================================================================================================================================================================================================================================
Upgrade  2 Packages

Total download size: 3.4 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/2): zabbix-agent-7.0.28-release1.el7.x86_64.rpm                                                                                                                                                                                     | 660 kB  00:00:00     
(2/2): kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64.rpm                                                                                                                                                                                        | 2.8 MB  00:00:01     
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                                                                                         2.8 MB/s | 3.4 MB  00:00:01     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Updating   : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64                                                                                                                                                                                                       1/4 
  Updating   : zabbix-agent-7.0.28-release1.el7.x86_64                                                                                                                                                                                                    2/4 
  Cleanup    : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64                                                                                                                                                                                                       3/4 
  Cleanup    : zabbix-agent-7.0.27-release1.el7.x86_64                                                                                                                                                                                                    4/4 
  Verifying  : zabbix-agent-7.0.28-release1.el7.x86_64                                                                                                                                                                                                    1/4 
  Verifying  : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64                                                                                                                                                                                                       2/4 
  Verifying  : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64                                                                                                                                                                                                       3/4 
  Verifying  : zabbix-agent-7.0.27-release1.el7.x86_64                                                                                                                                                                                                    4/4 

Updated:
  kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3                                                                                       zabbix-agent.x86_64 0:7.0.28-release1.el7                                                                                      

Complete!

@poddingue I'm fully up to date, but only at version 9.2.16

[14:57 ovbh-pprod-xen02 ~]# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090210
DRBD_KERNEL_VERSION=9.2.16
DRBDADM_VERSION_CODE=0x092100
DRBDADM_VERSION=9.33.0
[14:57 ovbh-pprod-xen02 ~]# drbd-reactor --version
drbd-reactor 1.9.0
[14:57 ovbh-pprod-xen02 ~]# 

There are no updates that I can install

[14:08 ovbh-pprod-xen02 ~]# yum check-update
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
zabbix                                                                                                                                                                                                                                 | 3.0 kB  00:00:00     
zabbix-non-supported                                                                                                                                                                                                                   | 2.9 kB  00:00:00     
zabbix/x86_64/primary_db                                                                                                                                                                                                               | 114 kB  00:00:00     

zabbix-agent.x86_64                                                                                                         7.0.28-release1.el7                                                                                                         zabbix

@MajorP93 No we do not have pro support at the moment

hmmm. Really seems like a build with 9.2.19 could fix all of this.
How can I request for a build @poddingue ?

https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L61
9.2.17

• Fix a kernel crash triggered by a crafted/invalid netlink message

https://github.com/LINBIT/drbd/blob/drbd-9.2.19/ChangeLog#L24
9.2.19

• Fix an AB-BA deadlock between online resize and activity-log transactions
• Fix several races during connection teardown that could crash or
  hang (ack_sender requeue, pending ping work, in-progress lb-tcp
  connect)

Logs from xen04 look like they are the same too. Which makes sense.

20:01:53.805: The Linstor DeviceManager on xen04 triggered an asynchronous alignment for a newly created volume (pvc-fac0a18b-2c23-4fe8-b005-b4f492cec92f), padding it from 26220040 KiB to 26222592 KiB.

20:01:54.034: Simultaneously, an orchestrator (like Velero) began tearing the volume down, shifting the DRBD state to conn( Connected -> TearDown ) and pdsk( Diskless -> DUnknown ).

20:01:58.484: The DRBD worker thread evaluated the shrinking bitmap while the alignment was trying to alter it. This caused the general protection fault: 0000 [#1] SMP NOPTI at instruction RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd], crashing the kernel.

I was just moving around the velero backup schedules when it decided to run some again even though it was not time. Crashed xen04 this time. Deleting the possible offending backup schedule until this bug can be identified and fixed.

Uploaded new bug bundle to nextcloud too.

My first idea was to have velero use a different storage class. Something like

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-ephemeral-retain
parameters:
  linstor.csi.linbit.com/allowRemoteVolumeAccess: "true"
  linstor.csi.linbit.com/disklessOnRemaining: "true"
  linstor.csi.linbit.com/disklessStoragePool: DfltDisklessStorPool
  linstor.csi.linbit.com/placementCount: "3"
provisioner: linstor.csi.linbit.com
**reclaimPolicy: Retain**
volumeBindingMode: WaitForFirstConsumer

Then have a cronjob collected the Released pvcs every hour. The problem is that can not work, as velero uses the storage class of the backed up pvc. So I would have to change every linstor storage class to Retain, and would effectively disable the automatic garbage collection for the cluster. Which might be fine, but would love to avoid it.

My second idea is to eliminate the need for the resizing (another work around to hopefully get away from this race condition/bug)
I am also not seeing any way to modify the storageclass to round up or pad the pv. This would be to attempt to eliminate the need for drbd_bm_resize.
Currently looking into Kyverno Mutating Webhook to dynamically pad these volumes on the fly to see if that would help me out at all.
All pvs are already whole numbers of Gi.
Velero calculates the size of the temporary clone pvc based on the exact byte-count of the used/allocated snapshot metadata, not the #Gi specification of the original pvc.
Just confirmed with a test

Looks to be similar to last time

Time,Node,Component / Service,Logged Event & Impact
23:17:13,xen01,LINSTOR/Controller,Resource Created: linstor-csi provisions a short-lived volume: pvc-395da199-d455-4004-ac0d-f6c6016ebf4e.
23:17:16,xen01,kernel (drbd),Sync Begins: xen01 connects to xen05 as a SyncSource and begins synchronizing the volume to it.
23:17:49,xen01,LINSTOR/Controller,Teardown Requested: The CSI plugin changes its mind or finishes its task, requesting sequential deletion (DelRsc) of the volume replicas across the cluster.
23:17:51,xen01,LINSTOR/Controller,Source Disk Removed: Linstor removes the volume from xen01 itself (Toggle Disk ... removing disk).
23:17:52,xen01,kernel (drbd),Source Goes Diskless: xen01 detaches the underlying storage block (UpToDate -> Detaching -> Diskless). It alerts peers it can no longer satisfy read requests, calling drbd_bm_resize with capacity == 0.
23:17:53,xen05,LINSTOR/Satellite, While the cluster connection is actively tearing down, xen05's local background DeviceManager wakes up and detects the volume. It attempts an online re-alignment to round up the block layer size to match the 4MiB extent rule (26220040 KiB to 26222592 KiB).
23:17:54,xen05,kernel (drbd),Fatal Crash: xen05's DRBD driver tries to resize and count its synchronization bitmap block pointers. Because its peer source (xen01) dropped to zero capacity mid-flight, DRBD hits memory corruption. It triggers a General Protection Fault at drbd_bm_count_bits+0x223/0x300 and panics the kernel.

@poddingue This time it was host xen05. I got the request link for upload, so I have uploaded the bundle as well as all logs from xen05 and xen01 (xcp-ng pool master and had linstor-controller) So hoping we have everything needed. I will be digging into it to. It is in json format with all labels. Nothing in /var/crash/

@poddingue Just had a new crash. Got the export bug-report-20260719233946.tar.bz2 it is ~63MB There is a way to share large files but it has been a bit since I have done that so have forgotten.
I've got all the logs. Will dig into it in the morning. Also looking into ways to share grafana links securely. We have it behind Cloudflare Zero Trust.

Here you can see when the host was offline.

@poddingue

I should already have all the logs, including kernel

Nothing has happened since. I am also increasing dom0 ram for all hosts.

OK so that resource was a pain and did not want to give up, so I restarted the host.

Migration is running good now.