All hosts are restarted with 9.2.18 loaded now. Will see what happens next.
[15:56 ovbh-pprod-xen05 ~]# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090212
DRBD_KERNEL_VERSION=9.2.18
DRBDADM_VERSION_CODE=0x092100
DRBDADM_VERSION=9.33.0
I have amazing news!
After the upgrade to xcp-ng 8.3, I retested velero backup, and it all just works 
Completed Backup
jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml backup describe grafana-test
Name: grafana-test
Namespace: velero
Labels: objectset.rio.cattle.io/hash=c2b5f500ab5d9b8ffe14f2c70bf3742291df565c
velero.io/storage-location=default
Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/4SSQW/bPgzFvwvPtv9OajeJj/8N22HdBqxFL0MPlEQlWmTRkOhgQ5HvPsixE2yH7iji8ffIJ74CDu6ZYnIcoIMTeYpcOf7vtIICji4Y6OB/1MdxgAJ6EjQoCN0rYAgsKI5Dyk9WP0hLIqmi40qjiKfMcRlAq7pBY+py26qmbEi15a5p78vtaqe0oqbVVsO5AI+K/Ju4A6YDdKDXqrVtXaNqzU5traVVY9d6Uyt7t2nW693K2Pa+naABe4IO9hEtBiyFksClmgbUdN06a9NAOtvr5B4DDunA8uR64lGgg7u6rxMUYMji6OWZ/dhTeuIPaQ6os+gTFUA/tR8NmXd+TELxUfNA5hslHqOmBN13OF16ZwvNQShIqpZClYQj7qk6blPlGF5uzC/L3P+kvok7MB9z0OcCXPiLPLHmuLLWCfVfB4rTZ9/iaA5zHovNZz7R++k6JI50q89BXcuXYR5YT0DolkChABEPHWzW9cK+rPQx8jgsH/KQj+QT/frzXCdduc/Ca9u1Y7aaFvMu5Ang5Xz+HQAA//8X7Fu+/QIAAA
objectset.rio.cattle.io/id=e104add0-85b4-4eb5-9456-819bcbe45cfc
velero.io/resource-timeout=10m0s
velero.io/source-cluster-k8s-gitversion=v1.33.4+rke2r1
velero.io/source-cluster-k8s-major-version=1
velero.io/source-cluster-k8s-minor-version=33
Phase: Completed
Namespaces:
Included: grafana
Excluded: <none>
Resources:
Included cluster-scoped: <none>
Excluded cluster-scoped: volumesnapshotcontents.snapshot.storage.k8s.io
Included namespace-scoped: *
Excluded namespace-scoped: volumesnapshots.snapshot.storage.k8s.io
Label selector: <none>
Or label selector: <none>
Storage Location: default
Velero-Native Snapshot PVs: true
Snapshot Move Data: true
Data Mover: velero
TTL: 720h0m0s
CSISnapshotTimeout: 30m0s
ItemOperationTimeout: 4h0m0s
Hooks: <none>
Backup Format Version: 1.1.0
Started: 2025-10-15 15:29:52 -0700 PDT
Completed: 2025-10-15 15:31:25 -0700 PDT
Expiration: 2025-11-14 14:29:52 -0800 PST
Total items to be backed up: 35
Items backed up: 35
Backup Item Operations: 1 of 1 completed successfully, 0 failed (specify --details for more information)
Backup Volumes:
Velero-Native Snapshots: <none included>
CSI Snapshots:
grafana/central-grafana:
Data Movement: included, specify --details for more information
Pod Volume Backups: <none included>
HooksAttempted: 0
HooksFailed: 0
Completed Restore
jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml restore describe restore-grafana-test --details
Name: restore-grafana-test
Namespace: velero
Labels: objectset.rio.cattle.io/hash=252addb3ed156c52d9fa9b8c045b47a55d66c0af
Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW7zIBBA7zJrO5/j35gzfE2rtsomymIM45jGBgTjbKLcvaKJm6qL7kDwnt7ABdDpHfmgrQEBZxrJ25W2/85rSOCkjQIBrxTYeoIEJmJUyAjiAmiMZWRtTYhb232Q5EC88tquJDKPFEU6GlpUG5UVZdpUdZ6WZZ+niOtNWtR1SypvqC8buCYwYkfjn7oBwwAC8ipHpbqC1LqqZZWrtse228isrLqywapSdS0z7KPU4EQgwN+mSI8eezSYMgWG22lwKOl7/MgERzJmdChPs9veDL9IGfSbQRcGy+96IjszCCiyCRLQRo6zIrVd5AHEfuHhkIBmmp4d+a/3e9Dl8LPoCZ3T5hg7FvQRcR8nxt6XL7sAgv1MCZztOE+01P23cvmnPYzaxNtwuF4/AwAA//8k6OwC/QEAAA
objectset.rio.cattle.io/id=9ad8d034-7562-44f2-aa18-3669ed27ef47
Phase: Completed
Total items to be restored: 33
Items restored: 33
Started: 2025-10-15 15:35:26 -0700 PDT
Completed: 2025-10-15 15:36:34 -0700 PDT
Warnings:
Velero: <none>
Cluster: <none>
Namespaces:
grafana-restore: could not restore, ConfigMap:elasticsearch-es-transport-ca-internal already exists. Warning: the in-cluster version is different than the backed-up version
could not restore, ConfigMap:kube-root-ca.crt already exists. Warning: the in-cluster version is different than the backed-up version
Backup: grafana-test
Namespaces:
Included: grafana
Excluded: <none>
Resources:
Included: *
Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io
Cluster-scoped: auto
Namespace mappings: grafana=grafana-restore
Label selector: <none>
Or label selector: <none>
Restore PVs: true
CSI Snapshot Restores:
grafana-restore/central-grafana:
Data Movement:
Operation ID: dd-ffa56e1c-9fd0-44b4-a8bb-8163f40a49e9.330b82fc-ca6a-423217ee5
Data Mover: velero
Uploader Type: kopia
Existing Resource Policy: <none>
ItemOperationTimeout: 4h0m0s
Preserve Service NodePorts: auto
Restore Item Operations:
Operation for persistentvolumeclaims grafana-restore/central-grafana:
Restore Item Action Plugin: velero.io/csi-pvc-restorer
Operation ID: dd-ffa56e1c-9fd0-44b4-a8bb-8163f40a49e9.330b82fc-ca6a-423217ee5
Phase: Completed
Progress: 856284762 of 856284762 complete (Bytes)
Progress description: Completed
Created: 2025-10-15 15:35:28 -0700 PDT
Started: 2025-10-15 15:36:06 -0700 PDT
Updated: 2025-10-15 15:36:26 -0700 PDT
HooksAttempted: 0
HooksFailed: 0
Resource List:
apps/v1/Deployment:
- grafana-restore/central-grafana(created)
- grafana-restore/grafana-debug(created)
apps/v1/ReplicaSet:
- grafana-restore/central-grafana-5448b9f65(created)
- grafana-restore/central-grafana-56887c6cb6(created)
- grafana-restore/central-grafana-56ddd4f497(created)
- grafana-restore/central-grafana-5f4757844b(created)
- grafana-restore/central-grafana-5f69f86c85(created)
- grafana-restore/central-grafana-64545dcdc(created)
- grafana-restore/central-grafana-69c66c54d9(created)
- grafana-restore/central-grafana-6c8d6f65b8(created)
- grafana-restore/central-grafana-7b479f79ff(created)
- grafana-restore/central-grafana-bc7d96cdd(created)
- grafana-restore/central-grafana-cb88bd49c(created)
- grafana-restore/grafana-debug-556845ff7b(created)
- grafana-restore/grafana-debug-6fb594cb5f(created)
- grafana-restore/grafana-debug-8f66bfbf6(created)
discovery.k8s.io/v1/EndpointSlice:
- grafana-restore/central-grafana-hkgd5(created)
networking.k8s.io/v1/Ingress:
- grafana-restore/central-grafana(created)
rbac.authorization.k8s.io/v1/Role:
- grafana-restore/central-grafana(created)
rbac.authorization.k8s.io/v1/RoleBinding:
- grafana-restore/central-grafana(created)
v1/ConfigMap:
- grafana-restore/central-grafana(created)
- grafana-restore/elasticsearch-es-transport-ca-internal(failed)
- grafana-restore/kube-root-ca.crt(failed)
v1/Endpoints:
- grafana-restore/central-grafana(created)
v1/PersistentVolume:
- pvc-e3f6578f-08b2-4e79-85f0-76bbf8985b55(skipped)
v1/PersistentVolumeClaim:
- grafana-restore/central-grafana(created)
v1/Pod:
- grafana-restore/central-grafana-cb88bd49c-fc5br(created)
v1/Secret:
- grafana-restore/fpinfra-net-cf-cert(created)
- grafana-restore/grafana(created)
v1/Service:
- grafana-restore/central-grafana(created)
v1/ServiceAccount:
- grafana-restore/central-grafana(created)
- grafana-restore/default(skipped)
velero.io/v2alpha1/DataUpload:
- velero/grafana-test-nw7zj(skipped)
Image of working restore pod, with correct data in PV
Velero installed from helm: https://vmware-tanzu.github.io/helm-charts
Version: velero:11.1.0
Values
---
image:
repository: velero/velero
tag: v1.17.0
# Whether to deploy the restic daemonset.
deployNodeAgent: true
initContainers:
- name: velero-plugin-for-aws
image: velero/velero-plugin-for-aws:latest
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /target
name: plugins
configuration:
defaultItemOperationTimeout: 2h
features: EnableCSI
defaultSnapshotMoveData: true
backupStorageLocation:
- name: default
provider: aws
bucket: velero
config:
region: us-east-1
s3ForcePathStyle: true
s3Url: https://s3.location
# Destination VSL points to LINSTOR snapshot class
volumeSnapshotLocation:
- name: linstor
provider: velero.io/csi
config:
snapshotClass: linstor-vsc
credentials:
useSecret: true
existingSecret: velero-user
metrics:
enabled: true
serviceMonitor:
enabled: true
prometheusRule:
enabled: true
# Additional labels to add to deployed PrometheusRule
additionalLabels: {}
# PrometheusRule namespace. Defaults to Velero namespace.
# namespace: ""
# Rules to be deployed
spec:
- alert: VeleroBackupPartialFailures
annotations:
message: Velero backup {{ $labels.schedule }} has {{ $value | humanizePercentage }} partialy failed backups.
expr: |-
velero_backup_partial_failure_total{schedule!=""} / velero_backup_attempt_total{schedule!=""} > 0.25
for: 15m
labels:
severity: warning
- alert: VeleroBackupFailures
annotations:
message: Velero backup {{ $labels.schedule }} has {{ $value | humanizePercentage }} failed backups.
expr: |-
velero_backup_failure_total{schedule!=""} / velero_backup_attempt_total{schedule!=""} > 0.25
for: 15m
labels:
severity: warning
Also create the following.
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
name: linstor-vsc
labels:
velero.io/csi-volumesnapshot-class: "true"
driver: linstor.csi.linbit.com
deletionPolicy: Delete
We are using Piraeus operator to use xostor in k8s
https://github.com/piraeusdatastore/piraeus-operator.git
Version: v2.9.1
Values:
---
operator:
resources:
requests:
cpu: 250m
memory: 500Mi
limits:
memory: 1Gi
installCRDs: true
imageConfigOverride:
- base: quay.io/piraeusdatastore
components:
linstor-satellite:
image: piraeus-server
tag: v1.29.0
tls:
certManagerIssuerRef:
name: step-issuer
kind: StepClusterIssuer
group: certmanager.step.sm
Then we just connect to the xostor cluster like external linstor controller.
The problem was yum cache. If I did yum update right after yum update xcp-ng-release-linstor it would still fail. To get it working right away did the following
yum update xcp-ng-release-linstor
yum clean all
yum update
OK I figured it out! I made an init container that gets a manually created node label for the node the pod is running on. This value is the bare metal host for that k8s node. The init contianer then takes that value and makes a script wrapper and then calls linstor-csi with the correct values. After making these changes all the linstor csi containers are running with no errors.
Current problem comes from deploying and using storage class. Started with a basic one that failed, and noticed I did not know what the correct storage_pool_name name was, so went to http://IP:3370/v1/nodes/NODE/storage-pools and http://IP:3370/v1/nodes/NODE to get information.
Still troubleshooting, but wanted to provide info.
After getting back decided to run it.
[16:34 ovbh-pprod-xen01 ~]# python3 snapshot-fixer.py rewrite
INFO:root:Check HA...
INFO:root:Shutting down xapi...
INFO:root:Regenerating database...
INFO:root:The VM b532b6e3-df2a-42e2-ade1-ac424c782938 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9707615f-334e-43bf-9946-95205bf8869f has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 1555cf78-7546-2b86-771d-ee66a306faf3 has Ref:941 as its "snapshot_of" value, changing to null.
INFO:root:The VM 99532d6b-a2e3-4133-8737-c416e8ef5bda has Ref:14 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9b7d927c-b70d-27f3-5d08-100d0744780f has Ref:50 as its "snapshot_of" value, changing to null.
INFO:root:The VM a04e5d4a-1067-4f91-89d7-8b5addb205f2 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 2b712337-b1e6-1b0f-d39e-5f3803bd8643 has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5b5c6ae6-2c9c-d76f-d9ae-7eff962287f0 has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM d155c631-b18f-c0d2-b425-49d53ed065fb has Ref:90 as its "snapshot_of" value, changing to null.
INFO:root:The VM 8c0a4696-4ea7-34b2-ca14-0252b4d6e357 has Ref:243 as its "snapshot_of" value, changing to null.
INFO:root:The VM c23a131e-e724-7f02-adba-e199526b401d has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5a2311cd-bbb7-6444-0119-9bb8c8a9e593 has Ref:265 as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5a6c334e-040e-4825-9ced-55ee2b17aac6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI f887e9da-cda5-496e-9040-57aab22e510a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a3406a09-928a-4bf1-b558-53ab744dc41b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ea9d191b-0a2d-49d5-a046-7ebe483993fa has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI eca38833-219e-439a-91ac-d3321772f71a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI dc740184-28b4-4db5-938b-837234f08290 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1ee50d61-1426-4322-892b-e270cfd4c97e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7bc9add5-cf5d-49c6-8022-78fb386a1e3b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 91f43c21-07ba-426f-a146-2d678e7de83b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a8dd3a48-8035-4a49-90f6-d91acdbbb032 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7f128b35-5e6f-4e32-b0c8-15059878c10c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5235fea0-4f20-4f19-859a-044b99ebce3c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 52e7b2de-84cb-4e1b-9a1f-4b898399a650 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2418d9e7-c3a7-424a-af36-fd983d438163 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7fc59e4-8d90-4157-b526-bc5ecc549b66 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI fd507e6c-c3e2-4305-81c8-f92da2703db9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 70081652-9efd-4881-85b6-c39b0de6f90b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5bf279c2-cff6-4eb6-8e70-376467d649f6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b37f9db8-b2bc-42d1-88b5-10207f9aeeb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 25937e64-4344-4772-8d8b-b5e9fee7cd81 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd0511c9-b803-48f2-afae-35fa51225c9e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7063926-088b-4cf0-8004-d75ea15653c5 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aff3122f-1693-4371-8d6f-5a2975aec85d has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 17200bad-cc3e-4b5d-9844-d1e56604d2a3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 99ad1143-0b88-46f0-ac0e-e5a72b69f327 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55c5e0a1-0af7-4b36-b5a6-57cff1185208 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2c5dec71-cd9a-4208-9e96-922b44544d09 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 27ccf472-3fa9-4d23-bdb1-6ca70f570230 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI e95206e0-4940-48fc-8564-238f2f0f6440 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5544ccc6-dec4-4528-91b4-302a6552146b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d1f154d5-9bdf-4c01-897c-367863efe114 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2476d6b3-1b25-47ca-bc8d-869125eed779 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a4fc211d-9e68-4f7c-99e5-170f3786ad55 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2ce4926e-c8dc-4185-bbb4-ce887ef78b42 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09ee300e-ac03-48f8-a6cc-bc33398fd75f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 78a67732-4889-4053-8a36-fb5d3cf1c0d3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ad9b1572-a29d-42bb-842d-448e49bdbbb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 60ad964b-b5aa-4077-9ca0-6acb84cbea15 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6cfe58b0-3dd7-422d-924c-284d811afa97 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bde8b13b-2f8c-454a-8bae-6f07fa907b9f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a9c40305-9f7c-4af9-be56-b1dd4f26458c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bc31eab4-7720-45fb-b6e8-23d529381221 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6a1ff504-f448-420a-ba4b-8a05d8a3f2da has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cbaf2499-489f-4b48-a017-e48fb73b4987 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bd2b4490-3464-4650-8847-11a376393ae9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 8874451d-249f-4913-ad9c-fb8da214f663 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a05390d4-5ef7-458d-b090-799215c49c6e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1e949f7b-3f04-4fcc-bdc9-2052974aef34 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 156877f1-0701-484e-9c60-659841110d38 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ab4c71e6-ac41-4fc6-8435-546ab827c728 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bb36c707-d6a9-4811-b3dc-33c694b2faf8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5df27f53-4a98-41ba-b577-7468e3eb9557 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd158763-7f28-42ca-bddb-30ffcc89e18f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b741152c-ce78-4d00-8227-2031ba3778bc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 450bddc9-946f-43c1-b9a3-5f50678ac249 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d07a41d6-0aa2-49a5-a241-b755c6daa3d8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 415196a5-de15-4a2a-912a-675f804af98e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 31ecf300-cc63-4845-8e7c-d191f4042135 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 357c608a-a805-48ab-8a96-d93fe1157304 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5111fe82-17b7-467c-851c-82ccb18d7bb1 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 653f53d3-1e8b-41e7-a997-41025bc268ef has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 832e9bfd-899a-41e3-baa6-373175b4b0ec has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55300191-34ec-4252-aa9f-3b2bdd5f98c7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d55536f8-906e-419e-a17a-318095312563 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7884662d-967e-43b1-a789-e92c1c149403 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09d26ac2-82a5-4208-8ebc-9d0df97665fc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 848ff799-092f-428b-90de-3a85517c7315 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:Writing database to /var/lib/xcp/state.db
INFO:root:Starting up xapi...
Everything is looking good!
@Mathieu-L Yes, the hosts had been restarted after the updates were installed which included 9.2.16
@Mathieu-L https://xcp-ng.org/forum/post/106873 I had increased it to 16G and it still crashed.
@andrewperry I myself migrated our rancher management cluster from the original rke to a new rke2 cluster using this plan not too long ago, so you should not have much trouble. Feel free to ask questions 
@nathanael-h Nice 
If you have any questions let me know, I have been using this for all our on prem clusters for a while now.
I do not have any asks ATM, but I thought I would just share my plan that I use to create k8s clusters that we have been using for a while now.
It has grown over time and may be a bit messy, but figured better then nothing. We use this for rke2 rancher k8s clusters deployed onto out xcp-ng cluster. We use xostor for drives, and the vlan5 network is for piraeus operator to use for pv. We also use IPVS. We are using a rocky linux 9 vm template.
If these are useful to anyone and they have questions I will do my best to answer.
variable "pool" {
default = "OVBH-PROD-XENPOOL04"
}
variable "network0" {
default = "Native vRack"
}
variable "network1" {
default = "VLAN80"
}
variable "network2" {
default = "VLAN5"
}
variable "cluster_name" {
default = "Production K8s Cluster"
}
variable "enrollment_command" {
default = "curl -fL https://rancher.<redacted>.net/system-agent-install.sh | sudo sh -s - --server https://rancher.<redacted>.net --label 'cattle.io/os=linux' --token <redacted>"
}
variable "node_type" {
description = "Node type flag"
default = {
"1" = "--etcd --controlplane",
"2" = "--etcd --controlplane",
"3" = "--etcd --controlplane",
"4" = "--worker",
"5" = "--worker",
"6" = "--worker",
"7" = "--worker --taints smtp=true:NoSchedule",
"8" = "--worker --taints smtp=true:NoSchedule",
"9" = "--worker --taints smtp=true:NoSchedule"
}
}
variable "node_networks" {
description = "Node network flag"
default = {
"1" = "--internal-address 10.1.8.100 --address <redacted>",
"2" = "--internal-address 10.1.8.101 --address <redacted>",
"3" = "--internal-address 10.1.8.102 --address <redacted>",
"4" = "--internal-address 10.1.8.103 --address <redacted>",
"5" = "--internal-address 10.1.8.104 --address <redacted>",
"6" = "--internal-address 10.1.8.105 --address <redacted>",
"7" = "--internal-address 10.1.8.106 --address <redacted>",
"8" = "--internal-address 10.1.8.107 --address <redacted>",
"9" = "--internal-address 10.1.8.108 --address <redacted>"
}
}
variable "vm_name" {
description = "Node type flag"
default = {
"1" = "OVBH-VPROD-K8S01-MASTER01",
"2" = "OVBH-VPROD-K8S01-MASTER02",
"3" = "OVBH-VPROD-K8S01-MASTER03",
"4" = "OVBH-VPROD-K8S01-WORKER01",
"5" = "OVBH-VPROD-K8S01-WORKER02",
"6" = "OVBH-VPROD-K8S01-WORKER03",
"7" = "OVBH-VPROD-K8S01-WORKER04",
"8" = "OVBH-VPROD-K8S01-WORKER05",
"9" = "OVBH-VPROD-K8S01-WORKER06"
}
}
variable "preferred_host" {
default = {
"1" = "85838113-e4b8-4520-9f6d-8f3cf554c8f1",
"2" = "783c27ac-2dcb-4798-9ca8-27f5f30791f6",
"3" = "c03e1a45-4c4c-46f5-a2a1-d8de2e22a866",
"4" = "85838113-e4b8-4520-9f6d-8f3cf554c8f1",
"5" = "783c27ac-2dcb-4798-9ca8-27f5f30791f6",
"6" = "c03e1a45-4c4c-46f5-a2a1-d8de2e22a866",
"7" = "85838113-e4b8-4520-9f6d-8f3cf554c8f1",
"8" = "783c27ac-2dcb-4798-9ca8-27f5f30791f6",
"9" = "c03e1a45-4c4c-46f5-a2a1-d8de2e22a866"
}
}
variable "xoa_admin_password" {
}
variable "host_count" {
description = "All drives go to xostor"
default = {
"1" = "479ca676-20a1-4051-7189-a4a9ca47e00d",
"2" = "479ca676-20a1-4051-7189-a4a9ca47e00d",
"3" = "479ca676-20a1-4051-7189-a4a9ca47e00d",
"4" = "479ca676-20a1-4051-7189-a4a9ca47e00d",
"5" = "479ca676-20a1-4051-7189-a4a9ca47e00d",
"6" = "479ca676-20a1-4051-7189-a4a9ca47e00d",
"7" = "479ca676-20a1-4051-7189-a4a9ca47e00d",
"8" = "479ca676-20a1-4051-7189-a4a9ca47e00d",
"9" = "479ca676-20a1-4051-7189-a4a9ca47e00d"
}
}
variable "network1_ip_mapping" {
description = "Mapping for network1 ips, vlan80"
default = {
"1" = "10.1.8.100",
"2" = "10.1.8.101",
"3" = "10.1.8.102",
"4" = "10.1.8.103",
"5" = "10.1.8.104",
"6" = "10.1.8.105",
"7" = "10.1.8.106",
"8" = "10.1.8.107",
"9" = "10.1.8.108"
}
}
variable "network1_gateway" {
description = "Mapping for public ip gateways, from hosts"
default = "10.1.8.1"
}
variable "network1_prefix" {
description = "Prefix for the network used"
default = "22"
}
variable "network2_ip_mapping" {
description = "Mapping for network2 ips, VLAN5"
default = {
"1" = "10.2.5.30",
"2" = "10.2.5.31",
"3" = "10.2.5.32",
"4" = "10.2.5.33",
"5" = "10.2.5.34",
"6" = "10.2.5.35",
"7" = "10.2.5.36",
"8" = "10.2.5.37",
"9" = "10.2.5.38"
}
}
variable "network2_prefix" {
description = "Prefix for the network used"
default = "22"
}
variable "network0_ip_mapping" {
description = "Mapping for network0 ips, public"
default = {
<redacted>
}
}
variable "network0_gateway" {
description = "Mapping for public ip gateways, from hosts"
default = {
<redacted>
}
}
variable "network0_prefix" {
description = "Prefix for the network used"
default = {
<redacted>
}
}
# Instruct terraform to download the provider on `terraform init`
terraform {
required_providers {
xenorchestra = {
source = "vatesfr/xenorchestra"
version = "~> 0.29.0"
}
}
}
# Configure the XenServer Provider
provider "xenorchestra" {
# Must be ws or wss
url = "ws://10.2.0.5" # Or set XOA_URL environment variable
username = "admin@admin.net" # Or set XOA_USER environment variable
password = var.xoa_admin_password # Or set XOA_PASSWORD environment variable
}
data "xenorchestra_pool" "pool" {
name_label = var.pool
}
data "xenorchestra_template" "template" {
name_label = "Rocky Linux 9 Template"
pool_id = data.xenorchestra_pool.pool.id
}
data "xenorchestra_network" "net1" {
name_label = var.network1
pool_id = data.xenorchestra_pool.pool.id
}
data "xenorchestra_network" "net2" {
name_label = var.network2
pool_id = data.xenorchestra_pool.pool.id
}
data "xenorchestra_network" "net0" {
name_label = var.network0
pool_id = data.xenorchestra_pool.pool.id
}
resource "xenorchestra_cloud_config" "node" {
count = 9
name = "${lower(lookup(var.vm_name, count.index + 1))}_cloud_config"
template = <<EOF
#cloud-config
ssh_authorized_keys:
- ssh-rsa <redacted>
write_files:
- path: /etc/NetworkManager/conf.d/rke2-canal.conf
permissions: '0755'
owner: root
content: |
[keyfile]
unmanaged-devices=interface-name:cali*;interface-name:flannel*
- path: /tmp/selinux_kmod_drbd.log
permissions: '0640'
owner: root
content: |
type=AVC msg=audit(1661803314.183:778): avc: denied { module_load } for pid=148256 comm="insmod" path="/tmp/ko/drbd.ko" dev="overlay" ino=101839829 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=system permissive=0
type=AVC msg=audit(1661803314.185:779): avc: denied { module_load } for pid=148257 comm="insmod" path="/tmp/ko/drbd_transport_tcp.ko" dev="overlay" ino=101839831 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=system_u:object_r:var_lib_t:s0 tclass=system permissive=0
- path: /etc/sysconfig/modules/ipvs.modules
permissions: 0755
owner: root
content: |
#!/bin/bash
modprobe -- ip_vs
modprobe -- ip_vs_rr
modprobe -- ip_vs_wrr
modprobe -- ip_vs_sh
modprobe -- nf_conntrack
- path: /etc/modules-load.d/ipvs.conf
permissions: 0755
owner: root
content: |
ip_vs
ip_vs_rr
ip_vs_wrr
ip_vs_sh
nf_conntrack
#cloud-init
runcmd:
- sudo hostnamectl set-hostname --static ${lower(lookup(var.vm_name, count.index + 1))}.<redacted>.com
- sudo hostnamectl set-hostname ${lower(lookup(var.vm_name, count.index + 1))}.<redacted>.com
- nmcli -t -f NAME con show | xargs -d '\n' -I {} nmcli con delete "{}"
- nmcli con add type ethernet con-name public ifname enX0
- nmcli con mod public ipv4.address '${lookup(var.network0_ip_mapping, count.index + 1)}/${lookup(var.network0_prefix, count.index + 1)}'
- nmcli con mod public ipv4.method manual
- nmcli con mod public ipv4.ignore-auto-dns yes
- nmcli con mod public ipv4.gateway '${lookup(var.network0_gateway, count.index + 1)}'
- nmcli con mod public ipv4.dns "8.8.8.8 8.8.4.4"
- nmcli con mod public connection.autoconnect true
- nmcli con up public
- nmcli con add type ethernet con-name vlan80 ifname enX1
- nmcli con mod vlan80 ipv4.address '${lookup(var.network1_ip_mapping, count.index + 1)}/${var.network1_prefix}'
- nmcli con mod vlan80 ipv4.method manual
- nmcli con mod vlan80 ipv4.ignore-auto-dns yes
- nmcli con mod vlan80 ipv4.ignore-auto-routes yes
- nmcli con mod vlan80 ipv4.gateway '${var.network1_gateway}'
- nmcli con mod vlan80 ipv4.dns "${var.network1_gateway}"
- nmcli con mod vlan80 connection.autoconnect true
- nmcli con mod vlan80 ipv4.never-default true
- nmcli con mod vlan80 ipv6.never-default true
- nmcli con mod vlan80 ipv4.routes "10.0.0.0/8 ${var.network1_gateway}"
- nmcli con up vlan80
- nmcli con add type ethernet con-name vlan5 ifname enX2
- nmcli con mod vlan5 ipv4.address '${lookup(var.network2_ip_mapping, count.index + 1)}/${var.network2_prefix}'
- nmcli con mod vlan5 ipv4.method manual
- nmcli con mod vlan5 ipv4.ignore-auto-dns yes
- nmcli con mod vlan5 ipv4.ignore-auto-routes yes
- nmcli con mod vlan5 connection.autoconnect true
- nmcli con mod vlan5 ipv4.never-default true
- nmcli con mod vlan5 ipv6.never-default true
- nmcli con up vlan5
- systemctl restart NetworkManager
- dnf upgrade -y
- dnf install ipset ipvsadm -y
- bash /etc/sysconfig/modules/ipvs.modules
- dnf install chrony -y
- sudo systemctl enable --now chronyd
- yum install kernel-devel kernel-headers -y
- yum install elfutils-libelf-devel -y
- swapoff -a
- modprobe -- ip_tables
- systemctl disable --now firewalld.service
- systemctl disable --now rngd
- dnf config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo
- dnf install containerd.io tar -y
- dnf install policycoreutils-python-utils -y
- cat /tmp/selinux_kmod_drbd.log | sudo audit2allow -M insmoddrbd
- sudo semodule -i insmoddrbd.pp
- ${var.enrollment_command} ${lookup(var.node_type, count.index + 1)} ${lookup(var.node_networks, count.index + 1)}
bootcmd:
- swapoff -a
- modprobe -- ip_tables
EOF
}
resource "xenorchestra_vm" "master" {
count = 3
cpus = 4
memory_max = 8589934592
cloud_config = xenorchestra_cloud_config.node[count.index].template
name_label = lookup(var.vm_name, count.index + 1)
name_description = "${var.cluster_name} master"
template = data.xenorchestra_template.template.id
auto_poweron = true
affinity_host = lookup(var.preferred_host, count.index + 1)
network {
network_id = data.xenorchestra_network.net0.id
}
network {
network_id = data.xenorchestra_network.net1.id
}
network {
network_id = data.xenorchestra_network.net2.id
}
disk {
sr_id = lookup(var.host_count, count.index + 1)
name_label = "Terraform_disk_imavo"
size = 107374182400
}
}
resource "xenorchestra_vm" "worker" {
count = 3
cpus = 32
memory_max = 68719476736
cloud_config = xenorchestra_cloud_config.node[count.index + 3].template
name_label = lookup(var.vm_name, count.index + 3 + 1)
name_description = "${var.cluster_name} worker"
template = data.xenorchestra_template.template.id
auto_poweron = true
affinity_host = lookup(var.preferred_host, count.index + 3 + 1)
network {
network_id = data.xenorchestra_network.net0.id
}
network {
network_id = data.xenorchestra_network.net1.id
}
network {
network_id = data.xenorchestra_network.net2.id
}
disk {
sr_id = lookup(var.host_count, count.index + 3 + 1)
name_label = "Terraform_disk_imavo"
size = 322122547200
}
}
resource "xenorchestra_vm" "smtp" {
count = 3
cpus = 4
memory_max = 8589934592
cloud_config = xenorchestra_cloud_config.node[count.index + 6].template
name_label = lookup(var.vm_name, count.index + 6 + 1)
name_description = "${var.cluster_name} smtp worker"
template = data.xenorchestra_template.template.id
auto_poweron = true
affinity_host = lookup(var.preferred_host, count.index + 6 + 1)
network {
network_id = data.xenorchestra_network.net0.id
}
network {
network_id = data.xenorchestra_network.net1.id
}
network {
network_id = data.xenorchestra_network.net2.id
}
disk {
sr_id = lookup(var.host_count, count.index + 6 + 1)
name_label = "Terraform_disk_imavo"
size = 53687091200
}
}
After getting back decided to run it.
[16:34 ovbh-pprod-xen01 ~]# python3 snapshot-fixer.py rewrite
INFO:root:Check HA...
INFO:root:Shutting down xapi...
INFO:root:Regenerating database...
INFO:root:The VM b532b6e3-df2a-42e2-ade1-ac424c782938 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9707615f-334e-43bf-9946-95205bf8869f has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 1555cf78-7546-2b86-771d-ee66a306faf3 has Ref:941 as its "snapshot_of" value, changing to null.
INFO:root:The VM 99532d6b-a2e3-4133-8737-c416e8ef5bda has Ref:14 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9b7d927c-b70d-27f3-5d08-100d0744780f has Ref:50 as its "snapshot_of" value, changing to null.
INFO:root:The VM a04e5d4a-1067-4f91-89d7-8b5addb205f2 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 2b712337-b1e6-1b0f-d39e-5f3803bd8643 has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5b5c6ae6-2c9c-d76f-d9ae-7eff962287f0 has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM d155c631-b18f-c0d2-b425-49d53ed065fb has Ref:90 as its "snapshot_of" value, changing to null.
INFO:root:The VM 8c0a4696-4ea7-34b2-ca14-0252b4d6e357 has Ref:243 as its "snapshot_of" value, changing to null.
INFO:root:The VM c23a131e-e724-7f02-adba-e199526b401d has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5a2311cd-bbb7-6444-0119-9bb8c8a9e593 has Ref:265 as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5a6c334e-040e-4825-9ced-55ee2b17aac6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI f887e9da-cda5-496e-9040-57aab22e510a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a3406a09-928a-4bf1-b558-53ab744dc41b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ea9d191b-0a2d-49d5-a046-7ebe483993fa has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI eca38833-219e-439a-91ac-d3321772f71a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI dc740184-28b4-4db5-938b-837234f08290 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1ee50d61-1426-4322-892b-e270cfd4c97e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7bc9add5-cf5d-49c6-8022-78fb386a1e3b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 91f43c21-07ba-426f-a146-2d678e7de83b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a8dd3a48-8035-4a49-90f6-d91acdbbb032 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7f128b35-5e6f-4e32-b0c8-15059878c10c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5235fea0-4f20-4f19-859a-044b99ebce3c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 52e7b2de-84cb-4e1b-9a1f-4b898399a650 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2418d9e7-c3a7-424a-af36-fd983d438163 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7fc59e4-8d90-4157-b526-bc5ecc549b66 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI fd507e6c-c3e2-4305-81c8-f92da2703db9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 70081652-9efd-4881-85b6-c39b0de6f90b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5bf279c2-cff6-4eb6-8e70-376467d649f6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b37f9db8-b2bc-42d1-88b5-10207f9aeeb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 25937e64-4344-4772-8d8b-b5e9fee7cd81 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd0511c9-b803-48f2-afae-35fa51225c9e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7063926-088b-4cf0-8004-d75ea15653c5 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aff3122f-1693-4371-8d6f-5a2975aec85d has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 17200bad-cc3e-4b5d-9844-d1e56604d2a3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 99ad1143-0b88-46f0-ac0e-e5a72b69f327 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55c5e0a1-0af7-4b36-b5a6-57cff1185208 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2c5dec71-cd9a-4208-9e96-922b44544d09 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 27ccf472-3fa9-4d23-bdb1-6ca70f570230 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI e95206e0-4940-48fc-8564-238f2f0f6440 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5544ccc6-dec4-4528-91b4-302a6552146b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d1f154d5-9bdf-4c01-897c-367863efe114 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2476d6b3-1b25-47ca-bc8d-869125eed779 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a4fc211d-9e68-4f7c-99e5-170f3786ad55 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2ce4926e-c8dc-4185-bbb4-ce887ef78b42 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09ee300e-ac03-48f8-a6cc-bc33398fd75f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 78a67732-4889-4053-8a36-fb5d3cf1c0d3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ad9b1572-a29d-42bb-842d-448e49bdbbb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 60ad964b-b5aa-4077-9ca0-6acb84cbea15 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6cfe58b0-3dd7-422d-924c-284d811afa97 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bde8b13b-2f8c-454a-8bae-6f07fa907b9f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a9c40305-9f7c-4af9-be56-b1dd4f26458c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bc31eab4-7720-45fb-b6e8-23d529381221 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6a1ff504-f448-420a-ba4b-8a05d8a3f2da has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cbaf2499-489f-4b48-a017-e48fb73b4987 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bd2b4490-3464-4650-8847-11a376393ae9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 8874451d-249f-4913-ad9c-fb8da214f663 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a05390d4-5ef7-458d-b090-799215c49c6e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1e949f7b-3f04-4fcc-bdc9-2052974aef34 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 156877f1-0701-484e-9c60-659841110d38 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ab4c71e6-ac41-4fc6-8435-546ab827c728 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bb36c707-d6a9-4811-b3dc-33c694b2faf8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5df27f53-4a98-41ba-b577-7468e3eb9557 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd158763-7f28-42ca-bddb-30ffcc89e18f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b741152c-ce78-4d00-8227-2031ba3778bc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 450bddc9-946f-43c1-b9a3-5f50678ac249 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d07a41d6-0aa2-49a5-a241-b755c6daa3d8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 415196a5-de15-4a2a-912a-675f804af98e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 31ecf300-cc63-4845-8e7c-d191f4042135 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 357c608a-a805-48ab-8a96-d93fe1157304 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5111fe82-17b7-467c-851c-82ccb18d7bb1 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 653f53d3-1e8b-41e7-a997-41025bc268ef has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 832e9bfd-899a-41e3-baa6-373175b4b0ec has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55300191-34ec-4252-aa9f-3b2bdd5f98c7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d55536f8-906e-419e-a17a-318095312563 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7884662d-967e-43b1-a789-e92c1c149403 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09d26ac2-82a5-4208-8ebc-9d0df97665fc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 848ff799-092f-428b-90de-3a85517c7315 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:Writing database to /var/lib/xcp/state.db
INFO:root:Starting up xapi...
Everything is looking good!
Dry-run output
[14:09 ovbh-pprod-xen03 ~]# python3 snapshot-fixer.py dry-run
INFO:root:Regenerating database...
INFO:root:The VM b532b6e3-df2a-42e2-ade1-ac424c782938 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9707615f-334e-43bf-9946-95205bf8869f has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 1555cf78-7546-2b86-771d-ee66a306faf3 has Ref:941 as its "snapshot_of" value, changing to null.
INFO:root:The VM 99532d6b-a2e3-4133-8737-c416e8ef5bda has Ref:14 as its "snapshot_of" value, changing to null.
INFO:root:The VM 9b7d927c-b70d-27f3-5d08-100d0744780f has Ref:50 as its "snapshot_of" value, changing to null.
INFO:root:The VM a04e5d4a-1067-4f91-89d7-8b5addb205f2 has Ref:1 as its "snapshot_of" value, changing to null.
INFO:root:The VM 2b712337-b1e6-1b0f-d39e-5f3803bd8643 has Ref:9 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5b5c6ae6-2c9c-d76f-d9ae-7eff962287f0 has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM d155c631-b18f-c0d2-b425-49d53ed065fb has Ref:90 as its "snapshot_of" value, changing to null.
INFO:root:The VM 8c0a4696-4ea7-34b2-ca14-0252b4d6e357 has Ref:243 as its "snapshot_of" value, changing to null.
INFO:root:The VM c23a131e-e724-7f02-adba-e199526b401d has Ref:20 as its "snapshot_of" value, changing to null.
INFO:root:The VM 5a2311cd-bbb7-6444-0119-9bb8c8a9e593 has Ref:265 as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5a6c334e-040e-4825-9ced-55ee2b17aac6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI f887e9da-cda5-496e-9040-57aab22e510a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a3406a09-928a-4bf1-b558-53ab744dc41b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ea9d191b-0a2d-49d5-a046-7ebe483993fa has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI eca38833-219e-439a-91ac-d3321772f71a has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI dc740184-28b4-4db5-938b-837234f08290 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1ee50d61-1426-4322-892b-e270cfd4c97e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7bc9add5-cf5d-49c6-8022-78fb386a1e3b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 91f43c21-07ba-426f-a146-2d678e7de83b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a8dd3a48-8035-4a49-90f6-d91acdbbb032 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7f128b35-5e6f-4e32-b0c8-15059878c10c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5235fea0-4f20-4f19-859a-044b99ebce3c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 52e7b2de-84cb-4e1b-9a1f-4b898399a650 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2418d9e7-c3a7-424a-af36-fd983d438163 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7fc59e4-8d90-4157-b526-bc5ecc549b66 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI fd507e6c-c3e2-4305-81c8-f92da2703db9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 70081652-9efd-4881-85b6-c39b0de6f90b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5bf279c2-cff6-4eb6-8e70-376467d649f6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b37f9db8-b2bc-42d1-88b5-10207f9aeeb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 25937e64-4344-4772-8d8b-b5e9fee7cd81 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd0511c9-b803-48f2-afae-35fa51225c9e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c7063926-088b-4cf0-8004-d75ea15653c5 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aff3122f-1693-4371-8d6f-5a2975aec85d has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 17200bad-cc3e-4b5d-9844-d1e56604d2a3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 99ad1143-0b88-46f0-ac0e-e5a72b69f327 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55c5e0a1-0af7-4b36-b5a6-57cff1185208 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2c5dec71-cd9a-4208-9e96-922b44544d09 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 27ccf472-3fa9-4d23-bdb1-6ca70f570230 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI e95206e0-4940-48fc-8564-238f2f0f6440 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5544ccc6-dec4-4528-91b4-302a6552146b has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d1f154d5-9bdf-4c01-897c-367863efe114 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2476d6b3-1b25-47ca-bc8d-869125eed779 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a4fc211d-9e68-4f7c-99e5-170f3786ad55 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 2ce4926e-c8dc-4185-bbb4-ce887ef78b42 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09ee300e-ac03-48f8-a6cc-bc33398fd75f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 78a67732-4889-4053-8a36-fb5d3cf1c0d3 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ad9b1572-a29d-42bb-842d-448e49bdbbb2 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 60ad964b-b5aa-4077-9ca0-6acb84cbea15 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6cfe58b0-3dd7-422d-924c-284d811afa97 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bde8b13b-2f8c-454a-8bae-6f07fa907b9f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a9c40305-9f7c-4af9-be56-b1dd4f26458c has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bc31eab4-7720-45fb-b6e8-23d529381221 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 6a1ff504-f448-420a-ba4b-8a05d8a3f2da has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cbaf2499-489f-4b48-a017-e48fb73b4987 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bd2b4490-3464-4650-8847-11a376393ae9 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 8874451d-249f-4913-ad9c-fb8da214f663 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI a05390d4-5ef7-458d-b090-799215c49c6e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 1e949f7b-3f04-4fcc-bdc9-2052974aef34 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 156877f1-0701-484e-9c60-659841110d38 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI ab4c71e6-ac41-4fc6-8435-546ab827c728 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI bb36c707-d6a9-4811-b3dc-33c694b2faf8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5df27f53-4a98-41ba-b577-7468e3eb9557 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI cd158763-7f28-42ca-bddb-30ffcc89e18f has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI b741152c-ce78-4d00-8227-2031ba3778bc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 450bddc9-946f-43c1-b9a3-5f50678ac249 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d07a41d6-0aa2-49a5-a241-b755c6daa3d8 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 415196a5-de15-4a2a-912a-675f804af98e has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 31ecf300-cc63-4845-8e7c-d191f4042135 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 357c608a-a805-48ab-8a96-d93fe1157304 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 5111fe82-17b7-467c-851c-82ccb18d7bb1 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 653f53d3-1e8b-41e7-a997-41025bc268ef has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 832e9bfd-899a-41e3-baa6-373175b4b0ec has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 55300191-34ec-4252-aa9f-3b2bdd5f98c7 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI d55536f8-906e-419e-a17a-318095312563 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 7884662d-967e-43b1-a789-e92c1c149403 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 09d26ac2-82a5-4208-8ebc-9d0df97665fc has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
INFO:root:The VDI 848ff799-092f-428b-90de-3a85517c7315 has OpaqueRef:780aa24c-76c2-90c1-47d6-c4059770196c as its "snapshot_of" value, changing to null.
@poddingue Appreciate it! I will check that out. Been digging to see wth is going on.
Hoped on xen03 to see what it had to say
[13:47 ovbh-pprod-xen03 ~]# ls -l /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0
lrwxrwxrwx 1 root root 17 Jul 30 13:15 /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0 -> ../../../drbd1227
[13:47 ovbh-pprod-xen03 ~]# vhd-util query -n /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0 -p
/dev/drbd/by-res/xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0
[13:47 ovbh-pprod-xen03 ~]# vhd-util read -p -n /dev/drbd/by-res/xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5/0
VHD Footer Summary:
-------------------
Cookie : conectix
Features : (0x00000002) <RESV>
File format version : Major: 1, Minor: 0
Data offset : 512
Timestamp : Sun Jul 26 03:46:10 2026
Creator Application : 'tap'
Creator version : Major: 1, Minor: 3
Creator OS : Unknown!
Original disk size : 102400 MB (107374182400 Bytes)
Current disk size : 102400 MB (107374182400 Bytes)
Geometry : Cyl: 51400, Hds: 16, Sctrs: 255
: = 102398 MB (107372544000 Bytes)
Disk type : Differencing hard disk
Checksum : 0xffffee7e|0xffffee7e (Good!)
UUID : c55dc0bf-77b9-431d-b16a-891509da7e54
Saved state : No
Hidden : 0
VHD Header Summary:
-------------------
Cookie : cxsparse
Data offset (unusd) : 18446744073709551615
Table offset : 1536
Header version : 0x00010000
Max BAT size : 1048576
Block size : 2097152 (2 MB)
Parent name : 0
Parent UUID : efc16e1a-8384-444a-ac04-00f88c852cb6
Parent timestamp : Thu Jul 30 13:14:52 2026
Checksum : 0xffffe512|0xffffe512 (Good!)
VHD Parent Locators:
--------------------
locator: : 0
code : PLAT_CODE_MACX
data_space : 512
data_length : 59
data_offset : 4327424
decoded name : ../xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0
locator: : 1
code : PLAT_CODE_W2KU
data_space : 512
data_length : 108
data_offset : 4327936
decoded name : ./../xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0
locator: : 2
code : PLAT_CODE_W2RU
data_space : 512
data_length : 108
data_offset : 4328448
decoded name : ./../xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d/0
VHD Batmap Summary:
-------------------
Batmap offset : 4196352
Batmap size (secs) : 256
Batmap version : 0x00010002
Checksum : 0xfffd78cf|0xfffd78cf (Good!)
Which points to a base copy
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/volume-name │ xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d │
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep c7fc59e4-8d90-4157-b526-bc5ecc549b66
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/metadata │ {"name_label": "base copy", "name_description": "", "is_a_snapshot": false, "snapshot_of": "", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": true, "metadata_of_pool": ""} │
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/not-exists │ 0 │
│ xcp/volume/c7fc59e4-8d90-4157-b526-bc5ecc549b66/volume-name │ xcp-volume-353823c8-7dcb-4204-8a69-a4ecc9b3873d │
[13:47 ovbh-pprod-xen03 ~]# xe vdi-param-list uuid=c7fc59e4-8d90-4157-b526-bc5ecc549b66
uuid ( RO) : c7fc59e4-8d90-4157-b526-bc5ecc549b66
name-label ( RW): base copy
name-description ( RW):
is-a-snapshot ( RO): false
snapshot-of ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
snapshots ( RO):
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): copy; generate_config; clone; forget; update; resize; snapshot; destroy
current-operations (SRO):
sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
sr-name-label ( RO): linstor-sr
vbd-uuids (SRO):
crashdump-uuids (SRO):
virtual-size ( RO): 107374182400
physical-utilisation ( RO): 106782908416
location ( RO): c7fc59e4-8d90-4157-b526-bc5ecc549b66
type ( RO): User
sharable ( RO): false
read-only ( RO): true
storage-lock ( RO): false
managed ( RO): false
parent ( RO) [DEPRECATED]: <not in database>
missing ( RO): false
is-tools-iso ( RO): false
other-config (MRW):
xenstore-data (MRO):
sm-config (MRO): vhd-blocks: eJzt2T0KwCAMgNFAh44eu9fyKIVeQ/yJixVBLBQC8r0hyWKSIZspTQUNtzTvWs2fAwAAAAC2dlkvAPzptF4A+4mH3ezHbjQwcBq8SM2lcF8bcNBYE7r/iwziv7hc; image-format: vhd
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
tags (SRW):
cbt-enabled ( RO): false
which also points back to 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
Thought this was weird. Something happened and now most of the VDIs in linstor show up as snapshots now.
Looking at just these two, to compare.
LB VDI: 150cd8e3-633d-42af-91b2-faa0d1f7b658
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep 150cd8e3-633d-42af-91b2-faa0d1f7b658
│ xcp/volume/150cd8e3-633d-42af-91b2-faa0d1f7b658/metadata │ {"name_label": "OVBH-VPROD-RANCHERLB", "name_description": "", "is_a_snapshot": false, "snapshot_of": null, "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""} │
│ xcp/volume/150cd8e3-633d-42af-91b2-faa0d1f7b658/not-exists │ 0 │
│ xcp/volume/150cd8e3-633d-42af-91b2-faa0d1f7b658/volume-name │ xcp-volume-7fea7c97-3783-46b6-bcc6-cfd234b1f4a7 │
│ xcp/volume/ebdd16b6-5293-4c7d-ac37-b427fc3b368d/metadata │ {"name_label": "OVBH-VPROD-RANCHERLB", "name_description": "", "is_a_snapshot": true, "snapshot_of": "150cd8e3-633d-42af-91b2-faa0d1f7b658", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""} │
[13:12 ovbh-pprod-xen01 ~]# xe vdi-param-list uuid=150cd8e3-633d-42af-91b2-faa0d1f7b658
uuid ( RO) : 150cd8e3-633d-42af-91b2-faa0d1f7b658
name-label ( RW): OVBH-VPROD-RANCHERLB
name-description ( RW):
is-a-snapshot ( RO): false
snapshot-of ( RO): <not in database>
snapshots ( RO): ebdd16b6-5293-4c7d-ac37-b427fc3b368d
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): clone; snapshot
current-operations (SRO):
sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
sr-name-label ( RO): linstor-sr
vbd-uuids (SRO): 07b60986-8d19-aa23-f2ba-8d8ab96f93b4
crashdump-uuids (SRO):
virtual-size ( RO): 53687091200
physical-utilisation ( RO): 1108372480
location ( RO): 150cd8e3-633d-42af-91b2-faa0d1f7b658
type ( RO): User
sharable ( RO): false
read-only ( RO): false
storage-lock ( RO): false
managed ( RO): true
parent ( RO) [DEPRECATED]: <not in database>
missing ( RO): false
is-tools-iso ( RO): false
other-config (MRW):
xenstore-data (MRO):
sm-config (MRO): host_OpaqueRef:b3c659f5-efb3-4202-9268-b824e3eed523: RW; read-caching-reason-85838113-e4b8-4520-9f6d-8f3cf554c8f1: SR_NOT_SUPPORTED; read-caching-enabled-on-85838113-e4b8-4520-9f6d-8f3cf554c8f1: false; image-format: vhd; vdi_type: vhd; vhd-parent: 28d1e0e1-29e6-4734-90cd-5e394fe25af8
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
tags (SRW):
cbt-enabled ( RO): false
03 VDI: 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
│ xcp/volume/07f430b2-c7f1-41d8-bd9f-2fe67f4f458b/metadata │ {"name_label": "OVBH-VPROD-RANCHER03", "name_description": "", "is_a_snapshot": false, "snapshot_of": null, "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""} │
│ xcp/volume/07f430b2-c7f1-41d8-bd9f-2fe67f4f458b/not-exists │ 0 │
│ xcp/volume/07f430b2-c7f1-41d8-bd9f-2fe67f4f458b/volume-name │ xcp-volume-59213145-ddf0-4337-9ad7-c9bd74b905a5 │
│ xcp/volume/098cc02d-0d62-4f36-b86e-e85e4a2802e0/metadata │ {"name_label": "OVBH-VPROD-RANCHER03", "name_description": "", "is_a_snapshot": true, "snapshot_of": "07f430b2-c7f1-41d8-bd9f-2fe67f4f458b", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""} │
[13:10 ovbh-pprod-xen01 ~]# xe vdi-param-list uuid=07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
uuid ( RO) : 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
name-label ( RW): OVBH-VPROD-RANCHER03
name-description ( RW):
is-a-snapshot ( RO): false
snapshot-of ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
snapshots ( RO): 098cc02d-0d62-4f36-b86e-e85e4a2802e0
snapshot-time ( RO): 19700101T00:00:00Z
allowed-operations (SRO): clone; snapshot
current-operations (SRO):
sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
sr-name-label ( RO): linstor-sr
vbd-uuids (SRO): bdfc6a23-a89b-eb7e-7509-97e910e7ca79
crashdump-uuids (SRO):
virtual-size ( RO): 107374182400
physical-utilisation ( RO): 24027230208
location ( RO): 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b
type ( RO): User
sharable ( RO): false
read-only ( RO): false
storage-lock ( RO): false
managed ( RO): true
parent ( RO) [DEPRECATED]: <not in database>
missing ( RO): false
is-tools-iso ( RO): false
other-config (MRW):
xenstore-data (MRO):
sm-config (MRO): host_OpaqueRef:7f23da94-1933-41ad-8d9a-27dac34bc1a9: RW; read-caching-reason-c03e1a45-4c4c-46f5-a2a1-d8de2e22a866: SR_NOT_SUPPORTED; read-caching-enabled-on-c03e1a45-4c4c-46f5-a2a1-d8de2e22a866: false; vhd-parent: 59c33f05-2601-46dd-af09-9e86b3e166e3; image-format: vhd; vdi_type: vhd
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
tags (SRW):
cbt-enabled ( RO): false
HUH?
[13:14 ovbh-pprod-xen01 ~]# xe vdi-param-list uuid=1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
uuid ( RO) : 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
name-label ( RW): OVBH-VPROD-RANCHER01
name-description ( RW):
is-a-snapshot ( RO): false
snapshot-of ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
snapshots ( RO): 848ff799-092f-428b-90de-3a85517c7315; 09d26ac2-82a5-4208-8ebc-9d0df97665fc; 7884662d-967e-43b1-a789-e92c1c149403; d55536f8-906e-419e-a17a-318095312563; 55300191-34ec-4252-aa9f-3b2bdd5f98c7; 832e9bfd-899a-41e3-baa6-373175b4b0ec; 0a5aefe2-0f59-4447-b7ba-c8e9c4a97f67; aa0fdf8f-0a1f-4c1e-8b31-c36ea0343775; 653f53d3-1e8b-41e7-a997-41025bc268ef; 5111fe82-17b7-467c-851c-82ccb18d7bb1; 0eb9747b-02d3-4cd9-8dbb-d0a7d8a538e0; 357c608a-a805-48ab-8a96-d93fe1157304; 31ecf300-cc63-4845-8e7c-d191f4042135; 415196a5-de15-4a2a-912a-675f804af98e; d07a41d6-0aa2-49a5-a241-b755c6daa3d8; 450bddc9-946f-43c1-b9a3-5f50678ac249; c8a5ba4e-4140-4c4c-b272-9b44e4cd8dae; b741152c-ce78-4d00-8227-2031ba3778bc; cd158763-7f28-42ca-bddb-30ffcc89e18f; 5df27f53-4a98-41ba-b577-7468e3eb9557; bb36c707-d6a9-4811-b3dc-33c694b2faf8; ab4c71e6-ac41-4fc6-8435-546ab827c728; 156877f1-0701-484e-9c60-659841110d38; 1e949f7b-3f04-4fcc-bdc9-2052974aef34; a05390d4-5ef7-458d-b090-799215c49c6e; 8874451d-249f-4913-ad9c-fb8da214f663; 7520b2ae-a236-4065-8ec7-ac2cfa37a5a6; bd2b4490-3464-4650-8847-11a376393ae9; cbaf2499-489f-4b48-a017-e48fb73b4987; 6a1ff504-f448-420a-ba4b-8a05d8a3f2da; bc31eab4-7720-45fb-b6e8-23d529381221; 71c5c06b-a38a-4dc8-b4eb-8a3bfafdd271; a9c40305-9f7c-4af9-be56-b1dd4f26458c; bde8b13b-2f8c-454a-8bae-6f07fa907b9f; 6cfe58b0-3dd7-422d-924c-284d811afa97; 60ad964b-b5aa-4077-9ca0-6acb84cbea15; 03e6c2ec-c8bc-4bc3-b274-654e67e52ba7; c2ae7055-dbc8-4d74-8174-88e6b6c9dbb4; ad9b1572-a29d-42bb-842d-448e49bdbbb2; 78a67732-4889-4053-8a36-fb5d3cf1c0d3; 6ac57186-d928-4f34-8ce8-ae5f8b916107; 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0; 09ee300e-ac03-48f8-a6cc-bc33398fd75f; 2ce4926e-c8dc-4185-bbb4-ce887ef78b42; a4fc211d-9e68-4f7c-99e5-170f3786ad55; 2476d6b3-1b25-47ca-bc8d-869125eed779; d1f154d5-9bdf-4c01-897c-367863efe114; 5544ccc6-dec4-4528-91b4-302a6552146b; e95206e0-4940-48fc-8564-238f2f0f6440; 27ccf472-3fa9-4d23-bdb1-6ca70f570230; 2c5dec71-cd9a-4208-9e96-922b44544d09; 55c5e0a1-0af7-4b36-b5a6-57cff1185208; 99ad1143-0b88-46f0-ac0e-e5a72b69f327; 17200bad-cc3e-4b5d-9844-d1e56604d2a3; aff3122f-1693-4371-8d6f-5a2975aec85d; c7063926-088b-4cf0-8004-d75ea15653c5; cd0511c9-b803-48f2-afae-35fa51225c9e; 25937e64-4344-4772-8d8b-b5e9fee7cd81; 07f430b2-c7f1-41d8-bd9f-2fe67f4f458b; b37f9db8-b2bc-42d1-88b5-10207f9aeeb2; 5bf279c2-cff6-4eb6-8e70-376467d649f6; 70081652-9efd-4881-85b6-c39b0de6f90b; fd507e6c-c3e2-4305-81c8-f92da2703db9; c7fc59e4-8d90-4157-b526-bc5ecc549b66; 2418d9e7-c3a7-424a-af36-fd983d438163; 52e7b2de-84cb-4e1b-9a1f-4b898399a650; 5235fea0-4f20-4f19-859a-044b99ebce3c; 7f128b35-5e6f-4e32-b0c8-15059878c10c; a8dd3a48-8035-4a49-90f6-d91acdbbb032; 91f43c21-07ba-426f-a146-2d678e7de83b; 7bc9add5-cf5d-49c6-8022-78fb386a1e3b; 5488fdac-d7cf-4fc0-ac0f-535a36bf93d6; 1ee50d61-1426-4322-892b-e270cfd4c97e; dc740184-28b4-4db5-938b-837234f08290; eca38833-219e-439a-91ac-d3321772f71a; ea9d191b-0a2d-49d5-a046-7ebe483993fa; a3406a09-928a-4bf1-b558-53ab744dc41b; f887e9da-cda5-496e-9040-57aab22e510a; 5a6c334e-040e-4825-9ced-55ee2b17aac6
snapshot-time ( RO): 20260716T19:53:31Z
allowed-operations (SRO): copy; generate_config; clone; forget; update; resize; snapshot; destroy
current-operations (SRO):
sr-uuid ( RO): 479ca676-20a1-4051-7189-a4a9ca47e00d
sr-name-label ( RO): linstor-sr
vbd-uuids (SRO):
crashdump-uuids (SRO):
virtual-size ( RO): 107374182400
physical-utilisation ( RO): 21519360
location ( RO): 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
type ( RO): User
sharable ( RO): false
read-only ( RO): false
storage-lock ( RO): false
managed ( RO): true
parent ( RO) [DEPRECATED]: <not in database>
missing ( RO): false
is-tools-iso ( RO): false
other-config (MRW):
xenstore-data (MRO):
sm-config (MRO): vhd-parent: 848ff799-092f-428b-90de-3a85517c7315; image-format: vhd; vdi_type: vhd
on-boot ( RW): persist
allow-caching ( RW): false
metadata-latest ( RO): false
metadata-of-pool ( RO): <not in database>
tags (SRW):
cbt-enabled ( RO): false
wth
jonathon@jonathon-framework:~$ linstor --controllers=10.2.0.10,10.2.0.11,10.2.0.12,10.2.0.13,10.2.0.14 kv show xcp-sr-linstor_group_thin_device | grep 1516113c-9dbd-4e23-b76d-e1ec8cbed0c0
│ xcp/volume/1516113c-9dbd-4e23-b76d-e1ec8cbed0c0/metadata │ {"name_label": "OVBH-VPROD-RANCHER01", "name_description": "", "is_a_snapshot": true, "snapshot_of": "752611f2-bd1a-4009-9e6d-ce76b82144c4", "snapshot_time": "", "type": "user", "vdi_type": "vhd", "read_only": false, "metadata_of_pool": ""} │
│ xcp/volume/1516113c-9dbd-4e23-b76d-e1ec8cbed0c0/not-exists │ 0 │
│ xcp/volume/1516113c-9dbd-4e23-b76d-e1ec8cbed0c0/volume-name │ xcp-volume-cc57261d-6309-4a88-a672-f5eed4b0bc8d │
All hosts are restarted with 9.2.18 loaded now. Will see what happens next.
[15:56 ovbh-pprod-xen05 ~]# drbdadm --version
DRBDADM_BUILDTAG=GIT-hash:\ 71c8bcff6ea77a022b272a7eba649a774251bac4\ build\ by\ @buildsystem\,\ 2025-11-03\ 10:21:36
DRBDADM_API_VERSION=2
DRBD_KERNEL_VERSION_CODE=0x090212
DRBD_KERNEL_VERSION=9.2.18
DRBDADM_VERSION_CODE=0x092100
DRBDADM_VERSION=9.33.0
New crash, this time xen02. The time is different because I updated the velero backup schedules to not all be in the middle of the night. Also still happened with the openvas schedule removed, which makes sense.
Xen02 now has 9.2.18 loaded. Will restart other hosts now.
I also uploaded logs if those are desired.
@MajorP93 My mistake. No I have not rebooted the hosts yet. https://xcp-ng.org/forum/post/107164 I am aware that the reboot is needed for it to be picked up, I went ahead and installed from test so that if host crashes again it is there.
@Mathieu-L Yes, the hosts had been restarted after the updates were installed which included 9.2.16
@Mathieu-L https://xcp-ng.org/forum/post/106873 I had increased it to 16G and it still crashed.
For now I have upgraded all hosts to use --enablerepo=xcp-ng-linstor-testing to get kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3
That way it is there to be picked up if the host crashes again.
https://github.com/xcp-ng-rpms/drbd/blob/master/SPECS/kmod-drbd.spec
https://koji.xcp-ng.org/buildinfo?buildID=5747
[12:15 ovbh-pprod-xen04 ~]# yum upgrade --enablerepo=xcp-ng-linstor-testing
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
* xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
* xcp-ng-updates: mirrors.xcp-ng.org
grafana/signature | 810 B 00:00:00
grafana/signature | 3.0 kB 00:00:00 !!!
xcp-ng-linstor-testing/signature | 473 B 00:00:00
xcp-ng-linstor-testing/signature | 3.0 kB 00:00:00 !!!
zabbix | 3.0 kB 00:00:00
zabbix-non-supported | 2.9 kB 00:00:00
(1/3): grafana/primary_db | 879 kB 00:00:00
(2/3): xcp-ng-linstor-testing/primary_db | 2.4 kB 00:00:00
(3/3): zabbix/x86_64/primary_db | 114 kB 00:00:00
Resolving Dependencies
--> Running transaction check
---> Package kmod-drbd.x86_64 0:9.2.16-1.0.xcpng8.3 will be updated
---> Package kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3 will be an update
---> Package zabbix-agent.x86_64 0:7.0.27-release1.el7 will be updated
---> Package zabbix-agent.x86_64 0:7.0.28-release1.el7 will be an update
--> Finished Dependency Resolution
Dependencies Resolved
==============================================================================================================================================================================================================================================================
Package Arch Version Repository Size
==============================================================================================================================================================================================================================================================
Updating:
kmod-drbd x86_64 9.2.18-2.0.xcpng8.3 xcp-ng-linstor-testing 2.8 M
zabbix-agent x86_64 7.0.28-release1.el7 zabbix 660 k
Transaction Summary
==============================================================================================================================================================================================================================================================
Upgrade 2 Packages
Total download size: 3.4 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/2): zabbix-agent-7.0.28-release1.el7.x86_64.rpm | 660 kB 00:00:00
(2/2): kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64.rpm | 2.8 MB 00:00:01
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 2.8 MB/s | 3.4 MB 00:00:01
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Updating : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64 1/4
Updating : zabbix-agent-7.0.28-release1.el7.x86_64 2/4
Cleanup : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64 3/4
Cleanup : zabbix-agent-7.0.27-release1.el7.x86_64 4/4
Verifying : zabbix-agent-7.0.28-release1.el7.x86_64 1/4
Verifying : kmod-drbd-9.2.18-2.0.xcpng8.3.x86_64 2/4
Verifying : kmod-drbd-9.2.16-1.0.xcpng8.3.x86_64 3/4
Verifying : zabbix-agent-7.0.27-release1.el7.x86_64 4/4
Updated:
kmod-drbd.x86_64 0:9.2.18-2.0.xcpng8.3 zabbix-agent.x86_64 0:7.0.28-release1.el7
Complete!