• XOSTOR hyperconvergence preview

    Pinned Moved
    458
    1
    6 Votes
    458 Posts
    749k Views
    J
    I have amazing news! After the upgrade to xcp-ng 8.3, I retested velero backup, and it all just works Completed Backup jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml backup describe grafana-test Name: grafana-test Namespace: velero Labels: objectset.rio.cattle.io/hash=c2b5f500ab5d9b8ffe14f2c70bf3742291df565c velero.io/storage-location=default Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/4SSQW/bPgzFvwvPtv9OajeJj/8N22HdBqxFL0MPlEQlWmTRkOhgQ5HvPsixE2yH7iji8ffIJ74CDu6ZYnIcoIMTeYpcOf7vtIICji4Y6OB/1MdxgAJ6EjQoCN0rYAgsKI5Dyk9WP0hLIqmi40qjiKfMcRlAq7pBY+py26qmbEi15a5p78vtaqe0oqbVVsO5AI+K/Ju4A6YDdKDXqrVtXaNqzU5traVVY9d6Uyt7t2nW693K2Pa+naABe4IO9hEtBiyFksClmgbUdN06a9NAOtvr5B4DDunA8uR64lGgg7u6rxMUYMji6OWZ/dhTeuIPaQ6os+gTFUA/tR8NmXd+TELxUfNA5hslHqOmBN13OF16ZwvNQShIqpZClYQj7qk6blPlGF5uzC/L3P+kvok7MB9z0OcCXPiLPLHmuLLWCfVfB4rTZ9/iaA5zHovNZz7R++k6JI50q89BXcuXYR5YT0DolkChABEPHWzW9cK+rPQx8jgsH/KQj+QT/frzXCdduc/Ca9u1Y7aaFvMu5Ang5Xz+HQAA//8X7Fu+/QIAAA objectset.rio.cattle.io/id=e104add0-85b4-4eb5-9456-819bcbe45cfc velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.33.4+rke2r1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=33 Phase: Completed Namespaces: Included: grafana Excluded: <none> Resources: Included cluster-scoped: <none> Excluded cluster-scoped: volumesnapshotcontents.snapshot.storage.k8s.io Included namespace-scoped: * Excluded namespace-scoped: volumesnapshots.snapshot.storage.k8s.io Label selector: <none> Or label selector: <none> Storage Location: default Velero-Native Snapshot PVs: true Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 30m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2025-10-15 15:29:52 -0700 PDT Completed: 2025-10-15 15:31:25 -0700 PDT Expiration: 2025-11-14 14:29:52 -0800 PST Total items to be backed up: 35 Items backed up: 35 Backup Item Operations: 1 of 1 completed successfully, 0 failed (specify --details for more information) Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: grafana/central-grafana: Data Movement: included, specify --details for more information Pod Volume Backups: <none included> HooksAttempted: 0 HooksFailed: 0 Completed Restore jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml restore describe restore-grafana-test --details Name: restore-grafana-test Namespace: velero Labels: objectset.rio.cattle.io/hash=252addb3ed156c52d9fa9b8c045b47a55d66c0af Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW7zIBBA7zJrO5/j35gzfE2rtsomymIM45jGBgTjbKLcvaKJm6qL7kDwnt7ABdDpHfmgrQEBZxrJ25W2/85rSOCkjQIBrxTYeoIEJmJUyAjiAmiMZWRtTYhb232Q5EC88tquJDKPFEU6GlpUG5UVZdpUdZ6WZZ+niOtNWtR1SypvqC8buCYwYkfjn7oBwwAC8ipHpbqC1LqqZZWrtse228isrLqywapSdS0z7KPU4EQgwN+mSI8eezSYMgWG22lwKOl7/MgERzJmdChPs9veDL9IGfSbQRcGy+96IjszCCiyCRLQRo6zIrVd5AHEfuHhkIBmmp4d+a/3e9Dl8LPoCZ3T5hg7FvQRcR8nxt6XL7sAgv1MCZztOE+01P23cvmnPYzaxNtwuF4/AwAA//8k6OwC/QEAAA objectset.rio.cattle.io/id=9ad8d034-7562-44f2-aa18-3669ed27ef47 Phase: Completed Total items to be restored: 33 Items restored: 33 Started: 2025-10-15 15:35:26 -0700 PDT Completed: 2025-10-15 15:36:34 -0700 PDT Warnings: Velero: <none> Cluster: <none> Namespaces: grafana-restore: could not restore, ConfigMap:elasticsearch-es-transport-ca-internal already exists. Warning: the in-cluster version is different than the backed-up version could not restore, ConfigMap:kube-root-ca.crt already exists. Warning: the in-cluster version is different than the backed-up version Backup: grafana-test Namespaces: Included: grafana Excluded: <none> Resources: Included: * Excluded: nodes, events, events.events.k8s.io, backups.velero.io, restores.velero.io, resticrepositories.velero.io, csinodes.storage.k8s.io, volumeattachments.storage.k8s.io, backuprepositories.velero.io Cluster-scoped: auto Namespace mappings: grafana=grafana-restore Label selector: <none> Or label selector: <none> Restore PVs: true CSI Snapshot Restores: grafana-restore/central-grafana: Data Movement: Operation ID: dd-ffa56e1c-9fd0-44b4-a8bb-8163f40a49e9.330b82fc-ca6a-423217ee5 Data Mover: velero Uploader Type: kopia Existing Resource Policy: <none> ItemOperationTimeout: 4h0m0s Preserve Service NodePorts: auto Restore Item Operations: Operation for persistentvolumeclaims grafana-restore/central-grafana: Restore Item Action Plugin: velero.io/csi-pvc-restorer Operation ID: dd-ffa56e1c-9fd0-44b4-a8bb-8163f40a49e9.330b82fc-ca6a-423217ee5 Phase: Completed Progress: 856284762 of 856284762 complete (Bytes) Progress description: Completed Created: 2025-10-15 15:35:28 -0700 PDT Started: 2025-10-15 15:36:06 -0700 PDT Updated: 2025-10-15 15:36:26 -0700 PDT HooksAttempted: 0 HooksFailed: 0 Resource List: apps/v1/Deployment: - grafana-restore/central-grafana(created) - grafana-restore/grafana-debug(created) apps/v1/ReplicaSet: - grafana-restore/central-grafana-5448b9f65(created) - grafana-restore/central-grafana-56887c6cb6(created) - grafana-restore/central-grafana-56ddd4f497(created) - grafana-restore/central-grafana-5f4757844b(created) - grafana-restore/central-grafana-5f69f86c85(created) - grafana-restore/central-grafana-64545dcdc(created) - grafana-restore/central-grafana-69c66c54d9(created) - grafana-restore/central-grafana-6c8d6f65b8(created) - grafana-restore/central-grafana-7b479f79ff(created) - grafana-restore/central-grafana-bc7d96cdd(created) - grafana-restore/central-grafana-cb88bd49c(created) - grafana-restore/grafana-debug-556845ff7b(created) - grafana-restore/grafana-debug-6fb594cb5f(created) - grafana-restore/grafana-debug-8f66bfbf6(created) discovery.k8s.io/v1/EndpointSlice: - grafana-restore/central-grafana-hkgd5(created) networking.k8s.io/v1/Ingress: - grafana-restore/central-grafana(created) rbac.authorization.k8s.io/v1/Role: - grafana-restore/central-grafana(created) rbac.authorization.k8s.io/v1/RoleBinding: - grafana-restore/central-grafana(created) v1/ConfigMap: - grafana-restore/central-grafana(created) - grafana-restore/elasticsearch-es-transport-ca-internal(failed) - grafana-restore/kube-root-ca.crt(failed) v1/Endpoints: - grafana-restore/central-grafana(created) v1/PersistentVolume: - pvc-e3f6578f-08b2-4e79-85f0-76bbf8985b55(skipped) v1/PersistentVolumeClaim: - grafana-restore/central-grafana(created) v1/Pod: - grafana-restore/central-grafana-cb88bd49c-fc5br(created) v1/Secret: - grafana-restore/fpinfra-net-cf-cert(created) - grafana-restore/grafana(created) v1/Service: - grafana-restore/central-grafana(created) v1/ServiceAccount: - grafana-restore/central-grafana(created) - grafana-restore/default(skipped) velero.io/v2alpha1/DataUpload: - velero/grafana-test-nw7zj(skipped) Image of working restore pod, with correct data in PV [image: 1760568537496-34d87db1-19ae-4348-8d4e-6599375d7634-image.png] Velero installed from helm: https://vmware-tanzu.github.io/helm-charts Version: velero:11.1.0 Values --- image: repository: velero/velero tag: v1.17.0 # Whether to deploy the restic daemonset. deployNodeAgent: true initContainers: - name: velero-plugin-for-aws image: velero/velero-plugin-for-aws:latest imagePullPolicy: IfNotPresent volumeMounts: - mountPath: /target name: plugins configuration: defaultItemOperationTimeout: 2h features: EnableCSI defaultSnapshotMoveData: true backupStorageLocation: - name: default provider: aws bucket: velero config: region: us-east-1 s3ForcePathStyle: true s3Url: https://s3.location # Destination VSL points to LINSTOR snapshot class volumeSnapshotLocation: - name: linstor provider: velero.io/csi config: snapshotClass: linstor-vsc credentials: useSecret: true existingSecret: velero-user metrics: enabled: true serviceMonitor: enabled: true prometheusRule: enabled: true # Additional labels to add to deployed PrometheusRule additionalLabels: {} # PrometheusRule namespace. Defaults to Velero namespace. # namespace: "" # Rules to be deployed spec: - alert: VeleroBackupPartialFailures annotations: message: Velero backup {{ $labels.schedule }} has {{ $value | humanizePercentage }} partialy failed backups. expr: |- velero_backup_partial_failure_total{schedule!=""} / velero_backup_attempt_total{schedule!=""} > 0.25 for: 15m labels: severity: warning - alert: VeleroBackupFailures annotations: message: Velero backup {{ $labels.schedule }} has {{ $value | humanizePercentage }} failed backups. expr: |- velero_backup_failure_total{schedule!=""} / velero_backup_attempt_total{schedule!=""} > 0.25 for: 15m labels: severity: warning Also create the following. apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: linstor-vsc labels: velero.io/csi-volumesnapshot-class: "true" driver: linstor.csi.linbit.com deletionPolicy: Delete We are using Piraeus operator to use xostor in k8s https://github.com/piraeusdatastore/piraeus-operator.git Version: v2.9.1 Values: --- operator: resources: requests: cpu: 250m memory: 500Mi limits: memory: 1Gi installCRDs: true imageConfigOverride: - base: quay.io/piraeusdatastore components: linstor-satellite: image: piraeus-server tag: v1.29.0 tls: certManagerIssuerRef: name: step-issuer kind: StepClusterIssuer group: certmanager.step.sm Then we just connect to the xostor cluster like external linstor controller.
  • XOSTOR 2 node Diskfull et 1 node Diskless

    3
    0 Votes
    3 Posts
    162 Views
    F
    @olivierlambert Thanks !
  • Multiple disks groups

    6
    0 Votes
    6 Posts
    2k Views
    henri9813H
    Hello, @DustinB The https://vates.tech/xostor/ says: The maximum size of any single Virtual Disk Image (VDI) will always be limited by the smallest disk in your cluster. But in this case, maybe it can be stored in the "2TB disks" ? Maybe others can answer, i didn't test it.
  • XOSTOR Global network disruption test

    1
    1
    1 Votes
    1 Posts
    165 Views
    No one has replied
  • Unable to add new node to pool using XOSTOR

    10
    0 Votes
    10 Posts
    755 Views
    henri9813H
    Hello, I tried on a new pool. a little different scenario since i don't create xostor for now, on my previous example, i tried to add a node as replacement of an existing one.. I just run the install script only on node 1. When i try make node2 join the pool, i reproduce the incompatible sm error i got previously. The things which is "bizarre", is i don't have the license issue i got on Xen-orchestra. ( maybe it was finally not related ? ) Here is the complete logs. pool.mergeInto { "sources": [ "17510fe0-db23-9414-f3df-2941bd34f8dc" ], "target": "cc91fcdc-c7a8-a44c-65b3-a76dced49252", "force": true } { "code": "POOL_JOINING_SM_FEATURES_INCOMPATIBLE", "params": [ "OpaqueRef:090b8da1-9654-066c-84f9-7ab15cb101fd", "" ], "call": { "duration": 1061, "method": "pool.join_force", "params": [ "* session id *", "<MASTER_IP>", "root", "* obfuscated *" ] }, "message": "POOL_JOINING_SM_FEATURES_INCOMPATIBLE(OpaqueRef:090b8da1-9654-066c-84f9-7ab15cb101fd, )", "name": "XapiError", "stack": "XapiError: POOL_JOINING_SM_FEATURES_INCOMPATIBLE(OpaqueRef:090b8da1-9654-066c-84f9-7ab15cb101fd, ) at Function.wrap (file:///etc/xen-orchestra/packages/xen-api/_XapiError.mjs:16:12) at file:///etc/xen-orchestra/packages/xen-api/transports/json-rpc.mjs:38:21 at runNextTicks (node:internal/process/task_queues:60:5) at processImmediate (node:internal/timers:454:9) at process.callbackTrampoline (node:internal/async_hooks:130:17)" }```
  • Backup fail whereas xostor cluster is "healthy"

    4
    1
    0 Votes
    4 Posts
    372 Views
    henri9813H
    Hello @ronan-a I will reproduce the case, i will re-destroy one hypervisor and retrigger the case. Thank you @ronan-a et @olivierlambert If you need me to tests some special case don't hesit, we have a pool dedicated for this
  • Recovery from lost node in HA

    3
    0 Votes
    3 Posts
    258 Views
    henri9813H
    @olivierlambert No, For once, i followed the installation step carefully ^^'
  • XOSTOR on 8.3?

    xostor xcp-ng 8.3
    35
    0 Votes
    35 Posts
    6k Views
    olivierlambertO
    Sorry I forgot to publish in here the news: https://xcp-ng.org/blog/2025/06/16/xcp-ng-8-3-is-now-lts/ Indeed, since June the 16th, XOSTOR is available on 8.3
  • Recovery from lost node

    Solved
    5
    0 Votes
    5 Posts
    514 Views
    olivierlambertO
    Excellent news, thanks!
  • Matching volume/resource/lvm on disk to VDI/VHD?

    3
    0 Votes
    3 Posts
    396 Views
    dthenotD
    @cmd Hello, It's described here in the documentation https://docs.xcp-ng.org/xostor/#map-linstor-resource-names-to-xapi-vdi-uuids It might be possible to add a parameter in the sm-config of the VDI to ease this link, I'll put a card in our backlog to see if it's doable.
  • Talos K8s Cluster with XOSTOR

    4
    0 Votes
    4 Posts
    747 Views
    T
    @nathanael-h Thanks for the feedback.
  • Adding a node to xostor

    3
    0 Votes
    3 Posts
    415 Views
    J
    @olivierlambert I did open a ticket but thought I would post here as well to see if anyone had insights. Thanks.
  • XOSTOR as shared storage for VDIs?

    4
    0 Votes
    4 Posts
    471 Views
    olivierlambertO
    Have you read the doc first? https://docs.xcp-ng.org/xostor/ This gives a nice overview on how it works
  • XOSTOR 8.3 controller crash with guest OSes shutting down filesystem

    8
    1
    0 Votes
    8 Posts
    809 Views
    D
    @ronan-a [...] 64 bytes from 172.27.18.161: icmp_seq=21668 ttl=64 time=0.805 ms 64 bytes from 172.27.18.161: icmp_seq=21669 ttl=64 time=0.737 ms 64 bytes from 172.27.18.161: icmp_seq=21670 ttl=64 time=0.750 ms 64 bytes from 172.27.18.161: icmp_seq=21671 ttl=64 time=0.780 ms 64 bytes from 172.27.18.161: icmp_seq=21672 ttl=64 time=0.774 ms 64 bytes from 172.27.18.161: icmp_seq=21673 ttl=64 time=0.737 ms 64 bytes from 172.27.18.161: icmp_seq=21674 ttl=64 time=0.773 ms 64 bytes from 172.27.18.161: icmp_seq=21675 ttl=64 time=0.835 ms 64 bytes from 172.27.18.161: icmp_seq=21676 ttl=64 time=0.755 ms 1004711/1004716 packets, 0% loss, min/avg/ewma/max = 0.712/1.033/0.775/195.781 ms I am attaching simple ping stats for last 11 days. I don't think we can blame the network
  • Support Status of XOSTOR

    2
    0 Votes
    2 Posts
    309 Views
    DanpD
    Hi, XOSTOR on XCP-ng 8.2.1 has been supported since it was released approx 9 months ago. XOSTOR on XCP-ng 8.3 is still in the beta phase, so not officially supported. Regards, Dan
  • Any negative about using a bonded vlan interface for xostor traffic?

    1
    0 Votes
    1 Posts
    305 Views
    No one has replied
  • XOSTOR from source

    8
    0 Votes
    8 Posts
    3k Views
    olivierlambertO
    Yes. Meaning we are sure it was correctly installed on supported hosts. This limits the possible outcomes if there's a problem (a bit like XOA vs XO sources, but we have like 10 years of feedback from XO sources, so we can do community support in here with a relative confidence)
  • XOSTOR and mdadm software RAID

    6
    0 Votes
    6 Posts
    1k Views
    J
    @OhSoNoob I've used XOSTOR on top of MDRAID and it seemed to work well for me during my testing. I ran tests of it on top of MD RAID 1, 5, and 10 (MDRAID's "RAID 10" which isn't really RAID 10) and had good luck with it. The XOSTOR is really adding a second layer of redundancy at that point, similar to MDRAID 5+1 builds so is almost overkill. Almost. Where I see the most benefit from XOSTOR on MDRAID would be on top of RAID 10 or RAID 0 arrays. Depending on the speed of your drives, you might get some benefit from the increased read speed (and read/write speed for RAID 0). In addition, RAID 10 would give you some additional redundancy so that losing a drive wouldn't mean the loss of that node for XOSTOR's purposes, possibly making recovery easier. The ability for some redundancy might also be useful for a stretched cluster or some other situation where your network links between XOSTOR nodes isn't as fast as it should be; The ability to recover at the RAID level might be much faster than recovering or rebuilding an entire node over a slow link. @ronan-a, I'm not sure if you remember, but the very first test of XOSTOR I ran, shortly after it was introduced,, were on top of RAID 10 arrays. I kept that test cluster alive and running until equipment failure (failed motherboards, nothing related to XOSTOR or MDRAID) forced me to scrap it. I had similar teething pains to others while XOSTOR was being developed and debugged during the test phase, but nothing related to running on top of MDRAID as far as I could tell.
  • How to manage XOSTOR SRs (add/remove)

    1
    0 Votes
    1 Posts
    389 Views
    No one has replied
  • XOSTOR Performance

    8
    0 Votes
    8 Posts
    2k Views
    olivierlambertO
    Not only but that's where it's more visible.