Categories

  • All news regarding Xen and XCP-ng ecosystem

    144 Topics
    5k Posts
    rzrR
    @marcoi said: Applied the latest pushed updated to production, no major issues. Thank you for testing The only thing to note is my backup XCP config failed Sunday, (...) This morning the backup ran without issues. Glad it has finally to work, if problem occurs again feel free to ask about this feature in this subforum: https://xcp-ng.org/forum/category/21/backup

  • Everything related to the virtualization platform

    1k Topics
    15k Posts
    P
    @yannsionneau said: segment bus device function address of the device Hi all, thanks for your reponses! the below is done with the assumption that dom0 is just the host cli. If that is incorrect please let me know and I can update this info: dom0: lspci -vvv -nn -s 03:00.0: 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller]) Subsystem: ASRock Incorporation Device [1849:5413] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+ Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 120 Region 0: Memory at 7f000000000 (64-bit, prefetchable) [size=32G] Region 2: Memory at 7f800000000 (64-bit, prefetchable) [size=256M] Region 4: I/O ports at 3000 [size=256] Region 5: Memory at f2200000 (32-bit, non-prefetchable) [size=512K] Expansion ROM at f2280000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ DevCtl: Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed unknown, Width x16, ASPM L1, Exit Latency L0s unlimited, L1 <1us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+ ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed unknown, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+, LTR+, OBFF Not Supported DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis- Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS- Compliance De-emphasis: -6dB LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1- EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest- Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150 v2] Advanced Error Reporting UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol- UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO+ CmpltAbrt- UnxCmplt+ RxOF+ MalfTLP+ ECRC+ UnsupReq- ACSViol- CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+ CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr- AERCap: First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn- Capabilities: [200 v1] #15 Capabilities: [240 v1] Power Budgeting <?> Capabilities: [270 v1] #19 Capabilities: [2a0 v1] Access Control Services ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans- Capabilities: [2d0 v1] Process Address Space ID (PASID) PASIDCap: Exec+ Priv+, Max PASID Width: 10 PASIDCtl: Enable- Exec- Priv- Capabilities: [320 v1] Latency Tolerance Reporting Max snoop latency: 0ns Max no snoop latency: 0ns Capabilities: [410 v1] #26 Capabilities: [450 v1] #27 Capabilities: [500 v1] #2a Kernel driver in use: pciback VM lspci -vvv -nn -s 03:00.0: 00:08.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:7551] (rev c0) (prog-if 00 [VGA controller]) Subsystem: ASRock Incorporation Device [1849:5413] Physical Slot: 8 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx- Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx+ Latency: 0 Interrupt: pin A routed to IRQ 17 Region 0: Memory at 800000000 (64-bit, prefetchable) [size=32G] Region 2: Memory at c0000000 (64-bit, prefetchable) [size=256M] Region 4: I/O ports at c300 [size=256] Region 5: Memory at e0800000 (32-bit, non-prefetchable) [size=512K] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [64] Express (v2) Legacy Endpoint, MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ MaxPayload 128 bytes, MaxReadReq 512 bytes DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend- LnkCap: Port #0, Speed 32GT/s, Width x16, ASPM L1, Exit Latency L1 <1us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk- ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt- LnkSta: Speed 32GT/s, Width x16 TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt- DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+ 10BitTagComp+ 10BitTagReq+ OBFF Not Supported, ExtFmt+ EETLPPrefix+, MaxEETLPPrefixes 1 EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit- FRS- AtomicOpsCap: 32bit+ 64bit+ 128bitCAS- DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled, AtomicOpsCtl: ReqEn+ LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1- EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest- Retimer- 2Retimers- CrosslinkRes: unsupported Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Kernel driver in use: amdgpu Kernel modules: amdgpu dom0 xl dmesg output: xldmesg.txt dom0 dmesg output: dom0dmesg.txt vm dmesg output: vmdmesg.txt I believe this is all the output requested (I only provided the lspci output for a single GPU as they are identical GPUs , but happy to provide all the outputs if desired (will note I am not entirely sure I selected the exact GPU in both contexts due to the mismatched IDs in vm vs dom0, not sure that is an issue but may be worth calling out) edit: all output was taken with the VM running (but at idle). I did apply the grub changes to use the -d.gz file and iommu=debug as well so ideally that output should be included)

  • All Xen Orchestra related questions/reports

    3k Topics
    29k Posts
    acebmxerA
    Added local user support. Current sample config file. # Example xo-apply configuration. # Copy this into your own PRIVATE repo, edit, then: # export XO_URL=https://xo.example.lan XO_TOKEN=... # xo-apply diff config.yaml # xo-apply apply config.yaml # # Secrets never go in this file: use ${env:VAR_NAME} placeholders, # resolved from environment variables when the tool runs. # # A section that is ABSENT is unmanaged (xo-apply won't touch or report that # resource type). A present-but-empty section means "manage this type, none # should exist" (only deleted when you pass --prune). remotes: # NFS share on a NAS - name: nas-backups type: nfs host: 192.168.1.50 path: /export/xo-backups # port: 2049 # optional # mountOptions: vers=4 # optional mount(8) options # SMB / Windows share — host is "HOST\share" (single backslash in YAML # double-quoted strings must be written as \\) - name: windows-share type: smb host: "192.168.1.60\\backups" domain: WORKGROUP username: backup password: ${env:SMB_BACKUP_PASSWORD} # path: xo # optional subfolder inside the share # S3-compatible object storage (AWS, MinIO, Backblaze B2, ...) - name: offsite-s3 type: s3 host: s3.us-east-1.amazonaws.com path: my-bucket/xo-backups # bucket/directory accessKey: AKIAEXAMPLE secretKey: ${env:S3_SECRET_KEY} region: us-east-1 # protocol: http # for http-only endpoints (e.g. local MinIO) # Directory local to the XO VM (e.g. a mounted USB disk) - name: local-disk type: local path: /mnt/backup-disk backupJobs: # Delta (incremental) backup of every VM tagged "critical", every night - name: nightly-critical mode: delta vms: tag: critical remotes: [nas-backups, offsite-s3] settings: # optional global job settings, passed through to XO concurrency: 2 # timezone: America/New_York # maxExportRate: 104857600 # bytes/s # nRetriesVmBackupFailures: 2 # reportWhen: failure schedules: - name: nightly cron: "0 2 * * *" retention: 14 # backups kept on the remotes # snapshotRetention: 3 # snapshots kept on the pool # timezone: America/New_York # enabled: false # schedules are enabled by default # Weekly full backup of specific VMs, selected by name - name: weekly-full mode: full compression: zstd vms: names: [dc-01, mail-01] # ...or select by uuid: uuids: [770aa52a-fd42-8faf-f167-8c5c4a237cac] # ...or pass a raw XO smart-mode pattern for anything more complex: # raw: # type: VM # tags: # __or: [[prod]] remotes: [nas-backups] schedules: - name: weekly cron: "0 3 * * 0" retention: 8 # Disaster Recovery / Continuous Replication: instead of (or in addition to) # remotes, target one or more SRs. mode:full => DR, mode:delta => CR. - name: dr-critical mode: full # delta = Continuous Replication vms: tag: critical srs: [4991d4aa-ed84-599b-7d19-97f2f943a366] # target SR UUID(s) # remotes: [] # SR-only is fine; may be combined with remotes schedules: - name: hourly-dr cron: "0 * * * *" retention: 3 # replicas kept on the SR # Metadata backups: pool metadata and/or XO's own configuration. metadataBackups: - name: xo-config xoMetadata: true # back up XO's own config pools: [939ed551-fbd6-9868-52d8-d3997b7bf7da] # pool UUID(s) for pool metadata remotes: [nas-backups] schedules: - name: daily cron: "0 21 * * *" xoRetention: 7 # XO metadata backups kept poolRetention: 7 # pool metadata backups kept # Mirror backups: copy an existing remote's backups onto other remote(s), # e.g. push local backups offsite to S3. mirrorBackups: - name: offsite-mirror mode: full # full or delta, to match the source backups sourceRemote: nas-backups remotes: [offsite-s3] schedules: - name: nightly-mirror cron: "0 5 * * *" retention: 14 # Sequences: run backup schedules one after another. Each step names a job and # one of its schedules (from any job kind above, or already in XO). The sequence # has its own cron for when the whole chain runs. sequences: - name: nightly-then-metadata steps: - { job: nightly-critical, schedule: nightly } - { job: xo-config, schedule: daily } cron: "0 22 * * *" # enabled: false # sequences are enabled by default # timezone: America/New_York # Local users (XO's internal auth provider). Users created by an external auth # plugin (LDAP/SAML/GitHub) are NOT managed here and are never pruned. # # Passwords are write-only: XO never returns them, so a real password can't be # exported or diffed — but XO REQUIRES a password to create a user. So `export` # writes the placeholder `password: ChangeMe` for every user. # ⚠️ CHANGE these before importing into a real XO (or use a ${env:...} ref), # otherwise every new user is created with the password "ChangeMe". # The password is only used when a user is CREATED; for an existing user it is # ignored (change it in the XO UI). Only `permission` is compared for drift. users: - email: ops@example.com password: ChangeMe # required by XO; change before import permission: admin # none | read | write | admin (default: none) # Local groups. Members are referenced by email and resolved to ids at apply # time; each member must be a user defined above or already present in XO. groups: - name: operators users: [ops@example.com]

  • Our hyperconverged storage solution

    48 Topics
    759 Posts
    J
    Happens at around the same time. Which point to something on a schedule, and I think it has been found. TLDR; It looks like velero backup is hitting the hosts too fast and a possible synchronization bug causes a host to crash. I have attempted to slow down velero with the following helm chart value change. Will see if it happens again. configuration: # Lowers API call velocity to stagger volume Create/Delete cycles clientQPS: 3 clientBurst: 5 # Forces serial execution across volume structures, preventing concurrent lifecycle operations itemBlockWorkerCount: 1 I have all the logs in loki, would it be possible for someone to look at this and let me know what other logs are needed to attempt to identify some issue with drbd? Timeline of crash On Saturday, July 4, 2026, at 23:16:03 PDT, xcp-ng host ovbh-pprod-xen04 suffered a hard kernel panic. The crash was caused by an asynchronous race condition within the DRBD kernel module. It looks like this happens during the openvas velero backup. This crash has been happening at the exact same time of day when it crashes. So far this is the only thing I can find happening during this time. Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDT Stage 1: Backup Initialization & Storage Provisioning Burst Timestamp: 23:15:17 - 23:15:26 PDT Velero starts the openvas-daily-20260705061500 snapshot which has PVC volumes (gvmd-pvc, openvas-pvc, psql-pvc). The linstor-csi-controller issues volume creation calls. Linstor establishes metadata across the cluster nodes, provisioning device structures and setting up three-way replication mappings across xen01, xen02, and xen05. It then dynamically drops xen02 and redistributes the active target allocation mirror directly onto xen04 for some reason. 2026-07-04T23:15:17-07:00 {"time":"2026-07-05T02:15:17.684469063-04:00","stream":"stderr","message":"I0705 06:15:17.683982 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/CreateVolume\" ... name=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:17-07:00 2026-07-05 02:15:17.723 [grizzly-http-server-19] INFO LINSTOR/Controller/3de0de SYSTEM - Resource definition created pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:18-07:00 2026-07-05 02:15:18.811 [MainWorkerPool-1] INFO LINSTOR/Controller/00e49b SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen01' 2026-07-04T23:15:20-07:00 2026-07-05 02:15:20.983 [MainWorkerPool-2] INFO LINSTOR/Controller/00da44 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen02' 2026-07-04T23:15:22-07:00 2026-07-05 02:15:22.516 [MainWorkerPool-16] INFO LINSTOR/Controller/00dfdd SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-pprod-xen04' 2026-07-04T23:15:23-07:00 2026-07-05 02:15:23.294 [MainWorkerPool-12] INFO LINSTOR/Controller/00dcb8 SYSTEM - Resource deleted [ovbh-pprod-xen02]/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:15:26-07:00 2026-07-05 02:15:26.027 [MainWorkerPool-16] INFO LINSTOR/Controller/00be18 SYSTEM - ACR: Created resource 'pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1' on 'ovbh-vprod-k8s01-worker03.floatplane.com' Stage 2: Rapid Backup Job Completion & Teardown Execution Timestamp: 23:15:38 - 23:16:01 PDT The temporary target volume (openvas-pvc) completes its read stream sequence inside a brief 14-second operation due to a minuscule dataset size (5,540 bytes). The PersistentVolume state drops to Released. Velero instantly sends a container cleanup execution sequence down the line. The Linstor CSI controller catches the drop and issues an immediate DeleteResource directive to all replication satellites, commanding xen04 to disconnect the live disk configuration mapping. 2026-07-04T23:15:38-07:00 {"time":"2026-07-05T02:15:38.099681495-04:00","stream":"stdout","message":"time=\"2026-07-05T06:15:38Z\" level=info msg=\"Received event for data path ... reason: SuccessfulAttachVolume ...\""} 2026-07-04T23:15:52-07:00 {"time":"2026-07-05T02:15:52.97309346-04:00","stream":"stderr","message":"I0705 06:15:52.972697 69 controller.go:1316] \"shouldDelete is true\" PV=\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\""} 2026-07-04T23:15:56-07:00 {"time":"2026-07-05T02:15:56.223202404-04:00","stream":"stderr","message":"I0705 06:15:56.222906 1 reconciler.go:279] \"attacherDetacher.DetachVolume started\" node=\"ovbh-vprod-k8s01-worker03.floatplane.com\""} 2026-07-04T23:15:59-07:00 {"time":"2026-07-05T02:15:59.977217319-04:00","stream":"stderr","message":"I0705 06:15:59.976833 69 connection.go:264] \"GRPC call\" method=\"/csi.v1.Controller/DeleteVolume\" request=\"{\\\"volume_id\\\":\\\"pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\"}\""} 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.147 [grizzly-http-server-19] INFO LINSTOR/Controller/864b2a SYSTEM - Deleting resource ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 2026-07-04T23:16:00-07:00 2026-07-05 02:16:00.245 [MainWorkerPool-3] INFO LINSTOR/Controller/00e4cf SYSTEM - Toggle Disk on ovbh-pprod-xen01/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 removing disk 2026-07-04T23:16:01-07:00 2026-07-05 02:16:01.558 [grizzly-http-server-20] INFO LINSTOR/Controller/e89b52 SYSTEM - Deleting resource ovbh-pprod-xen04/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 Stage 3: Memory Teardown, GPF, and Hard Kernel Panic Timestamp: 23:16:03 - 23:16:05 PDT As xen04 unconfigures the storage resource footprint, the unconfigure worker frees up metadata allocation tree parameters. Simultaneously, the connection receiver thread executes an un-serialized bit-counting function scanning the replication tracking context. The receiver thread pulls a stale pointer reference, hitting unmapped virtual memory space and throwing a General Protection Fault. Two seconds later, while stepping into a secondary rescue routine to drop conflicting operations, it reads a dead memory address structure inside rb_first(), throwing a critical Kernel Paging Oops that completely crashes the host operating system context. 2026-07-04T23:16:03-07:00 general protection fault: 0000 [#1] SMP NOPTI 2026-07-04T23:16:03-07:00 CPU: 3 PID: 3722867 Comm: drbd_w_pvc-385f Tainted: G O 4.19.0+1 #1 2026-07-04T23:16:03-07:00 Hardware name: /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025 2026-07-04T23:16:03-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd] 2026-07-04T23:16:03-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48 2026-07-04T23:16:03-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016 2026-07-04T23:16:05-07:00 BUG: unable to handle kernel paging request at ffffc90047dcfdc0 2026-07-04T23:16:05-07:00 PGD 1f69c6067 P4D 1f69c6067 PUD 265d1f067 PMD 1a26fe067 PTE 0 2026-07-04T23:16:05-07:00 Oops: 0000 [#2] SMP NOPTI 2026-07-04T23:16:05-07:00 CPU: 3 PID: 3722891 Comm: drbd_r_pvc-385f Tainted: G D O 4.19.0+1 #1 2026-07-04T23:16:05-07:00 RIP: e030:rb_first+0xd/0x20 2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660 2026-07-04T23:16:05-07:00 Call Trace: 2026-07-04T23:16:05-07:00 drbd_cancel_conflicting_resync_requests+0x38/0x2d0 [drbd] 2026-07-04T23:16:05-07:00 ? conn_disconnect+0x19c/0xce0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_receiver+0x361/0x9a0 [drbd] 2026-07-04T23:16:05-07:00 ? drbd_thread_setup+0xd0/0x230 [drbd] Stage 4: Cascading Failures & Pool Master Dropout Timestamp: 23:16:05 - 23:21:50 PDT As xen04 drops dead instantly, surviving nodes encounter abrupt link terminations (BrokenPipe). The core etcd consensus cluster node on master01 encounters an extensive 2.92-second serialization freeze while attempting to log state, following a complete loss of response from the crashed host. At 23:21:50, the Pool Master node xen01 officially declares xen04 dead after exhausting its network PingAck heartbeat retry limits. 2026-07-04T23:16:05-07:00 {"time":"2026-07-05T02:16:05.872109Z","stream":"stderr","message":"{\"level\":\"warn\",\"ts\":\"2026-07-05T06:16:05.872109Z\",\"caller\":\"txn/util.go:93\",\"msg\":\"apply request took too long\",\"took\":\"2.926882374s\",\"expected-duration\":\"100ms\",\"prefix\":\"read-only range \",\"request\":\"key:\\\"/registry/persistentvolumes/pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1\\\" limit:1 \"}"} 2026-07-04T23:16:17-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown ) 2026-07-04T23:21:50-07:00 drbd xcp-volume-3232b7e9-10d4-45ec-84ca-ad598bf428c6 ovbh-pprod-xen04: PingAck did not arrive in time. Full velero details jonathon@jonathon-framework:~$ velero --kubeconfig k8s_configs/production.yaml describe backup openvas-daily-20260705061500 --details Name: openvas-daily-20260705061500 Namespace: velero Labels: app.kubernetes.io/name=openvas velero.io/schedule-name=openvas-daily velero.io/storage-location=default Annotations: objectset.rio.cattle.io/applied=H4sIAAAAAAAA/3yRTW+cMBCG/0r0HivYAgtL1uce+3FIlUuVw2APxY2xER6Qqoj/XpmGbltVFSdGM+/H4xfQZB95jjZ4KKzseA4nG96uJTI8W2+g8KAHNotjZBhZyJAQ1AvI+yAkNviYfkP3jbVEltNsw0mTiOOkZJNE099Tz3WXl13T5zUXRX4tLl1elW3dUNffN9ceWwZHHbv/yg0UByhUl67uGtO2bVFrXdK5KgpdtmwaalrD10tVnnXf10nU08hQCBP7lWJuyLrv+DmOE2n+VTwtx4l18o9HaYWyubvcvUkfMgiPkyPhtKOjffA0xSHIZztyWAQK52JEBuu1Wwybj4dJhPpyRMBTBis8fpp43gHeruvhL8o3IjRNp+el49mzcEws/iyG7Lf3O+Ln/yy/paKvyT+Eld/tZjIvfJs/BreMKfbrWMJMX/l90HtiKBjuaXGSmIiDQlsVKfy63x1cjv29vrM+yeBp27YfAQAA//+XI8ykegIAAA objectset.rio.cattle.io/id=5f8afe4b-1b5f-4e00-906b-21745abf859f velero.io/resource-timeout=10m0s velero.io/source-cluster-k8s-gitversion=v1.35.4+rke2r1 velero.io/source-cluster-k8s-major-version=1 velero.io/source-cluster-k8s-minor-version=35 Phase: Completed Namespaces: Included: openvas Excluded: <none> Resources: Included cluster-scoped: <none> Excluded cluster-scoped: volumesnapshotcontents.snapshot.storage.k8s.io Included namespace-scoped: * Excluded namespace-scoped: volumesnapshots.snapshot.storage.k8s.io Label selector: <none> Or label selector: <none> Storage Location: default Velero-Native Snapshot PVs: true Snapshot Move Data: true Data Mover: velero TTL: 720h0m0s CSISnapshotTimeout: 30m0s ItemOperationTimeout: 4h0m0s Hooks: <none> Backup Format Version: 1.1.0 Started: 2026-07-04 23:15:00 -0700 PDT Completed: 2026-07-04 23:17:33 -0700 PDT Expiration: 2026-08-03 23:15:00 -0700 PDT Total items to be backed up: 42 Items backed up: 42 Backup Item Operations: Operation for persistentvolumeclaims openvas/gvmd-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-v86h2 Phase: Completed Progress: 8896914780 of 8896914780 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:08 -0700 PDT Started: 2026-07-04 23:15:30 -0700 PDT Updated: 2026-07-04 23:17:11 -0700 PDT Operation for persistentvolumeclaims openvas/openvas-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-92dtr Phase: Completed Progress: 5540 of 5540 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:13 -0700 PDT Started: 2026-07-04 23:15:38 -0700 PDT Updated: 2026-07-04 23:16:54 -0700 PDT Operation for persistentvolumeclaims openvas/psql-pvc: Backup Item Action Plugin: velero.io/csi-pvc-backupper Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Items to Update: datauploads.velero.io velero/openvas-daily-20260705061500-jp7gd Phase: Completed Progress: 24267413354 of 24267413354 complete (Bytes) Progress description: Completed Created: 2026-07-04 23:15:18 -0700 PDT Started: 2026-07-04 23:15:42 -0700 PDT Updated: 2026-07-04 23:17:15 -0700 PDT Resource List: apiextensions.k8s.io/v1/CustomResourceDefinition: - certificaterequests.cert-manager.io - certificates.cert-manager.io - clusterserviceversions.operators.coreos.com apps/v1/Deployment: - openvas/openvas - openvas/test apps/v1/ReplicaSet: - openvas/openvas-64786df449 - openvas/openvas-69bb998d7c - openvas/openvas-6db7bcd6ff - openvas/openvas-6dd46f4f79 - openvas/openvas-6f8d45b667 - openvas/openvas-78dc6f74c6 - openvas/openvas-7c5654b9d6 - openvas/openvas-844c5f796d - openvas/openvas-86d9b75d9f - openvas/openvas-b97749f55 - openvas/openvas-d9d48c74 - openvas/test-5879777846 - openvas/test-5bd7f76f8c - openvas/test-6487578786 - openvas/test-6b467857f5 cert-manager.io/v1/Certificate: - openvas/openvas-ingress-cert-fp-pki cert-manager.io/v1/CertificateRequest: - openvas/openvas-ingress-cert-fp-pki-15 discovery.k8s.io/v1/EndpointSlice: - openvas/openvas-service-6zghf networking.k8s.io/v1/Ingress: - openvas/openvas operators.coreos.com/v1alpha1/ClusterServiceVersion: - openvas/node-healthcheck-operator.v0.11.0 - openvas/self-node-remediation.v0.12.1 v1/ConfigMap: - openvas/elasticsearch-es-transport-ca-internal - openvas/init-scripts-config - openvas/kube-root-ca.crt v1/Endpoints: - openvas/openvas-service v1/Namespace: - openvas v1/PersistentVolume: - pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 - pvc-23fb6277-0f6f-483c-a245-3198dbeff0b2 - pvc-a1974f4d-2923-4036-8187-43ed0d26e1c4 v1/PersistentVolumeClaim: - openvas/gvmd-pvc - openvas/openvas-pvc - openvas/psql-pvc v1/Pod: - openvas/openvas-78dc6f74c6-wcwp2 v1/Secret: - openvas/gvmd-secrets - openvas/openvas-ingress-tls-fp-pki v1/Service: - openvas/openvas-service v1/ServiceAccount: - openvas/default Backup Volumes: Velero-Native Snapshots: <none included> CSI Snapshots: openvas/gvmd-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.a1974f4d-2923-403c839c5 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 8896914780 openvas/openvas-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.105299a3-d117-45630a803 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 5540 openvas/psql-pvc: Data Movement: Operation ID: du-4d7f65aa-ec24-421d-8b30-9ab70a10172b.23fb6277-0f6f-4836bb7b8 Data Mover: velero Uploader Type: kopia Moved data Size (bytes): 24267413354 Pod Volume Backups: <none included> HooksAttempted: 0 HooksFailed: 0 [image: 1783370374694-f9797a22-d4cf-40ae-8da5-6a7d580ef137-image-resized.jpeg] linstor-pvc-velero-logs.txt xen04-07042026-panic.txt linstor-controller-logs.txt

  • Non-English speakers

    36 Topics
    116 Posts
    AtaxyaNetworkA
    @bvivi57 Hey, merci pour ce tuto ! Je vais comparer ce que tu as fait avec mes template que j'ai créer de mon coté. Si tu veux comparer, le repo est ici https://github.com/disruptivemindseu/xcpng-template-builder/tree/main