2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: conn( StandAlone -> Unconnected ) [connect]
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: Starting receiver thread (peer-node-id 2)
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: conn( Connecting -> Disconnecting ) [del-peer]
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: Terminating sender thread
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: Connection closed
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: conn( Disconnecting -> StandAlone ) [disconnected]
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: Terminating receiver thread
2026-07-04T23:16:00-07:00 drbd pvc-5c7b9c88-7f62-47fd-8847-017e2f29bfc1 ovbh-vprod-k8s01-worker01.floatplane.com: Terminating sender thread
2026-07-04T23:16:00-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1: Preparing remote state change 4065073604: 1->all disk( Detaching )
2026-07-04T23:16:00-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: Committing remote state change 4065073604 (primary_nodes=0)
2026-07-04T23:16:00-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1/0 drbd1149 ovbh-pprod-xen01: pdsk( UpToDate -> Detaching ) repl( SyncTarget -> Established ) [remote]
2026-07-04T23:16:00-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1/0 drbd1149 ovbh-pprod-xen05: updated UUIDs 51E248C780D0D80C:0000000000000000:0000000000000000:0000000000000000
2026-07-04T23:16:00-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1/0 drbd1149: clearing bitmap UUID and content (5702265 bits) for node 1 (ovbh-pprod-xen01)(slot 1)
2026-07-04T23:16:00-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1/0 drbd1149 ovbh-pprod-xen01: pdsk( Detaching -> Diskless ) [peer-state]
2026-07-04T23:16:00-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1/0 drbd1149: bitmap WRITE of 1219 pages took 20 ms
2026-07-04T23:16:01-07:00 drbd pvc-c3aa6ea5-ec9a-4160-ad50-bf8bddc32dd8: Preparing remote state change 784915458: 2->all role( Secondary )
2026-07-04T23:16:01-07:00 drbd pvc-c3aa6ea5-ec9a-4160-ad50-bf8bddc32dd8 ovbh-vprod-k8s01-worker02.floatplane.com: Committing remote state change 784915458 (primary_nodes=0)
2026-07-04T23:16:01-07:00 drbd pvc-c3aa6ea5-ec9a-4160-ad50-bf8bddc32dd8 ovbh-vprod-k8s01-worker02.floatplane.com: peer( Primary -> Secondary ) [remote]
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1: Preparing cluster-wide state change 527138171: 3->1 conn( Disconnecting )
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1: State change 527138171: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1: Committing cluster-wide state change 527138171 (0ms)
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: conn( Connected -> Disconnecting ) peer( Secondary -> Unknown ) [down]
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1/0 drbd1149 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off ) [down]
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1: Preparing remote state change 4226802184: 0->3 conn( Disconnecting )
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen05: Committing remote state change 4226802184 (primary_nodes=0)
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen05: conn( Connected -> TearDown ) peer( Secondary -> Unknown ) [remote]
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1/0 drbd1149 ovbh-pprod-xen05: repl( PausedSyncT -> Off ) resync-susp( peer -> no ) [remote]
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen05: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:01-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 general protection fault: 0000 [#1] SMP NOPTI
2026-07-04T23:16:03-07:00 CPU: 3 PID: 3722867 Comm: drbd_w_pvc-385f Tainted: G           O      4.19.0+1 #1
2026-07-04T23:16:03-07:00 Hardware name:  /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025
2026-07-04T23:16:03-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd]
2026-07-04T23:16:03-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48
2026-07-04T23:16:03-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016
2026-07-04T23:16:03-07:00 RAX: 00000000000003ff RBX: 00000000000080c0 RCX: 00000000000d03ff
2026-07-04T23:16:03-07:00 RDX: 00000000000d001f RSI: 0005088000000000 RDI: ffff88803c2fc000
2026-07-04T23:16:03-07:00 RBP: 00000000000d0020 R08: 0000000000000340 R09: ffff88821ae02c00
2026-07-04T23:16:03-07:00 R10: 00000000000000e0 R11: 0000000000000000 R12: ffff88821ae02c08
2026-07-04T23:16:03-07:00 R13: ffff88821ae00200 R14: 00000000000d03ff R15: 0000000000000000
2026-07-04T23:16:03-07:00 FS:  0000000000000000(0000) GS:ffff88826f0c0000(0000) knlGS:0000000000000000
2026-07-04T23:16:03-07:00 CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
2026-07-04T23:16:03-07:00 CR2: 00007f7d66baa168 CR3: 000000024f934000 CR4: 0000000000040660
2026-07-04T23:16:03-07:00 Call Trace:
2026-07-04T23:16:03-07:00 ? __switch_to_asm+0x40/0x70
2026-07-04T23:16:03-07:00 ? w_update_peers+0x4c7/0x630 [drbd]
2026-07-04T23:16:03-07:00 ? do_unqueued_peer_device_work+0x37/0x5a0 [drbd]
2026-07-04T23:16:03-07:00 ? drbd_worker+0xfd/0x410 [drbd]
2026-07-04T23:16:03-07:00 ? wait_woken+0x80/0x80
2026-07-04T23:16:03-07:00 ? drbd_thread_setup+0xd0/0x230 [drbd]
2026-07-04T23:16:03-07:00 ? kthread+0xf8/0x130
2026-07-04T23:16:03-07:00 ? w_bitmap_io+0x1f0/0x1f0 [drbd]
2026-07-04T23:16:03-07:00 ? kthread_bind+0x10/0x10
2026-07-04T23:16:03-07:00 ? ret_from_fork+0x22/0x40
2026-07-04T23:16:03-07:00 Modules linked in: dm_snapshot tun tcp_diag inet_diag drbd_transport_tcp(O) tls dm_thin_pool bcache crc64 dm_cache dm_persistent_data dm_bio_prison dm_bufio dm_writecache nvme_rdma nvme_fabrics nvmet_rdma nvmet rdma_cm iw_cm ib_cm ib_core handshake(O) drbd(O) lru_cache bnx2fc(O) cnic(O) uio fcoe libfcoe libfc scsi_transport_fc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 nf_conncount nf_nat 8021q garp mrp stp llc ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_multiport xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c dm_multipath iptable_filter sunrpc crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper sg cdc_ether usbnet mii dm_mod i2c_piix4 ipmi_si ipmi_devintf ipmi_msghandler ip_tables x_tables
2026-07-04T23:16:03-07:00 sr_mod cdrom hid_generic usbhid hid uas usb_storage raid1 md_mod ahci libahci mlx5_core(O) mlxfw(O) psample mlxdevm(O) xhci_pci nvme mlx_compat(O) devlink libata xhci_hcd nvme_core scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua scsi_mod ipv6 crc_ccitt
2026-07-04T23:16:03-07:00 ---[ end trace fdfd26d52fcc9451 ]---
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen03: Duplicate prepare for remote state change 4272871889
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: sock was shut down by peer
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929/0 drbd1019 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60/0 drbd1170 ovbh-pprod-xen03: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8/0 drbd1136 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8/0 drbd1202 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65/0 drbd1023 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: susp-io( no -> quorum )
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8/0 drbd1136 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: sock was shut down by peer
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8: Preparing remote state change 4174347395: 1->all empty
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60/0 drbd1170: disk( UpToDate -> Consistent ) quorum( yes -> no )
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8: Preparing remote state change 1325374950: 2->all empty
2026-07-04T23:16:03-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60/0 drbd1170 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen03: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:03-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec/0 drbd1017 ovbh-pprod-xen03: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf/0 drbd1147 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151/0 drbd1199 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e/0 drbd1200 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe/0 drbd1184 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7/0 drbd1181 ovbh-pprod-xen01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:03-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: Preparing cluster-wide state change 1273649670: 1->all empty
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65: Preparing remote state change 3112661015: 3->all empty
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:03-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen05: Duplicate prepare for remote state change 3112661015
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe: Preparing remote state change 2200325397: 2->all empty
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:03-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf: Preparing cluster-wide state change 233361753: 1->all empty
2026-07-04T23:16:03-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen02: Duplicate prepare for remote state change 2200325397
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151: Preparing remote state change 4272871889: 3->all empty
2026-07-04T23:16:03-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929: Preparing remote state change 1865792774: 1->all empty
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen03: Committing remote state change 4272871889 (primary_nodes=4)
2026-07-04T23:16:03-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:03-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151: Preparing cluster-wide state change 3769889763: 1->all empty
2026-07-04T23:16:03-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen05: Rejecting concurrent remote state change 4272871889 because of state change 3769889763
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: Retrying cluster-wide state change after 92ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen05: Committing remote state change 3112661015 (primary_nodes=10)
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: Preparing remote state change 1212301607: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65: Preparing cluster-wide state change 339318965: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7: Preparing remote state change 2352481183: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen01: Committing remote state change 2200325397 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151: State change 3769889763: primary_nodes=4, weak_nodes=FFFFFFFFFFFFFFF0
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e: Preparing remote state change 2178153575: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151: Committing cluster-wide state change 3769889763 (52ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe: Preparing cluster-wide state change 4145954673: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen01: Committing remote state change 4174347395 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen02: Duplicate prepare for remote state change 2178153575
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8: Preparing cluster-wide state change 748674649: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen02: Rejecting concurrent remote state change 2200325397 because of state change 4145954673
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen01: Aborting remote state change 1212301607
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: Preparing remote state change 158232203: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd]
2026-07-04T23:16:05-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48
2026-07-04T23:16:05-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016
2026-07-04T23:16:05-07:00 RAX: 00000000000003ff RBX: 00000000000080c0 RCX: 00000000000d03ff
2026-07-04T23:16:05-07:00 RDX: 00000000000d001f RSI: 0005088000000000 RDI: ffff88803c2fc000
2026-07-04T23:16:05-07:00 RBP: 00000000000d0020 R08: 0000000000000340 R09: ffff88821ae02c00
2026-07-04T23:16:05-07:00 R10: 00000000000000e0 R11: 0000000000000000 R12: ffff88821ae02c08
2026-07-04T23:16:05-07:00 R13: ffff88821ae00200 R14: 00000000000d03ff R15: 0000000000000000
2026-07-04T23:16:05-07:00 FS:  0000000000000000(0000) GS:ffff88826f0c0000(0000) knlGS:0000000000000000
2026-07-04T23:16:05-07:00 CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
2026-07-04T23:16:05-07:00 CR2: 00007f7d66baa168 CR3: 000000024f934000 CR4: 0000000000040660
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen03: Duplicate prepare for remote state change 1865792774
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe: State change 4145954673: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe: Committing cluster-wide state change 4145954673 (52ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen02: Committing remote state change 2178153575 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: Preparing remote state change 3077763496: 2->all empty
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-vtest-k8s01-worker03.floatplane.com: Aborting remote state change 3077763496
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen01: Committing remote state change 158232203 (primary_nodes=4)
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60/0 drbd1170: disk( Consistent -> Outdated ) [far-away]
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: Preparing remote state change 1024803395: 0->all empty
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen03: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen01: Committing remote state change 1865792774 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen03: Committing remote state change 2352481183 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen05: Rejecting concurrent remote state change 3911772948 because of state change 233361753
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen02: Rejecting concurrent remote state change 3911772948 because of state change 233361753
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: Rejecting concurrent remote state change 3077763496 because of state change 479927433
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec/0 drbd1017: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec/0 drbd1017 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf: Retrying cluster-wide state change after 28ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen02: Rejecting concurrent remote state change 3112661015 because of state change 339318965
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7: Preparing cluster-wide state change 3773831608: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8: State change 748674649: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8: Committing cluster-wide state change 748674649 (56ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen01: Committing remote state change 1024803395 (primary_nodes=4)
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929: Preparing cluster-wide state change 4092115343: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen03: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen03: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: Preparing cluster-wide state change 3858186813: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen02: Duplicate prepare for remote state change 1325374950
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e: Preparing cluster-wide state change 1992521308: 1->all empty
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929: State change 4092115343: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929: Committing cluster-wide state change 4092115343 (0ms)
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen02: Aborting remote state change 1325374950
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: Preparing remote state change 479927433: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8: Preparing remote state change 3982997967: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e: State change 1992521308: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e: Committing cluster-wide state change 1992521308 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen02: Committing remote state change 3982997967 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65: State change 339318965: primary_nodes=10, weak_nodes=FFFFFFFFFFFFFFE8
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65: Committing cluster-wide state change 339318965 (56ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8: Preparing cluster-wide state change 353764226: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7: State change 3773831608: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7: Committing cluster-wide state change 3773831608 (4ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8: State change 353764226: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8: Committing cluster-wide state change 353764226 (4ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-vtest-k8s01-worker03.floatplane.com: Committing remote state change 479927433 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: Preparing cluster-wide state change 1794356897: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: State change 3858186813: primary_nodes=4, weak_nodes=FFFFFFFFFFFFFFF2
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: Committing cluster-wide state change 3858186813 (4ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: Preparing cluster-wide state change 3181084749: 1->all empty
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: State change 1794356897: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
2026-07-04T23:16:05-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: Committing cluster-wide state change 1794356897 (4ms)
2026-07-04T23:16:05-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: State change 3181084749: primary_nodes=4, weak_nodes=FFFFFFFFFFFFFFF2
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60: Committing cluster-wide state change 3181084749 (0ms)
2026-07-04T23:16:05-07:00 BUG: unable to handle kernel paging request at ffffc90047dcfdc0
2026-07-04T23:16:05-07:00 PGD 1f69c6067 P4D 1f69c6067 PUD 265d1f067 PMD 1a26fe067 PTE 0
2026-07-04T23:16:05-07:00 Oops: 0000 [#2] SMP NOPTI
2026-07-04T23:16:05-07:00 CPU: 3 PID: 3722891 Comm: drbd_r_pvc-385f Tainted: G      D    O      4.19.0+1 #1
2026-07-04T23:16:05-07:00 Hardware name:  /S8056GME-HOV-B, BIOS V1.25 (0x80) 01/17/2025
2026-07-04T23:16:05-07:00 RIP: e030:rb_first+0xd/0x20
2026-07-04T23:16:05-07:00 Code: 89 e7 e8 76 f3 8f ff 66 90 5b 5d 89 d0 41 5c c3 90 90 90 90 90 90 90 90 90 90 90 90 90 48 8b 07 48 85 c0 75 05 eb 0e 48 89 d0 <48> 8b 50 10 48 85 d2 75 f4 f3 c3 31 c0 c3 0f 1f 44 00 00 48 8b 07
2026-07-04T23:16:05-07:00 RSP: e02b:ffffc90047fe7c20 EFLAGS: 00010082
2026-07-04T23:16:05-07:00 RAX: ffffc90047dcfdb0 RBX: ffffc90047fe7d40 RCX: 0000000000000001
2026-07-04T23:16:05-07:00 RDX: 0000000000000001 RSI: ffff88816b571170 RDI: ffff88816b570210
2026-07-04T23:16:05-07:00 RBP: ffff88816b570000 R08: 0000000000000000 R09: 0000000000000000
2026-07-04T23:16:05-07:00 R10: 0000000000007ff0 R11: 0000000000000000 R12: ffff88816b571000
2026-07-04T23:16:05-07:00 R13: ffff88816b570204 R14: 0000000000000000 R15: ffff88816b570118
2026-07-04T23:16:05-07:00 FS:  0000000000000000(0000) GS:ffff88826f0c0000(0000) knlGS:0000000000000000
2026-07-04T23:16:05-07:00 CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660
2026-07-04T23:16:05-07:00 Call Trace:
2026-07-04T23:16:05-07:00 drbd_cancel_conflicting_resync_requests+0x38/0x2d0 [drbd]
2026-07-04T23:16:05-07:00 ? drbd_process_rs_discards+0x2ca/0x520 [drbd]
2026-07-04T23:16:05-07:00 ? idr_get_next+0x43/0x90
2026-07-04T23:16:05-07:00 ? drain_resync_activity+0x1fe/0x8f1 [drbd]
2026-07-04T23:16:05-07:00 ? _raw_spin_lock_irqsave+0x15/0x40
2026-07-04T23:16:05-07:00 ? lock_timer_base+0x67/0x80
2026-07-04T23:16:05-07:00 ? try_to_del_timer_sync+0x4d/0x80
2026-07-04T23:16:05-07:00 ? dtt_free+0xd7/0x120 [drbd_transport_tcp]
2026-07-04T23:16:05-07:00 ? conn_disconnect+0x19c/0xce0 [drbd]
2026-07-04T23:16:05-07:00 ? drbd_receiver+0x361/0x9a0 [drbd]
2026-07-04T23:16:05-07:00 ? drbd_thread_setup+0xd0/0x230 [drbd]
2026-07-04T23:16:05-07:00 ? kthread+0xf8/0x130
2026-07-04T23:16:05-07:00 ? w_bitmap_io+0x1f0/0x1f0 [drbd]
2026-07-04T23:16:05-07:00 ? kthread_bind+0x10/0x10
2026-07-04T23:16:05-07:00 ? ret_from_fork+0x22/0x40
2026-07-04T23:16:05-07:00 Modules linked in: dm_snapshot tun tcp_diag inet_diag drbd_transport_tcp(O) tls dm_thin_pool bcache crc64 dm_cache dm_persistent_data dm_bio_prison dm_bufio dm_writecache nvme_rdma nvme_fabrics nvmet_rdma nvmet rdma_cm iw_cm ib_cm ib_core handshake(O) drbd(O) lru_cache bnx2fc(O) cnic(O) uio fcoe libfcoe libfc scsi_transport_fc nfnetlink_cttimeout nfnetlink openvswitch nsh nf_nat_ipv6 nf_nat_ipv4 nf_conncount nf_nat 8021q garp mrp stp llc ipt_REJECT nf_reject_ipv4 xt_tcpudp xt_multiport xt_conntrack nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c dm_multipath iptable_filter sunrpc crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcbc aesni_intel aes_x86_64 crypto_simd cryptd glue_helper sg cdc_ether usbnet mii dm_mod i2c_piix4 ipmi_si ipmi_devintf ipmi_msghandler ip_tables x_tables
2026-07-04T23:16:05-07:00 sr_mod cdrom hid_generic usbhid hid uas usb_storage raid1 md_mod ahci libahci mlx5_core(O) mlxfw(O) psample mlxdevm(O) xhci_pci nvme mlx_compat(O) devlink libata xhci_hcd nvme_core scsi_dh_rdac scsi_dh_hp_sw scsi_dh_emc scsi_dh_alua scsi_mod ipv6 crc_ccitt
2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0
2026-07-04T23:16:05-07:00 ---[ end trace fdfd26d52fcc9452 ]---
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b/0 drbd1183 ovbh-pprod-xen03: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-vtest-k8s01-worker03.floatplane.com: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-vtest-k8s01-worker03.floatplane.com: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec/0 drbd1017: disk( UpToDate -> Consistent )
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec/0 drbd1017 ovbh-vtest-k8s01-worker03.floatplane.com: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf: Preparing cluster-wide state change 1298072304: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf/0 drbd1147 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf: Retrying cluster-wide state change after 136ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b/0 drbd1183 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-vprod-k8s01-worker02.floatplane.com: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen01: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen01: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-vprod-k8s01-worker02.floatplane.com: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b/0 drbd1209: disk( UpToDate -> Consistent )
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6 ovbh-vtest-k8s01-worker04.floatplane.com: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88/0 drbd1122 ovbh-pprod-xen01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d/0 drbd1179 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b/0 drbd1209 ovbh-vprod-k8s01-worker02.floatplane.com: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6 ovbh-vtest-k8s01-worker04.floatplane.com: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen05: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen02: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466/0 drbd1231 ovbh-pprod-xen01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844/0 drbd1073 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd/0 drbd1002 ovbh-pprod-xen01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6/0 drbd1158: disk( UpToDate -> Consistent )
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8/0 drbd1136: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8/0 drbd1136 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6/0 drbd1158 ovbh-vtest-k8s01-worker04.floatplane.com: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf/0 drbd1147: disk( UpToDate -> Consistent ) quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844/0 drbd1073: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844/0 drbd1073 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf/0 drbd1147 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274/0 drbd1167 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen01: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88/0 drbd1122: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e/0 drbd1124 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen02: PingAck did not arrive in time.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88/0 drbd1122 ovbh-pprod-xen03: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed/0 drbd1133 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081/0 drbd1066 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 RIP: e030:drbd_bm_count_bits+0x223/0x300 [drbd]
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 Code: 24 08 01 49 39 ee 4c 8b 4c 24 18 44 8b 54 24 20 0f 83 5c ff ff ff 44 89 fb e9 69 fe ff ff 89 d8 44 01 d3 48 83 c5 20 c1 e8 05 <8b> 3c 86 f3 0f b8 c7 90 89 c7 49 01 ff 81 fb ff 7f 00 00 77 a3 48
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 RSP: e02b:ffffc90047dcfcb8 EFLAGS: 00010016
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e/0 drbd1124: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d/0 drbd1206 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 RAX: 00000000000003ff RBX: 00000000000080c0 RCX: 00000000000d03ff
2026-07-04T23:16:05-07:00 RDX: 00000000000d001f RSI: 0005088000000000 RDI: ffff88803c2fc000
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e/0 drbd1124 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 RBP: 00000000000d0020 R08: 0000000000000340 R09: ffff88821ae02c00
2026-07-04T23:16:05-07:00 R10: 00000000000000e0 R11: 0000000000000000 R12: ffff88821ae02c08
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25/0 drbd1025: disk( UpToDate -> Consistent )
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25/0 drbd1025 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b: Preparing cluster-wide state change 984942204: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 R13: ffff88821ae00200 R14: 00000000000d03ff R15: 0000000000000000
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25/0 drbd1025 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 FS:  0000000000000000(0000) GS:ffff88826f0c0000(0000) knlGS:0000000000000000
2026-07-04T23:16:05-07:00 CS:  e033 DS: 0000 ES: 0000 CR0: 0000000080050033
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen05: Rejecting concurrent remote state change 3244501293 because of state change 984942204
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 CR2: ffffc90047dcfdc0 CR3: 000000024fbb6000 CR4: 0000000000040660
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274/0 drbd1167: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274/0 drbd1167 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88: Preparing remote state change 359168528: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5/0 drbd1013 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen02: Rejecting concurrent remote state change 3244501293 because of state change 984942204
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081: Preparing remote state change 2852676590: 1->all empty
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0/0 drbd1102 ovbh-pprod-xen01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-vtest-k8s01-worker03.floatplane.com: Duplicate prepare for remote state change 359168528
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e/0 drbd1124 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632/0 drbd1126 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e/0 drbd1004 ovbh-pprod-xen01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen01: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081/0 drbd1066: disk( UpToDate -> Consistent ) quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed/0 drbd1133 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081/0 drbd1066 ovbh-pprod-xen01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d/0 drbd1088 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-vprod-k8s01-worker02.floatplane.com: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d: Preparing cluster-wide state change 2390071335: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274: Preparing remote state change 544062394: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844/0 drbd1073 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5/0 drbd1013 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25: Preparing cluster-wide state change 4281090338: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed/0 drbd1133: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b/0 drbd1221 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed/0 drbd1133 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b/0 drbd1148 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: Duplicate prepare for remote state change 544062394
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9/0 drbd1220 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632/0 drbd1126 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6: Preparing cluster-wide state change 3257760637: 0->all empty
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6: Committing cluster-wide state change 3257760637 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf: Preparing cluster-wide state change 3172068816: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6/0 drbd1158: disk( Consistent -> UpToDate ) [lost-peer]
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf: Committing cluster-wide state change 3172068816 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e: Preparing cluster-wide state change 1601852901: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e: Committing cluster-wide state change 1601852901 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632: Preparing remote state change 3032329728: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen02: Duplicate prepare for remote state change 3123537867
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b: Preparing remote state change 3123537867: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9: Preparing remote state change 2184272450: 3->all empty
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: Preparing cluster-wide state change 2550292950: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: Preparing cluster-wide state change 2573175200: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: Committing cluster-wide state change 2573175200 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b/0 drbd1221 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen01: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec/0 drbd1017: disk( Consistent -> UpToDate ) [lost-peer]
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466: Preparing cluster-wide state change 2515747668: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b: Preparing remote state change 3234871441: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d/0 drbd1179 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-vtest-k8s01-worker03.floatplane.com: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-vtest-k8s01-worker03.floatplane.com: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88/0 drbd1122: disk( UpToDate -> Consistent )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88/0 drbd1122 ovbh-vtest-k8s01-worker03.floatplane.com: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen05: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b/0 drbd1221: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b/0 drbd1221 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274/0 drbd1167 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566/0 drbd1020 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357/0 drbd1026 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf/0 drbd1222 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566: Preparing remote state change 3913212880: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen02: Rejecting concurrent remote state change 2238819964 because of state change 984942204
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf: Preparing remote state change 423443464: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b/0 drbd1183 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-vtest-k8s01-worker03.floatplane.com: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen02: Rejecting concurrent remote state change 2752720340 because of state change 984942204
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b: Retrying cluster-wide state change after 132ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566/0 drbd1020: disk( UpToDate -> Consistent ) quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566/0 drbd1020 ovbh-pprod-xen01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-vtest-k8s01-worker03.floatplane.com: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf/0 drbd1147: disk( Consistent -> UpToDate ) [lost-peer]
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-vtest-k8s01-worker03.floatplane.com: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf: Preparing remote state change 1800760556: 3->all empty
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen05: Duplicate prepare for remote state change 1800760556
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: Preparing remote state change 1542393208: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen03: Aborting local state change 4281090338 to yield to remote state change 2260212268.
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa/0 drbd1130 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5: Preparing cluster-wide state change 583650861: 1->all empty
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen01: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen02: Duplicate prepare for remote state change 3032329728
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25: Retrying cluster-wide state change after 96ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d: Preparing cluster-wide state change 2384981514: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25: Preparing remote state change 2260212268: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466/0 drbd1231 ovbh-pprod-xen05: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466: Retrying cluster-wide state change after 156ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 tcp:ovbh-pprod-xen02: dtt_send_page: size=52 len=52 sent=-32
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen03: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466/0 drbd1231: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25/0 drbd1025: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466/0 drbd1231 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25/0 drbd1025 ovbh-pprod-xen03: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b/0 drbd1183: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b/0 drbd1183 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0 ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0 ovbh-pprod-xen05: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-vprod-k8s01-worker02.floatplane.com: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0/0 drbd1037 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen01: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d/0 drbd1088: disk( UpToDate -> Consistent )
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d/0 drbd1088 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd: Preparing remote state change 943495428: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen05: Duplicate prepare for remote state change 1383038850
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen03: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen03: Rejecting concurrent remote state change 2230631626 because of state change 2384981514
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0: Preparing remote state change 3914501981: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen03: Rejecting concurrent remote state change 2523906456 because of state change 2384981514
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d: Retrying cluster-wide state change after 168ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen02: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd/0 drbd1002: disk( UpToDate -> Consistent ) quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e/0 drbd1004: disk( UpToDate -> Consistent ) quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd/0 drbd1002 ovbh-pprod-xen03: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e/0 drbd1004 ovbh-pprod-xen02: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e: Preparing remote state change 1063731606: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen02: Rejecting concurrent remote state change 1348302853 because of state change 2550292950
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3/0 drbd1215 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen02: Rejecting concurrent remote state change 2092728745 because of state change 2550292950
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen03: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Preparing cluster-wide state change 2807295764: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e: Preparing remote state change 231439285: 2->all empty
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen01: Rejecting concurrent remote state change 1348302853 because of state change 2550292950
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen01: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: Duplicate prepare for remote state change 231439285
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen01: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b/0 drbd1209 ovbh-pprod-xen01: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen02: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen05: Rejecting concurrent remote state change 4116130682 because of state change 2807295764
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen03: Duplicate prepare for remote state change 1368865886
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d: Preparing remote state change 2258252419: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen03: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3: Preparing remote state change 1383038850: 3->all empty
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0: Preparing cluster-wide state change 1799113491: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen03: Duplicate prepare for remote state change 3948823408
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen02: Committing remote state change 3123537867 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen03: Aborting remote state change 2258252419
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466: Preparing remote state change 1204541303: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d: Preparing remote state change 643196917: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: conn( Connected -> BrokenPipe ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5/0 drbd1166 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen01: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen05: Duplicate prepare for remote state change 1204541303
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen01: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: Retrying cluster-wide state change after 56ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen02: Duplicate prepare for remote state change 3961059858
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen05: Rejecting concurrent remote state change 2258252419 because of state change 643196917
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357: Preparing remote state change 711872880: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen03: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen03: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357/0 drbd1026: disk( UpToDate -> Consistent )
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357/0 drbd1026 ovbh-pprod-xen03: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen05: sock was shut down by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen02: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de ovbh-pprod-xen02: Duplicate prepare for remote state change 1382681832
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed: Preparing remote state change 1380170656: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: Duplicate prepare for remote state change 1380170656
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen02: Rejecting concurrent remote state change 1288406529 because of state change 2390071335
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen05: Rejecting concurrent remote state change 1288406529 because of state change 2390071335
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen01: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-vtest-k8s01-worker03.floatplane.com: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen02: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen02: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Retrying cluster-wide state change after 20ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6 ovbh-vtest-k8s01-worker04.floatplane.com: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-vtest-k8s01-worker03.floatplane.com: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen01: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6 ovbh-vtest-k8s01-worker04.floatplane.com: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0 ovbh-pprod-xen05: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen05: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen03: Aborting remote state change 3948823408
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen02: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen02: Duplicate prepare for remote state change 423443464
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen03: Committing remote state change 643196917 (primary_nodes=2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen02: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d/0 drbd1179 ovbh-pprod-xen05: pdsk( DUnknown -> Outdated ) [far-away]
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d: Preparing remote state change 2421588810: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen03: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen02: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen03: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-vprod-k8s01-worker10.floatplane.com: Aborting local state change 1799113491 to yield to remote state change 736980783.
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen02: Committing remote state change 1383038850 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen02: Committing remote state change 423443464 (primary_nodes=1)
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-vprod-k8s01-worker01.floatplane.com: sock was reset by peer
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-vprod-k8s01-worker01.floatplane.com: conn( Connected -> BrokenPipe ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa/0 drbd1130: disk( UpToDate -> Consistent )
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa/0 drbd1130 ovbh-vprod-k8s01-worker01.floatplane.com: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844: Preparing remote state change 3961059858: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen03: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0: Retrying cluster-wide state change after 244ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf: Preparing cluster-wide state change 868145251: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen03: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen03: Committing remote state change 2421588810 (primary_nodes=2)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen05: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5: Preparing remote state change 3948823408: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8: Preparing cluster-wide state change 616466881: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8: Committing cluster-wide state change 616466881 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen02: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen02: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen01: Rejecting concurrent remote state change 423443464 because of state change 868145251
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5: Preparing cluster-wide state change 2152966367: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-vprod-k8s01-worker01.floatplane.com: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen03: Duplicate prepare for remote state change 2184272450
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen01: Starting sender thread (peer-node-id 1)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen02: Aborting remote state change 3032329728
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632: Preparing remote state change 3191407870: 3->all empty
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-vprod-k8s01-worker01.floatplane.com: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-vprod-k8s01-worker02.floatplane.com: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5: Retrying cluster-wide state change after 188ms
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-vprod-k8s01-worker02.floatplane.com: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen02: Duplicate prepare for remote state change 711872880
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0 ovbh-vprod-k8s01-worker02.floatplane.com: Duplicate prepare for remote state change 3914501981
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274: Two-phase commit 544062394 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen05: Starting sender thread (peer-node-id 2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274: Preparing cluster-wide state change 2118519810: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274: Committing cluster-wide state change 2118519810 (0ms)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Preparing remote state change 4116130682: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d: Retrying cluster-wide state change after 16ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d: Preparing cluster-wide state change 3411365495: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d: Retrying cluster-wide state change after 24ms
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5: State change 2152966367: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFF0
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5: Committing cluster-wide state change 2152966367 (4ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen02: Aborting remote state change 711872880
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357: Preparing remote state change 84285060: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen02: Committing remote state change 3191407870 (primary_nodes=1)
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632: Preparing remote state change 4026443164: 2->all empty
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen03: Aborting remote state change 4116130682
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Preparing remote state change 951392088: 0->all empty
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: Preparing remote state change 792877273: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen02: Committing remote state change 4026443164 (primary_nodes=1)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen03: Committing remote state change 951392088 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa/0 drbd1130: disk( Consistent -> Outdated ) [far-away]
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf ovbh-pprod-xen05: Duplicate prepare for remote state change 1563007911
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de/0 drbd1053 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18/0 drbd1109 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3: Preparing cluster-wide state change 651197539: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5: Preparing remote state change 2612856092: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: Starting sender thread (peer-node-id 4)
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de ovbh-pprod-xen02: Committing remote state change 1382681832 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de: Preparing remote state change 1382681832: 3->all empty
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0: susp-io( no -> quorum )
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen05: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0/0 drbd1102: quorum( yes -> no )
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0/0 drbd1102 ovbh-pprod-xen05: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf: State change 868145251: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFF0
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf: Committing cluster-wide state change 868145251 (4ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e: Two-phase commit 1063731606 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566: Two-phase commit 3913212880 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd: Two-phase commit 943495428 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed: Two-phase commit 1380170656 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b: Two-phase commit 3234871441 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466: Two-phase commit 1204541303 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0: Preparing remote state change 736980783: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-vprod-k8s01-worker10.floatplane.com: Rejecting concurrent remote state change 3010595847 because of state change 736980783
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec: Two-phase commit 1542393208 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf: Two-phase commit 1800760556 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen02: Aborting remote state change 2612856092
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen05: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5: Preparing remote state change 2300784824: 3->all empty
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-vprod-k8s01-worker03.floatplane.com: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d: Preparing remote state change 3511170193: 3->all empty
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-vprod-k8s01-worker03.floatplane.com: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18/0 drbd1109: disk( UpToDate -> Consistent )
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18/0 drbd1109 ovbh-vprod-k8s01-worker03.floatplane.com: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18: Preparing remote state change 1368865886: 3->all empty
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-vprod-k8s01-worker03.floatplane.com: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-vprod-k8s01-worker03.floatplane.com: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de: Preparing cluster-wide state change 2288081081: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen05: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen05: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen05: Starting sender thread (peer-node-id 0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf ovbh-pprod-xen01: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf/0 drbd1236 ovbh-pprod-xen01: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen02: Committing remote state change 2300784824 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen03: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf ovbh-pprod-xen05: Committing remote state change 1563007911 (primary_nodes=4)
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5: Preparing remote state change 2683524727: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf: Preparing remote state change 1563007911: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen02: Committing remote state change 2683524727 (primary_nodes=0)
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen03: Committing remote state change 3511170193 (primary_nodes=2)
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d/0 drbd1179 ovbh-pprod-xen01: pdsk( DUnknown -> Outdated ) [far-away]
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf ovbh-pprod-xen01: Starting sender thread (peer-node-id 3)
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566: Preparing cluster-wide state change 3716160062: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566: Committing cluster-wide state change 3716160062 (0ms)
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-vprod-k8s01-worker10.floatplane.com: Rejecting concurrent remote state change 3485467398 because of state change 736980783
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566/0 drbd1020: disk( Consistent -> UpToDate ) [lost-peer]
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d/0 drbd1088: Setting exposed data uuid: DC86795BE2DE0F6E
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88: Two-phase commit 359168528 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25: Two-phase commit 2260212268 timeout
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Preparing remote state change 2836873757: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de: State change 2288081081: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de: Committing cluster-wide state change 2288081081 (4ms)
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0 ovbh-pprod-xen03: Committing remote state change 3914501981 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0: Preparing cluster-wide state change 3271881166: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen02: Committing remote state change 792877273 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-vtest-k8s01-worker03.floatplane.com: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b/0 drbd1209: disk( Consistent -> Outdated ) [far-away]
2026-07-04T23:16:05-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081: Two-phase commit 2852676590 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081: Preparing cluster-wide state change 4281319460: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081: Committing cluster-wide state change 4281319460 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081/0 drbd1066: disk( Consistent -> UpToDate ) [lost-peer]
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen03: Committing remote state change 2836873757 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen01: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0 ovbh-vprod-k8s01-worker02.floatplane.com: Ignoring P_TWOPC_COMMIT packet 3914501981 current processing state change 3271881166
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen03: Aborting remote state change 1368865886
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18: Preparing remote state change 3711183657: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen02: meta connection shut down by peer.
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen03: Committing remote state change 3711183657 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3: State change 651197539: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18/0 drbd1109: disk( Consistent -> Outdated ) [far-away]
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3: Committing cluster-wide state change 651197539 (12ms)
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18: Preparing cluster-wide state change 3317236624: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d: Preparing remote state change 1177450837: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen03: Committing remote state change 2184272450 (primary_nodes=10)
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9: Preparing cluster-wide state change 3672480419: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen05: Rejecting concurrent remote state change 2184272450 because of state change 3672480419
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen02: Committing remote state change 84285060 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357/0 drbd1026: disk( Consistent -> Outdated ) [far-away]
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357/0 drbd1026: Setting exposed data uuid: 248A3D640A413252
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357: Preparing cluster-wide state change 3784323086: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen03: Committing remote state change 1177450837 (primary_nodes=4)
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d/0 drbd1088: disk( Consistent -> Outdated ) [far-away]
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9: State change 3672480419: primary_nodes=10, weak_nodes=FFFFFFFFFFFFFFE1
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9: Committing cluster-wide state change 3672480419 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357: State change 3784323086: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357: Committing cluster-wide state change 3784323086 (4ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357: Preparing cluster-wide state change 2603835256: 2->all empty
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0: State change 3271881166: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF0
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0: Committing cluster-wide state change 3271881166 (8ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357: State change 2603835256: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357: Committing cluster-wide state change 2603835256 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d: Preparing cluster-wide state change 166000517: 1->all empty
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Preparing cluster-wide state change 145301956: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d: State change 166000517: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d: Committing cluster-wide state change 166000517 (0ms)
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88: Preparing cluster-wide state change 2721144373: 0->all empty
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88: Committing cluster-wide state change 2721144373 (0ms)
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88/0 drbd1122: disk( Consistent -> UpToDate ) [lost-peer]
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e: Preparing cluster-wide state change 813053349: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e: Committing cluster-wide state change 813053349 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e/0 drbd1004: disk( Consistent -> UpToDate ) [lost-peer]
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf: Preparing cluster-wide state change 685834659: 1->all empty
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: State change 145301956: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf: State change 685834659: primary_nodes=4, weak_nodes=FFFFFFFFFFFFFFF0
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Committing cluster-wide state change 145301956 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf: Committing cluster-wide state change 685834659 (0ms)
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Preparing cluster-wide state change 4228200502: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd: Preparing cluster-wide state change 534360819: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd: Committing cluster-wide state change 534360819 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd/0 drbd1002: disk( Consistent -> UpToDate ) [lost-peer]
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632: Preparing cluster-wide state change 690689838: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b: Preparing cluster-wide state change 2701816609: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b: Committing cluster-wide state change 2701816609 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b: Preparing cluster-wide state change 4061595294: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632: State change 690689838: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFF0
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632: Committing cluster-wide state change 690689838 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b: State change 4061595294: primary_nodes=0, weak_nodes=0
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b: Committing cluster-wide state change 4061595294 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen03: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18: State change 3317236624: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18: Committing cluster-wide state change 3317236624 (12ms)
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: State change 4228200502: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Committing cluster-wide state change 4228200502 (4ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d: Preparing cluster-wide state change 460388445: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d: State change 460388445: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFFC
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d: Committing cluster-wide state change 460388445 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e: Two-phase commit 231439285 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844: Two-phase commit 3961059858 timeout
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844: Preparing cluster-wide state change 1355297105: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844: Committing cluster-wide state change 1355297105 (0ms)
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: Preparing remote state change 3534450881: 1->all empty
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen02: Committing remote state change 3534450881 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6 ovbh-vtest-k8s01-worker04.floatplane.com: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-8f6fac6d-de74-46be-8a38-b6ae09851bd6 ovbh-vtest-k8s01-worker04.floatplane.com: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: conn( Disconnecting -> StandAlone ) [disconnected]
2026-07-04T23:16:05-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: Terminating receiver thread
2026-07-04T23:16:05-07:00 drbd pvc-385f674a-3dcd-4cfb-8f4d-e4a7157becd1 ovbh-pprod-xen01: Terminating sender thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: conn( BrokenPipe -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-527b13f8-da1f-40e9-a62e-db9d075a7bfe ovbh-pprod-xen05: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d/0 drbd1179: new current UUID: 451AF57A717A7047 weak: FFFFFFFFFFFFFFFC
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-bb741dd9-b326-483e-a576-96d6644bb2de ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Preparing cluster-wide state change 3651973299: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-f3f31e26-bb77-45cc-8b9b-617dc45eeacf ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-6762c368-1c6b-425d-9337-0a384c2b29bf ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: Preparing cluster-wide state change 3615002424: 0->all empty
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: State change 3615002424: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF1
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: Committing cluster-wide state change 3615002424 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: Preparing cluster-wide state change 227217316: 0->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen03: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-670e24d0-c66f-40b3-b0c6-69dda19c8081 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-40798a48-e043-4eda-9763-998accec37d0 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: State change 227217316: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF1
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b: Committing cluster-wide state change 227217316 (4ms)
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18: Preparing remote state change 1802092655: 3->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-vprod-k8s01-worker01.floatplane.com: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-vprod-k8s01-worker01.floatplane.com: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-vtest-k8s01-worker03.floatplane.com: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-vtest-k8s01-worker03.floatplane.com: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen03: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-2653486a-dcb1-4d0b-a8fd-e4fd2cb462dd ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-df7b043a-01f7-413a-b58e-797f59a94a9e ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen03: Committing remote state change 1802092655 (primary_nodes=8)
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18: Preparing cluster-wide state change 996283122: 2->all empty
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: State change 3651973299: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa: Committing cluster-wide state change 3651973299 (20ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-baef811c-f9b8-4096-92af-1d227b62e22d ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-vprod-k8s01-worker03.floatplane.com: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-vprod-k8s01-worker03.floatplane.com: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-7a4ed2b6-9d33-4d69-9bbe-55fc3c67f7fa ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: conn( BrokenPipe -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-7922b0ba-77a7-40cb-b60c-160d465b999d ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18: State change 996283122: primary_nodes=8, weak_nodes=FFFFFFFFFFFFFFF4
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18: Committing cluster-wide state change 996283122 (16ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-50fd2c25-709e-49c2-927b-fd46348dd151 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-0a9d9b15-922c-4fef-a485-ce856aa22929 ovbh-pprod-xen02: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-452bc588-ea9a-460a-ac96-9d1f0f17821e ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-612fc6b7-a0b5-49ba-8e8d-9e466ad24fe7 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25: Preparing cluster-wide state change 1354218262: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25: Committing cluster-wide state change 1354218262 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25/0 drbd1025: disk( Consistent -> UpToDate ) [lost-peer]
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-105299a3-d117-4562-a2e7-8b6be53cfde0 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-553520c5-3044-4595-9fb6-1a0a8c507c18 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-vprod-k8s01-worker02.floatplane.com: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-df4b046a-597b-42d4-9b2e-1bedff11103b ovbh-vprod-k8s01-worker02.floatplane.com: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-vtest-k8s01-worker03.floatplane.com: Connection closed
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-vtest-k8s01-worker03.floatplane.com: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen01: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen05: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen03: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: conn( BrokenPipe -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff8dd941-c260-4036-872a-38f8e49317d8 ovbh-pprod-xen02: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-ff1b8be1-0555-4448-975f-36d2cd24da65 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen05: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: conn( BrokenPipe -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-634f6d39-a49b-4257-ae1d-ce3a7b653632 ovbh-pprod-xen05: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-b268e0cd-d277-47d7-b47d-58cbcafffc7b ovbh-pprod-xen05: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8cff886b-466a-4907-accc-cb87caeacb7b ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: Connection closed
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b: Preparing cluster-wide state change 327023154: 1->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b: Committing cluster-wide state change 327023154 (0ms)
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-889b875a-3cec-43df-9517-7149c9ba4274 ovbh-pprod-xen02: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3b8e7b9-5b4a-496c-8b8a-22d8c63fe844 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: conn( BrokenPipe -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-5958e03b-6729-4470-9509-71b3aa799566 ovbh-pprod-xen02: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-74cc3f54-c039-41e7-9612-e4faed647c25 ovbh-pprod-xen02: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd pvc-7bffac62-0d6d-4fe4-a784-c71fe007ac88 ovbh-pprod-xen03: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-cebb026d-d674-4ffc-a663-3419d68c5c2e ovbh-pprod-xen02: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-bafada2f-13eb-454e-a127-74db57f076f9 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen05: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-94ac6def-5b82-4663-94a2-acd968dee44d ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-6ff0faeb-1c2c-4865-a3df-29eb7f2c0af5 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-87ad533a-5854-44b3-b36b-b0c06272f357 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-c48964bd-7b7d-4104-85df-641dfb00acd3 ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: conn( BrokenPipe -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd pvc-160ad3d2-ee52-49b3-8718-dd5306e3bfec ovbh-pprod-xen02: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: conn( BrokenPipe -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-34e6854d-05e5-46a7-be32-3d7dd450f5d8 ovbh-pprod-xen05: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: conn( BrokenPipe -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-b3cfd182-ce10-4132-a965-3732747036a5 ovbh-pprod-xen05: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-3f8d6b42-93b5-43f8-9e7b-08d877148b60 ovbh-pprod-xen03: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-70622142-3a65-431e-ad06-545eabb0d9ed ovbh-pprod-xen02: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-eec028bb-744b-4869-a4e9-624e604a1aaf ovbh-pprod-xen01: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: helper command: /sbin/drbdadm disconnected exit code 0
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: conn( NetworkFailure -> Unconnected ) [disconnected]
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: Restarting receiver thread
2026-07-04T23:16:05-07:00 drbd xcp-volume-8a15f56b-a00c-4df5-a801-5056d57e781b ovbh-pprod-xen03: conn( Unconnected -> Connecting ) [connecting]
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466: Preparing cluster-wide state change 1923768138: 2->all empty
2026-07-04T23:16:05-07:00 drbd xcp-volume-424ad322-988c-4b30-a46e-390ed6e35466: Committing cluster-wide state change 1923768138 (0ms)