Feb 17 18:34:46 xencc-hp03 kernel: [1944466.346906] drbd xcp-persistent-database xenrt-1: PingAck did not arrive in time.
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.346934] drbd xcp-persistent-database/0 drbd1000: Would lose quorum, but using tiebreaker logic to keep
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.346944] drbd xcp-persistent-database xenrt-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.346946] drbd xcp-persistent-database/0 drbd1000 xenrt-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.347034] drbd xcp-persistent-database: Preparing cluster-wide state change 3110230134: 1->all empty
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.347179] drbd xcp-persistent-database xenrt-1: Terminating sender thread
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.347205] drbd xcp-persistent-database xenrt-1: Starting sender thread (peer-node-id 0)
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.371366] drbd xcp-persistent-database xenrt-1: Connection closed
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.371374] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.375292] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.375331] drbd xcp-persistent-database xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:34:46 xencc-hp03 kernel: [1944466.375340] drbd xcp-persistent-database xenrt-1: Restarting receiver thread
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.150908] drbd xcp-persistent-database xenrt-2: PingAck did not arrive in time.
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.150926] drbd xcp-persistent-database xenrt-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.150929] drbd xcp-persistent-database/0 drbd1000: quorum( yes -> no )
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.150931] drbd xcp-persistent-database/0 drbd1000 xenrt-2: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.150971] drbd xcp-persistent-database: Retrying cluster-wide state change after 0ms
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.151174] drbd xcp-persistent-database xenrt-2: Terminating sender thread
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.151195] drbd xcp-persistent-database xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.151436] drbd xcp-persistent-database/0 drbd1000: helper command: /sbin/drbdadm quorum-lost
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.155861] drbd xcp-persistent-database/0 drbd1000: helper command: /sbin/drbdadm quorum-lost exit code 0
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.158915] drbd xcp-persistent-database: Preparing cluster-wide state change 1760754051: 1->all empty
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.158918] drbd xcp-persistent-database: Committing cluster-wide state change 1760754051 (0ms)
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.158969] drbd xcp-persistent-database xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.191463] drbd xcp-persistent-database xenrt-2: Connection closed
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.191472] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.195288] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.195327] drbd xcp-persistent-database xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.195332] drbd xcp-persistent-database xenrt-2: Restarting receiver thread
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.195335] drbd xcp-persistent-database xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.721006] drbd xcp-persistent-database: Preparing cluster-wide state change 4171696264: 1->all role( Secondary )
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.721010] drbd xcp-persistent-database: Committing cluster-wide state change 4171696264 (0ms)
Feb 17 18:34:47 xencc-hp03 kernel: [1944467.721013] drbd xcp-persistent-database: role( Primary -> Secondary ) [secondary]
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.910928] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: PingAck did not arrive in time.
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.910943] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.910947] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-2: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.910977] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Preparing cluster-wide state change 2472470492: 0->all empty
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.911142] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Terminating sender thread
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.911192] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.951445] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Connection closed
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.951455] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.955453] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.955512] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:34:49 xencc-hp03 kernel: [1944468.955528] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Restarting receiver thread
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.418930] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: PingAck did not arrive in time.
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.418946] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.418949] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007: quorum( yes -> no )
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.418951] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.418983] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Retrying cluster-wide state change after 0ms
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.419270] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Terminating sender thread
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.419304] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.426933] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Preparing cluster-wide state change 1416914064: 0->all empty
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.426935] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Committing cluster-wide state change 1416914064 (0ms)
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.426963] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.459532] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Connection closed
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.459543] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.464233] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.464273] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.464278] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Restarting receiver thread
Feb 17 18:34:49 xencc-hp03 kernel: [1944469.464283] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:34:52 xencc-hp03 kernel: [1944471.850963] drbd xcp-persistent-database tcp:xenrt-1: Error receiving initial packet: -11
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.042990] drbd xcp-persistent-database xenrt-2: sock_recvmsg returned -11
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.043011] drbd xcp-persistent-database xenrt-2: conn( Connecting -> BrokenPipe )
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.043043] drbd xcp-persistent-database xenrt-2: short read (expected size 8)
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.043045] drbd xcp-persistent-database xenrt-2: timeout while waiting for feature packet
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.043236] drbd xcp-persistent-database xenrt-2: Terminating sender thread
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.043276] drbd xcp-persistent-database xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.075517] drbd xcp-persistent-database xenrt-2: Connection closed
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.075528] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.079622] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:34:52 xencc-hp03 kernel: [1944472.079661] drbd xcp-persistent-database xenrt-2: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:34:53 xencc-hp03 kernel: [1944473.098998] drbd xcp-persistent-database xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:34:54 xencc-hp03 kernel: [1944474.635132] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:34:54 xencc-hp03 kernel: [1944474.635135] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:34:55 xencc-hp03 kernel: [1944474.931338] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:34:55 xencc-hp03 kernel: [1944474.935101] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Preparing cluster-wide state change 166299536: 0->2 role( Secondary ) conn( Connected )
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.467026] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Connecting -> NetworkFailure )
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.467069] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: sock_recvmsg returned -4
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.467202] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Aborting cluster-wide state change 166299536 (532ms) rv = -21
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.467208] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Failure to connect: Interrupted state change (-21); retrying
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.467216] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Terminating sender thread
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.467255] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.503566] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Connection closed
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.503574] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.508566] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.508604] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.508609] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Restarting receiver thread
Feb 17 18:34:55 xencc-hp03 kernel: [1944475.508612] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.330991] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: PingAck did not arrive in time.
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.331002] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004: Would lose quorum, but using tiebreaker logic to keep
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.331010] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.331013] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.331064] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Preparing cluster-wide state change 3438469613: 0->all empty
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.331240] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Terminating sender thread
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.331262] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.355427] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Connection closed
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.355436] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.360030] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.360097] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:34:56 xencc-hp03 kernel: [1944476.360119] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Restarting receiver thread
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.907000] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: PingAck did not arrive in time.
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.907042] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.907048] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004: quorum( yes -> no )
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.907051] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-2: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.907085] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Retrying cluster-wide state change after 0ms
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.907357] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Terminating sender thread
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.907400] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.907660] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004: helper command: /sbin/drbdadm quorum-lost
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.912021] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004: helper command: /sbin/drbdadm quorum-lost exit code 0
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.915002] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Preparing cluster-wide state change 264378531: 0->all empty
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.915004] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Committing cluster-wide state change 264378531 (0ms)
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.915030] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.955635] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Connection closed
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.955645] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.959932] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.959973] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.959980] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Restarting receiver thread
Feb 17 18:34:57 xencc-hp03 kernel: [1944476.959984] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:34:58 xencc-hp03 kernel: [1944478.444295] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:34:58 xencc-hp03 kernel: [1944478.444298] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.459063] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.459076] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.459087] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: short read (expected size 16)
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.459089] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Authentication of peer failed, trying again.
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.459201] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Terminating sender thread
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.459234] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.491619] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Connection closed
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.491629] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.495794] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:00 xencc-hp03 kernel: [1944480.495836] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:01 xencc-hp03 kernel: [1944481.515081] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.443111] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 tcp:xenrt-2: initial packet M crossed
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.475053] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: PingAck did not arrive in time.
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.475072] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.475076] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-2: pdsk( Diskless -> DUnknown ) repl( Established -> Off )
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.475105] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: Preparing cluster-wide state change 2411540705: 0->all empty
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.475250] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Terminating sender thread
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.475278] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.527640] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Connection closed
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.527650] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.533269] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.533311] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:02 xencc-hp03 kernel: [1944482.533322] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Restarting receiver thread
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.987087] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: PingAck did not arrive in time.
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.987105] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Connected -> NetworkFailure ) peer( Primary -> Unknown )
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.987109] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005: disk( UpToDate -> Consistent ) quorum( yes -> no )
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.987111] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.987116] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005: Enabling local AL-updates
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.987151] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: Retrying cluster-wide state change after 0ms
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.987325] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Terminating sender thread
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.987359] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.995078] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: Preparing cluster-wide state change 2207755055: 0->all empty
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.995081] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: Committing cluster-wide state change 2207755055 (0ms)
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.995089] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005: disk( Consistent -> UpToDate ) [lost-peer]
Feb 17 18:35:03 xencc-hp03 kernel: [1944482.995134] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:03 xencc-hp03 kernel: [1944483.015675] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Connection closed
Feb 17 18:35:03 xencc-hp03 kernel: [1944483.015684] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:03 xencc-hp03 kernel: [1944483.019983] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:03 xencc-hp03 kernel: [1944483.020029] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:03 xencc-hp03 kernel: [1944483.020036] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Restarting receiver thread
Feb 17 18:35:03 xencc-hp03 kernel: [1944483.020039] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:04 xencc-hp03 kernel: [1944484.424570] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:04 xencc-hp03 kernel: [1944484.424573] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.116846] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.116849] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.412549] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.415154] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Preparing cluster-wide state change 559901527: 0->2 role( Primary ) conn( Connected )
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.443186] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.443192] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.443202] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: short read (expected size 16)
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.443204] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Authentication of peer failed, trying again.
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.443348] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Terminating sender thread
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.443376] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.447113] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: sock_recvmsg returned -11
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.447126] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Connecting -> BrokenPipe )
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.447136] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: short read (expected size 8)
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.447138] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: timeout while waiting for feature packet
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.447287] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Terminating sender thread
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.447311] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.483666] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Connection closed
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.483675] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.483682] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Connection closed
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.483689] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.488168] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.488218] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.488271] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.488339] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.664398] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:06 xencc-hp03 kernel: [1944486.664401] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.927107] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Connecting -> NetworkFailure )
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.927151] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: sock_recvmsg returned -4
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.927287] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Aborting cluster-wide state change 559901527 (512ms) rv = -21
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.927297] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Failure to connect: Interrupted state change (-21); retrying
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.927303] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Terminating sender thread
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.927323] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.971633] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Connection closed
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.971644] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.976057] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.976099] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.976106] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Restarting receiver thread
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.976110] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.988624] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: meta connection shut down by peer.
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.988661] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Connecting -> NetworkFailure )
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.988703] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: sock_recvmsg returned -4
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.988718] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Authentication of peer failed, trying again.
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.988842] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Terminating sender thread
Feb 17 18:35:07 xencc-hp03 kernel: [1944486.988862] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:07 xencc-hp03 kernel: [1944487.023582] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Connection closed
Feb 17 18:35:07 xencc-hp03 kernel: [1944487.023594] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:07 xencc-hp03 kernel: [1944487.027895] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:07 xencc-hp03 kernel: [1944487.027954] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:07 xencc-hp03 kernel: [1944487.499128] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:07 xencc-hp03 kernel: [1944487.499136] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:08 xencc-hp03 kernel: [1944487.851113] drbd xcp-persistent-database tcp:xenrt-2: Error receiving initial packet: -11
Feb 17 18:35:08 xencc-hp03 kernel: [1944488.043117] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:08 xencc-hp03 kernel: [1944488.363164] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f tcp:xenrt-1: Error receiving initial packet: -11
Feb 17 18:35:11 xencc-hp03 kernel: [1944491.151273] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:11 xencc-hp03 kernel: [1944491.151276] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:12 xencc-hp03 kernel: [1944492.200884] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:12 xencc-hp03 kernel: [1944492.200887] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:12 xencc-hp03 kernel: [1944492.524602] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: meta connection shut down by peer.
Feb 17 18:35:12 xencc-hp03 kernel: [1944492.524631] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Connecting -> NetworkFailure )
Feb 17 18:35:13 xencc-hp03 kernel: [1944492.971192] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: sock_recvmsg returned -11
Feb 17 18:35:13 xencc-hp03 kernel: [1944492.971214] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Connecting -> BrokenPipe )
Feb 17 18:35:13 xencc-hp03 kernel: [1944492.971228] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: short read (expected size 8)
Feb 17 18:35:13 xencc-hp03 kernel: [1944492.971228] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: timeout while waiting for feature packet
Feb 17 18:35:13 xencc-hp03 kernel: [1944492.971348] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Terminating sender thread
Feb 17 18:35:13 xencc-hp03 kernel: [1944492.971381] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:13 xencc-hp03 kernel: [1944492.999607] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Connection closed
Feb 17 18:35:13 xencc-hp03 kernel: [1944492.999616] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.003704] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.003755] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.163183] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: sock_recvmsg returned -11
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.163208] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Connecting -> BrokenPipe )
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.163233] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: short read (expected size 16)
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.163236] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Authentication of peer failed, trying again.
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.163366] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Terminating sender thread
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.163398] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.195629] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Connection closed
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.195638] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.199615] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.199658] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.707162] drbd xcp-persistent-database xenrt-2: sock_recvmsg returned -11
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.707176] drbd xcp-persistent-database xenrt-2: conn( Connecting -> BrokenPipe )
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.707186] drbd xcp-persistent-database xenrt-2: short read (expected size 8)
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.707187] drbd xcp-persistent-database xenrt-2: timeout while waiting for feature packet
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.707297] drbd xcp-persistent-database xenrt-2: Terminating sender thread
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.707330] drbd xcp-persistent-database xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.751714] drbd xcp-persistent-database xenrt-2: Connection closed
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.751736] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.756082] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.756121] drbd xcp-persistent-database xenrt-2: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.768559] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.768794] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Terminating sender thread
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.768830] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.775245] drbd xcp-persistent-database xenrt-1: Handshake to peer 0 successful: Agreed network protocol version 122
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.775247] drbd xcp-persistent-database xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.803668] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Connection closed
Feb 17 18:35:13 xencc-hp03 kernel: [1944493.803676] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:14 xencc-hp03 kernel: [1944493.808757] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:14 xencc-hp03 kernel: [1944493.808799] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:14 xencc-hp03 kernel: [1944493.808805] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Restarting receiver thread
Feb 17 18:35:14 xencc-hp03 kernel: [1944493.808809] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:14 xencc-hp03 kernel: [1944494.027191] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:14 xencc-hp03 kernel: [1944494.219192] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:14 xencc-hp03 kernel: [1944494.763223] drbd xcp-persistent-database xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:15 xencc-hp03 kernel: [1944494.988455] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:15 xencc-hp03 kernel: [1944494.988459] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:15 xencc-hp03 kernel: [1944495.603098] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 tcp:xenrt-1: Closing unexpected connection from 172.27.18.162
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.079256] drbd xcp-persistent-database xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.079272] drbd xcp-persistent-database xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.079287] drbd xcp-persistent-database xenrt-1: short read (expected size 16)
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.079290] drbd xcp-persistent-database xenrt-1: Authentication of peer failed, trying again.
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.079411] drbd xcp-persistent-database xenrt-1: Terminating sender thread
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.079443] drbd xcp-persistent-database xenrt-1: Starting sender thread (peer-node-id 0)
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.103651] drbd xcp-persistent-database xenrt-1: Connection closed
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.103660] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.107513] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.107553] drbd xcp-persistent-database xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.200616] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: meta connection shut down by peer.
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.200628] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Connecting -> NetworkFailure )
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.200659] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: sock_recvmsg returned -4
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.200671] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Authentication of peer failed, trying again.
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.200784] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Terminating sender thread
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.200811] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.239706] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Connection closed
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.239716] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.243533] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:16 xencc-hp03 kernel: [1944496.243572] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:17 xencc-hp03 kernel: [1944497.135213] drbd xcp-persistent-database xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:17 xencc-hp03 kernel: [1944497.259216] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:18 xencc-hp03 kernel: [1944498.699338] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f tcp:xenrt-2: Error receiving initial packet: -11
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.819341] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.819365] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.819384] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: short read (expected size 8)
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.819385] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: timeout while waiting for feature packet
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.819571] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Terminating sender thread
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.819627] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.867815] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Connection closed
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.867827] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.872360] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:20 xencc-hp03 kernel: [1944499.872401] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:20 xencc-hp03 kernel: [1944500.136467] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:20 xencc-hp03 kernel: [1944500.136471] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:21 xencc-hp03 kernel: [1944500.875245] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.155242] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.155255] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.155268] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: short read (expected size 16)
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.155270] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Authentication of peer failed, trying again.
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.155396] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Terminating sender thread
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.155427] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.191804] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Connection closed
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.191815] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.198408] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:22 xencc-hp03 kernel: [1944502.198465] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.211275] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.244544] drbd xcp-persistent-database tcp:xenrt-2: initial packet S crossed
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.371281] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: sock_recvmsg returned -11
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.371294] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Connecting -> BrokenPipe )
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.371306] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: short read (expected size 8)
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.371308] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: timeout while waiting for feature packet
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.371339] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Terminating sender thread
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.371374] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.411822] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Connection closed
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.411831] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.416742] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:23 xencc-hp03 kernel: [1944503.416785] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:24 xencc-hp03 kernel: [1944504.427299] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:24 xencc-hp03 kernel: [1944504.528259] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:24 xencc-hp03 kernel: [1944504.528262] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:25 xencc-hp03 kernel: [1944504.824184] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:25 xencc-hp03 kernel: [1944504.827307] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Preparing cluster-wide state change 3286236299: 0->1 role( Primary ) conn( Connected )
Feb 17 18:35:25 xencc-hp03 kernel: [1944505.003281] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 tcp:xenrt-2: Error receiving initial packet: -11
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.184837] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: drbd_sync_handshake:
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.184841] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: self 57EFF08A4B94178A:0000000000000000:FA42330CB13A545C:6E134E436749B322 bits:0 flags:520
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.184843] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: peer 57EFF08A4B94178A:0000000000000000:FA42330CB13A545C:C652E222C96B3A8A bits:0 flags:1120
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.184845] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: uuid_compare()=no-sync by rule=reconnected
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.220686] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: State change 3286236299: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.220689] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Committing cluster-wide state change 3286236299 (1392ms)
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.220723] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.220725] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004: quorum( no -> yes ) [connected]
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.220727] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [connected]
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.220881] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: cleared bm UUID and bitmap 57EFF08A4B94178A:0000000000000000:FA42330CB13A545C:6E134E436749B322
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.443392] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:26 xencc-hp03 kernel: [1944506.443396] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.051339] drbd xcp-persistent-database xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.051352] drbd xcp-persistent-database xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.051362] drbd xcp-persistent-database xenrt-1: short read (expected size 8)
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.051363] drbd xcp-persistent-database xenrt-1: timeout while waiting for feature packet
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.051483] drbd xcp-persistent-database xenrt-1: Terminating sender thread
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.051510] drbd xcp-persistent-database xenrt-1: Starting sender thread (peer-node-id 0)
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.075722] drbd xcp-persistent-database xenrt-1: Connection closed
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.075731] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.079552] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.079589] drbd xcp-persistent-database xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.307308] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.307322] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.307332] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: short read (expected size 8)
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.307335] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: timeout while waiting for feature packet
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.307444] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Terminating sender thread
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.307470] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.335812] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Connection closed
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.335825] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.339892] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.339932] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.556969] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: meta connection shut down by peer.
Feb 17 18:35:27 xencc-hp03 kernel: [1944507.556995] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Connecting -> NetworkFailure )
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.107339] drbd xcp-persistent-database xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.363331] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.459299] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: sock_recvmsg returned -11
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.459311] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: short read (expected size 16)
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.459313] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Authentication of peer failed, trying again.
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.459359] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Terminating sender thread
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.459389] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.491839] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Connection closed
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.491850] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.495977] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.496021] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.747310] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.747326] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.747337] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: short read (expected size 8)
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.747340] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: timeout while waiting for feature packet
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.747460] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Terminating sender thread
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.747497] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.779755] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Connection closed
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.779765] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.784744] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:28 xencc-hp03 kernel: [1944508.784794] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:29 xencc-hp03 kernel: [1944509.519379] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:29 xencc-hp03 kernel: [1944509.803363] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:30 xencc-hp03 kernel: [1944509.931496] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:30 xencc-hp03 kernel: [1944509.931501] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.835071] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: sock was reset by peer
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.835084] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Connecting -> BrokenPipe )
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.835095] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: short read (expected size 16)
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.835098] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Authentication of peer failed, trying again.
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.835217] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Terminating sender thread
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.835248] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.843359] drbd xcp-persistent-database xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.843361] drbd xcp-persistent-database xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.867809] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Connection closed
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.867819] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.871702] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:31 xencc-hp03 kernel: [1944510.871743] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:32 xencc-hp03 kernel: [1944511.883352] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.676691] drbd xcp-persistent-database xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.676917] drbd xcp-persistent-database xenrt-2: conn( Connecting -> NetworkFailure ) [disconnected]
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.677054] drbd xcp-persistent-database xenrt-2: Terminating sender thread
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.677082] drbd xcp-persistent-database xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.703927] drbd xcp-persistent-database xenrt-2: Connection closed
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.703936] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.707988] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.708071] drbd xcp-persistent-database xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.708086] drbd xcp-persistent-database xenrt-2: Restarting receiver thread
Feb 17 18:35:32 xencc-hp03 kernel: [1944512.708092] drbd xcp-persistent-database xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:35 xencc-hp03 kernel: [1944515.243364] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f tcp:xenrt-2: Error receiving initial packet: -11
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.779373] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: PingAck did not arrive in time.
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.779390] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Connected -> NetworkFailure ) peer( Secondary -> Unknown )
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.779392] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004: quorum( yes -> no )
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.779395] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: pdsk( UpToDate -> DUnknown ) repl( Established -> Off )
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.779447] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Preparing cluster-wide state change 595924193: 0->all empty
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.779458] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Committing cluster-wide state change 595924193 (0ms)
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.779792] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Terminating sender thread
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.779844] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.780097] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004: helper command: /sbin/drbdadm quorum-lost
Feb 17 18:35:36 xencc-hp03 kernel: [1944516.783928] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004: helper command: /sbin/drbdadm quorum-lost exit code 0
Feb 17 18:35:37 xencc-hp03 kernel: [1944516.811826] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Connection closed
Feb 17 18:35:37 xencc-hp03 kernel: [1944516.811834] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:37 xencc-hp03 kernel: [1944516.815664] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:37 xencc-hp03 kernel: [1944516.815748] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:37 xencc-hp03 kernel: [1944516.815757] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Restarting receiver thread
Feb 17 18:35:37 xencc-hp03 kernel: [1944516.815765] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:37 xencc-hp03 kernel: [1944517.796323] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:37 xencc-hp03 kernel: [1944517.796327] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.571462] drbd xcp-persistent-database xenrt-2: sock_recvmsg returned -11
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.571475] drbd xcp-persistent-database xenrt-2: conn( Connecting -> BrokenPipe )
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.571486] drbd xcp-persistent-database xenrt-2: short read (expected size 8)
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.571488] drbd xcp-persistent-database xenrt-2: timeout while waiting for feature packet
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.571594] drbd xcp-persistent-database xenrt-2: Terminating sender thread
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.571639] drbd xcp-persistent-database xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.603802] drbd xcp-persistent-database xenrt-2: Connection closed
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.603811] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.608370] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:38 xencc-hp03 kernel: [1944518.608409] drbd xcp-persistent-database xenrt-2: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.627443] drbd xcp-persistent-database xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.652136] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.652384] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Connecting -> NetworkFailure ) [disconnected]
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.652518] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Terminating sender thread
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.652537] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.683918] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Connection closed
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.683929] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.688114] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.688156] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.688163] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Restarting receiver thread
Feb 17 18:35:39 xencc-hp03 kernel: [1944519.688167] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.076756] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f tcp:xenrt-2: Closing unexpected connection from 172.27.18.161
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.107426] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.107438] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.107448] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: short read (expected size 8)
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.107450] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: timeout while waiting for feature packet
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.107576] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Terminating sender thread
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.107606] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.147913] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Connection closed
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.147922] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.151893] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:40 xencc-hp03 kernel: [1944520.151934] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:41 xencc-hp03 kernel: [1944521.167422] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:42 xencc-hp03 kernel: [1944521.900077] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:42 xencc-hp03 kernel: [1944521.900081] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.164716] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.167463] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Preparing cluster-wide state change 2716043952: 0->1 role( Primary ) conn( Connected )
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.699442] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Connecting -> NetworkFailure )
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.699489] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: sock_recvmsg returned -4
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.699633] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Aborting cluster-wide state change 2716043952 (532ms) rv = -21
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.699638] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Failure to connect: Interrupted state change (-21); retrying
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.699643] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Terminating sender thread
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.699681] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.735920] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Connection closed
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.735929] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.740267] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.740310] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.740317] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Restarting receiver thread
Feb 17 18:35:42 xencc-hp03 kernel: [1944522.740320] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.139491] drbd xcp-persistent-database xenrt-1: sock_recvmsg returned -11
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.139503] drbd xcp-persistent-database xenrt-1: conn( Connecting -> BrokenPipe )
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.139528] drbd xcp-persistent-database xenrt-1: short read (expected size 8)
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.139530] drbd xcp-persistent-database xenrt-1: timeout while waiting for feature packet
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.139637] drbd xcp-persistent-database xenrt-1: Terminating sender thread
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.139688] drbd xcp-persistent-database xenrt-1: Starting sender thread (peer-node-id 0)
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.175980] drbd xcp-persistent-database xenrt-1: Connection closed
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.175991] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.180122] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:44 xencc-hp03 kernel: [1944524.180162] drbd xcp-persistent-database xenrt-1: conn( BrokenPipe -> Unconnected ) [disconnected]
Feb 17 18:35:45 xencc-hp03 kernel: [1944525.195489] drbd xcp-persistent-database xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.399433] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f tcp:xenrt-2: initial packet M crossed
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.447489] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 tcp:xenrt-2: Error receiving initial packet: -11
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.505080] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.505084] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.558139] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.558258] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Connecting -> NetworkFailure ) [disconnected]
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.558369] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Terminating sender thread
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.558387] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.604045] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Connection closed
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.604058] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.608008] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.608071] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.608080] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Restarting receiver thread
Feb 17 18:35:46 xencc-hp03 kernel: [1944526.608085] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.105885] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.105888] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.106742] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.111484] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Preparing cluster-wide state change 775236089: 0->1 role( Primary ) conn( Connected )
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171562] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: drbd_sync_handshake:
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171565] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: self 57EFF08A4B94178A:0000000000000000:FA42330CB13A545C:6E134E436749B322 bits:0 flags:520
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171567] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: peer 57EFF08A4B94178A:0000000000000000:FA42330CB13A545C:C652E222C96B3A8A bits:0 flags:1120
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171569] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: uuid_compare()=no-sync by rule=reconnected
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171582] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: State change 775236089: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFFC
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171584] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Committing cluster-wide state change 775236089 (60ms)
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171613] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171615] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004: quorum( no -> yes ) [connected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171617] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [connected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.171720] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-1: cleared bm UUID and bitmap 57EFF08A4B94178A:0000000000000000:FA42330CB13A545C:6E134E436749B322
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.211653] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.211657] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.212701] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.219487] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: Preparing cluster-wide state change 1370630302: 0->1 role( Secondary ) conn( Connected )
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287568] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: drbd_sync_handshake:
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287572] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: self 873790D49A5C0B7A:0000000000000000:CF6BF6F8F1EEA388:EDD43057DEFA1C60 bits:0 flags:120
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287574] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: peer D6B5F821BE795335:873790D49A5C0B7A:CF6BF6F8F1EEA388:EDD43057DEFA1C60 bits:3 flags:1120
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287576] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: uuid_compare()=target-use-bitmap by rule=bitmap-peer
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287596] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: State change 1370630302: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF8
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287598] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: Committing cluster-wide state change 1370630302 (68ms)
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287626] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-1: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [connected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287628] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005: disk( UpToDate -> Outdated ) quorum( no -> yes ) [connected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287630] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [connected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287635] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005: Disabling local AL-updates (optimization)
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.287640] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005: Setting exposed data uuid: D6B5F821BE795334
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.290081] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 29(1), total 29; compression: 100.0%
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.290290] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 29(1), total 29; compression: 100.0%
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.290297] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: helper command: /sbin/drbdadm before-resync-target
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.294666] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: helper command: /sbin/drbdadm before-resync-target exit code 0
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.294703] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005: disk( Outdated -> Inconsistent ) [receive-bitmap]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.294705] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-2: resync-susp( no -> connection dependency ) [receive-bitmap]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.294706] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.294843] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: Began resync as SyncTarget (will sync 12 KB [3 bits set]).
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.301349] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: Resync done (total 1 sec; paused 0 sec; 12 K/sec)
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.301354] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: updated UUIDs D6B5F821BE795334:0000000000000000:CF6BF6F8F1EEA388:EDD43057DEFA1C60
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.301359] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005: disk( Inconsistent -> UpToDate ) [resync-finished]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.301361] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-2: resync-susp( connection dependency -> no ) [resync-finished]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.301363] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: repl( SyncTarget -> Established ) [resync-finished]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.301597] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: helper command: /sbin/drbdadm after-resync-target
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.306178] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-1: helper command: /sbin/drbdadm after-resync-target exit code 0
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.499627] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.499630] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.500757] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.500876] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Connecting -> NetworkFailure ) [disconnected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.500948] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Terminating sender thread
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.500967] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.536049] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Connection closed
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.536058] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.540063] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.540100] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.540106] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Restarting receiver thread
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.540109] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.659671] drbd xcp-persistent-database xenrt-1: Handshake to peer 0 successful: Agreed network protocol version 122
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.659675] drbd xcp-persistent-database xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.660699] drbd xcp-persistent-database xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.660897] drbd xcp-persistent-database xenrt-1: conn( Connecting -> NetworkFailure ) [disconnected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.661058] drbd xcp-persistent-database xenrt-1: Terminating sender thread
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.661089] drbd xcp-persistent-database xenrt-1: Starting sender thread (peer-node-id 0)
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.687941] drbd xcp-persistent-database xenrt-1: Connection closed
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.687954] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.692046] drbd xcp-persistent-database xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.692100] drbd xcp-persistent-database xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.692108] drbd xcp-persistent-database xenrt-1: Restarting receiver thread
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.692114] drbd xcp-persistent-database xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.710157] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.710159] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.711001] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.715485] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Preparing cluster-wide state change 361750818: 0->2 role( Primary ) conn( Connected )
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.738004] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: meta connection shut down by peer.
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.738031] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Connecting -> NetworkFailure )
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.738056] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: meta connection shut down by peer.
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.747718] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Aborting cluster-wide state change 361750818 (32ms) rv = -21
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.747744] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Failure to connect: Interrupted state change (-21); retrying
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.747751] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Terminating sender thread
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.747803] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.787951] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Connection closed
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.787959] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.792391] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.792431] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.792436] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Restarting receiver thread
Feb 17 18:35:47 xencc-hp03 kernel: [1944527.792440] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.075632] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.075639] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.076558] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.079503] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: Preparing cluster-wide state change 976694555: 0->2 role( Secondary ) conn( Connected )
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.135507] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-2: self D6B5F821BE795334:873790D49A5C0B7A:CF6BF6F8F1EEA388:EDD43057DEFA1C60 bits:0 flags:0
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.135509] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-2: peer's exposed UUID: D6B5F821BE795334
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.142114] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: State change 976694555: primary_nodes=2, weak_nodes=FFFFFFFFFFFFFFF8
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.142117] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f: Committing cluster-wide state change 976694555 (60ms)
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.142160] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f xenrt-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.142164] drbd xcp-volume-e9428d9d-97a7-4a37-a2bb-630f8b5f3f0f/0 drbd1005 xenrt-2: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.267614] drbd xcp-persistent-database xenrt-1: Handshake to peer 0 successful: Agreed network protocol version 122
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.267616] drbd xcp-persistent-database xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.268643] drbd xcp-persistent-database xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.295678] drbd xcp-persistent-database: Preparing remote state change 1559488521: 0->1 role( Primary ) conn( Connected )
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.318124] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.318127] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.319435] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.323490] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Preparing cluster-wide state change 3067017270: 0->2 role( Primary ) conn( Connected )
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.327536] drbd xcp-persistent-database/0 drbd1000 xenrt-1: drbd_sync_handshake:
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.327539] drbd xcp-persistent-database/0 drbd1000 xenrt-1: self B97FF5ABF29298EE:0000000000000000:8A57F94B44CB2D10:065AC82552A65674 bits:0 flags:520
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.327541] drbd xcp-persistent-database/0 drbd1000 xenrt-1: peer 9861044C2AFEEF0F:B97FF5ABF29298EE:07AD6A6892BC65BA:065AC82552A65674 bits:20 flags:1120
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.327544] drbd xcp-persistent-database/0 drbd1000 xenrt-1: uuid_compare()=target-use-bitmap by rule=bitmap-peer
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.328374] drbd xcp-persistent-database xenrt-1: Committing remote state change 1559488521 (primary_nodes=1)
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.328382] drbd xcp-persistent-database xenrt-1: conn( Connecting -> Connected ) peer( Unknown -> Primary ) [remote]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.328384] drbd xcp-persistent-database/0 drbd1000: disk( UpToDate -> Outdated ) quorum( no -> yes ) [remote]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.328386] drbd xcp-persistent-database/0 drbd1000 xenrt-1: pdsk( DUnknown -> UpToDate ) repl( Off -> WFBitMapT ) [remote]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.328393] drbd xcp-persistent-database/0 drbd1000: Disabling local AL-updates (optimization)
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.328397] drbd xcp-persistent-database/0 drbd1000: Setting exposed data uuid: 9861044C2AFEEF0E
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.364091] drbd xcp-persistent-database/0 drbd1000 xenrt-1: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 34(1), total 34; compression: 99.9%
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.364139] drbd xcp-persistent-database/0 drbd1000 xenrt-1: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 34(1), total 34; compression: 99.9%
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.364150] drbd xcp-persistent-database/0 drbd1000 xenrt-1: helper command: /sbin/drbdadm before-resync-target
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.368444] drbd xcp-persistent-database/0 drbd1000 xenrt-1: helper command: /sbin/drbdadm before-resync-target exit code 0
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.368490] drbd xcp-persistent-database/0 drbd1000: disk( Outdated -> Inconsistent ) [receive-bitmap]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.368492] drbd xcp-persistent-database/0 drbd1000 xenrt-2: resync-susp( no -> connection dependency ) [receive-bitmap]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.368494] drbd xcp-persistent-database/0 drbd1000 xenrt-1: repl( WFBitMapT -> SyncTarget ) [receive-bitmap]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.368648] drbd xcp-persistent-database/0 drbd1000 xenrt-1: Began resync as SyncTarget (will sync 80 KB [20 bits set]).
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.380049] drbd xcp-persistent-database/0 drbd1000 xenrt-1: Resync done (total 1 sec; paused 0 sec; 80 K/sec)
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.380055] drbd xcp-persistent-database/0 drbd1000 xenrt-1: updated UUIDs 9861044C2AFEEF0E:0000000000000000:07AD6A6892BC65BA:8A57F94B44CB2D10
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.380060] drbd xcp-persistent-database/0 drbd1000: disk( Inconsistent -> UpToDate ) [resync-finished]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.380062] drbd xcp-persistent-database/0 drbd1000 xenrt-2: resync-susp( connection dependency -> no ) [resync-finished]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.380064] drbd xcp-persistent-database/0 drbd1000 xenrt-1: repl( SyncTarget -> Established ) [resync-finished]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.380250] drbd xcp-persistent-database/0 drbd1000 xenrt-1: helper command: /sbin/drbdadm after-resync-target
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.384155] drbd xcp-persistent-database/0 drbd1000 xenrt-1: helper command: /sbin/drbdadm after-resync-target exit code 0
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.391524] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-2: self 57EFF08A4B94178A:0000000000000000:FA42330CB13A545C:6E134E436749B322 bits:0 flags:0
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.391527] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-2: peer's exposed UUID: 57EFF08A4B94178A
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.391535] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: State change 3067017270: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFF8
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.391537] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4: Committing cluster-wide state change 3067017270 (68ms)
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.391573] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4 xenrt-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
Feb 17 18:35:48 xencc-hp03 kernel: [1944528.391575] drbd xcp-volume-623a917e-614f-4176-8e58-505248ee9db4/0 drbd1004 xenrt-2: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.830117] drbd xcp-persistent-database xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.830120] drbd xcp-persistent-database xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.831030] drbd xcp-persistent-database xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.831156] drbd xcp-persistent-database xenrt-2: conn( Connecting -> NetworkFailure ) [disconnected]
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.831255] drbd xcp-persistent-database xenrt-2: Terminating sender thread
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.831274] drbd xcp-persistent-database xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.864094] drbd xcp-persistent-database xenrt-2: Connection closed
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.864104] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.868361] drbd xcp-persistent-database xenrt-2: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.868401] drbd xcp-persistent-database xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.868408] drbd xcp-persistent-database xenrt-2: Restarting receiver thread
Feb 17 18:35:49 xencc-hp03 kernel: [1944528.868411] drbd xcp-persistent-database xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.406142] drbd xcp-persistent-database xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.406148] drbd xcp-persistent-database xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.407004] drbd xcp-persistent-database xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.411541] drbd xcp-persistent-database: Preparing cluster-wide state change 2468230084: 1->2 role( Secondary ) conn( Connected )
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.483520] drbd xcp-persistent-database/0 drbd1000 xenrt-2: self 9861044C2AFEEF0E:B97FF5ABF29298EE:07AD6A6892BC65BA:8A57F94B44CB2D10 bits:0 flags:0
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.483523] drbd xcp-persistent-database/0 drbd1000 xenrt-2: peer's exposed UUID: 9861044C2AFEEF0E
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.483536] drbd xcp-persistent-database: State change 2468230084: primary_nodes=1, weak_nodes=FFFFFFFFFFFFFFF8
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.483538] drbd xcp-persistent-database: Committing cluster-wide state change 2468230084 (72ms)
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.483588] drbd xcp-persistent-database xenrt-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
Feb 17 18:35:49 xencc-hp03 kernel: [1944529.483590] drbd xcp-persistent-database/0 drbd1000 xenrt-2: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.646187] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.646190] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.647033] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.647151] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Connecting -> NetworkFailure ) [disconnected]
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.647241] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Terminating sender thread
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.647260] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Starting sender thread (peer-node-id 2)
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.675940] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Connection closed
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.675949] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.679215] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: helper command: /sbin/drbdadm disconnected exit code 1 (0x100)
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.679350] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.679368] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Restarting receiver thread
Feb 17 18:35:51 xencc-hp03 kernel: [1944531.679375] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.809987] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.809991] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.810893] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.829688] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Connecting -> NetworkFailure ) [disconnected]
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.829793] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Terminating sender thread
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.829829] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Starting sender thread (peer-node-id 1)
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.851942] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Connection closed
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.851952] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.855773] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: helper command: /sbin/drbdadm disconnected exit code 0
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.855817] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( NetworkFailure -> Unconnected ) [disconnected]
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.855823] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Restarting receiver thread
Feb 17 18:35:52 xencc-hp03 kernel: [1944531.855827] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Unconnected -> Connecting ) [connecting]
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.222169] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Handshake to peer 2 successful: Agreed network protocol version 122
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.222172] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.223164] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.227531] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Preparing cluster-wide state change 2189147752: 0->2 role( Secondary ) conn( Connected )
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.291565] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-2: self 03D475EC201461F8:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:0
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.291568] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-2: peer's exposed UUID: 03D475EC201461F8
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.291577] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: State change 2189147752: primary_nodes=0, weak_nodes=0
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.291579] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Committing cluster-wide state change 2189147752 (64ms)
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.291610] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-2: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.291613] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-2: pdsk( DUnknown -> Diskless ) repl( Off -> Established ) [connected]
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.417992] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Handshake to peer 1 successful: Agreed network protocol version 122
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.417995] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Feature flags enabled on protocol level: 0x7f TRIM THIN_RESYNC WRITE_SAME WRITE_ZEROES RESYNC_DAGTAG
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.418886] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: Peer authenticated using 20 bytes HMAC
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.419604] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Preparing cluster-wide state change 319696834: 0->1 role( Secondary ) conn( Connected )
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483614] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-1: drbd_sync_handshake:
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483618] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-1: self 03D475EC201461F8:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:120
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483620] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-1: peer 03D475EC201461F8:0000000000000000:0000000000000000:0000000000000000 bits:0 flags:1120
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483623] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-1: uuid_compare()=no-sync by rule=reconnected
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483648] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: State change 319696834: primary_nodes=0, weak_nodes=0
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483650] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278: Committing cluster-wide state change 319696834 (64ms)
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483677] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278 xenrt-1: conn( Connecting -> Connected ) peer( Unknown -> Secondary ) [connected]
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483679] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007: quorum( no -> yes ) [connected]
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483681] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-1: pdsk( DUnknown -> UpToDate ) repl( Off -> Established ) [connected]
Feb 17 18:35:52 xencc-hp03 kernel: [1944532.483786] drbd xcp-volume-9dd3dc66-aa58-40f2-aa56-14b8846a4278/0 drbd1007 xenrt-1: cleared bm UUID and bitmap 03D475EC201461F8:0000000000000000:0000000000000000:0000000000000000