What is dm-12?
Virtual server? how can i check which one?
[1053612.719918] Buffer I/O error on dev dm-12, logical block 6567808, async page read
[1053646.789183] Buffer I/O error on dev dm-12, logical block 6567920, async page r
```ead
What is dm-12?
Virtual server? how can i check which one?
[1053612.719918] Buffer I/O error on dev dm-12, logical block 6567808, async page read
[1053646.789183] Buffer I/O error on dev dm-12, logical block 6567920, async page r
```ead
@Danp said in Logs Partition Full:
I could be wrong, but to me "I/O error on dev xxx" indicates a hardware issue. On top of that, you continue to have issues with full log partitions and snapshots that aren't properly removed. Therefore, I assumed that you are having an issue with your storage system.
All vm´s running on local SSD storage.
Only backup snapshots running on Equallogic.
@Danp said in Logs Partition Full:
Quoted the above as it seems pertinent to the issue at hand. Do you have support for your Equallogic system? If so, then I would recommend reaching out to the vendor for assistance.
why do you think is Equallogic problem?
What is problem?
[1041538.079027] block tdbb: sector-size: 512/512 capacity: 104857600
[1041614.490832] block tdag: sector-size: 512/512 capacity: 41943040
[1041665.112325] block tdah: sector-size: 512/512 capacity: 83886080
[1041682.920728] block tdal: sector-size: 512/512 capacity: 83886080
[1041687.977055] block tdan: sector-size: 512/512 capacity: 83886080
[1041775.983839] block tdar: sector-size: 512/512 capacity: 314572800
[1041784.022923] block tdat: sector-size: 512/512 capacity: 314572800
[1041788.477265] block tdau: sector-size: 512/512 capacity: 314572800
[1041797.083981] Buffer I/O error on dev dm-65, logical block 13134816, async page read
[1041975.964437] Buffer I/O error on dev dm-65, logical block 13134756, async page read
[1041987.007220] block tdad: sector-size: 512/512 capacity: 52428800
[1042011.762333] block tdad: sector-size: 512/512 capacity: 314572800
[1042025.513177] block tdal: sector-size: 512/512 capacity: 314572800
[1042030.608114] block tdal: sector-size: 512/512 capacity: 314572800
[1042046.800416] block tdad: sector-size: 512/512 capacity: 52428800
[1042051.052154] block tdah: sector-size: 512/512 capacity: 52428800
[1042060.751052] block tdal: sector-size: 512/512 capacity: 52428800
[1042075.720653] block tdan: sector-size: 512/512 capacity: 52428800
[1042092.955679] block tdbb: sector-size: 512/512 capacity: 52428800
[1042098.149160] block tdbe: sector-size: 512/512 capacity: 52428800
[1042163.718485] Buffer I/O error on dev dm-46, logical block 10508192, async page read
[1042234.057144] Buffer I/O error on dev dm-34, logical block 39400446, async page read
[1042533.212350] Buffer I/O error on dev dm-34, logical block 13134816, async page read
[1042721.551045] Buffer I/O error on dev dm-34, logical block 264073, async page read
[1044849.455053] Buffer I/O error on dev dm-12, logical block 39400062, async page read
[1046391.419666] Buffer I/O error on dev dm-12, logical block 3941302, async page read
[1049772.497399] Buffer I/O error on dev dm-12, logical block 6567872, async page read
[1049857.595545] Buffer I/O error on dev dm-12, logical block 6567550, async page read
[1049929.102838] Buffer I/O error on dev dm-12, logical block 6567822, async page read
[1050167.988714] Buffer I/O error on dev dm-12, logical block 26267563, async page read
[1050366.554847] Buffer I/O error on dev dm-12, logical block 6567862, async page read
[1050776.365052] Buffer I/O error on dev dm-12, logical block 65665963, async page read
[1051056.348013] Buffer I/O error on dev dm-12, logical block 5255136, async page read
[1051092.751391] Buffer I/O error on dev dm-12, logical block 13134724, async page read
[1051328.387483] Buffer I/O error on dev dm-12, logical block 3941344, async page read
[1051711.573576] Buffer I/O error on dev dm-12, logical block 13134752, async page read
[1051848.129739] Buffer I/O error on dev dm-12, logical block 6567844, async page read
[1051992.984716] Buffer I/O error on dev dm-12, logical block 105064334, async page read
[1052434.107654] Buffer I/O error on dev dm-12, logical block 39400326, async page read
[1052695.987730] Buffer I/O error on dev dm-12, logical block 13134724, async page read
[1052923.659130] Buffer I/O error on dev dm-12, logical block 13134726, async page read
[1053136.646307] Buffer I/O error on dev dm-12, logical block 64153536, async page read
[1053612.719918] Buffer I/O error on dev dm-12, logical block 6567808, async page read
[1053646.789183] Buffer I/O error on dev dm-12, logical block 6567920, async page read
[1053778.875359] Buffer I/O error on dev dm-12, logical block 6567808, async page read
[1053838.326806] Buffer I/O error on dev dm-12, logical block 10508203, async page read
[1054000.750328] Buffer I/O error on dev dm-12, logical block 5255076, async page read
[1054451.772637] Buffer I/O error on dev dm-12, logical block 6567814, async page read
[1103083.191699] device vif7.0 left promiscuous mode
[1103208.344685] block tdh: sector-size: 512/512 capacity: 104857600
[1103217.894245] block tdh: sector-size: 512/512 capacity: 104857600
[1103218.758711] device vif47.0 entered promiscuous mode
[1103219.717538] vif vif-47-0 vif47.0: Guest Rx ready
[1103286.054941] device vif11.0 left promiscuous mode
[1103306.933523] block tdk: sector-size: 512/512 capacity: 104857600
[1103316.719467] block tdk: sector-size: 512/512 capacity: 104857600
[1103317.536966] device vif48.0 entered promiscuous mode
[1103319.391521] vif vif-48-0 vif48.0: Guest Rx ready
@stormi said in Logs Partition Full:
lsof +L1
[10:38 df-c01-node04 ~]# lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
rsyslogd 1070 root 11w REG 8,5 1945839857 0 15 /var/log/xensource.log (deleted)
monitor 2002 root 7u REG 8,1 141 0 180233 /tmp/tmpf6sKcHH (deleted)
ovsdb-ser 2003 root 7u REG 8,1 141 0 180233 /tmp/tmpf6sKcHH (deleted)
stunnel 13458 root 2w REG 0,19 861 0 97301510 /run/nonpersistent/forkexecd/stunnelcd2b39.log (deleted)
[17:21 df-c01-node04 ~]#
xensource.log is to big 2 GB
Im delete logfile, but disk space ist not free. Now cant see xensource.log, why?
Idea what is problem?
[20:31 df-c01-node04 log]# tail -50 xensource.log
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|16 ||xenops_server] TASK.signal 599093 (object deleted)
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|22 ||xenops_server] Queue.pop returned ["VM_check_state","f828ce90-06e0-024f-9c9b-3f30b1a959b4"]
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|22 |events|xenops_server] Task 599094 reference events: ["VM_check_state","f828ce90-06e0-024f-9c9b-3f30b1a959b4"]
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|4 |events|xenops_server] Received an event on managed VM f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|4 |queue|xenops_server] Queue.push ["VM_check_state","f828ce90-06e0-024f-9c9b-3f30b1a959b4"] onto redirected f828ce90-06e0-024f-9c9b-3f30b1a959b4:[ ]
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] xenopsd event: processing event for VM f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] Will update VM.allowed_operations because guest_agent has changed.
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] xenopsd event: Updating VM f828ce90-06e0-024f-9c9b-3f30b1a959b4 domid 9 guest_agent
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404650 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.slave_login D:d1538d321269 created by task D:c22abe907392
Jul 1 20:32:06 df-c01-node04 xapi: [ info|df-c01-node04|2404650 UNIX /var/lib/xcp/xapi|session.slave_login D:8d7d2a8f0ca6|xapi] Session.create trackid=59cc70e9542b52fdcd1622725fa443c3 pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|4 |events|xenops_server] Received an event on managed VM f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|4 |queue|xenops_server] Queue.push ["VM_check_state","f828ce90-06e0-024f-9c9b-3f30b1a959b4"] onto redirected f828ce90-06e0-024f-9c9b-3f30b1a959b4:[ ["VM_check_state","f828ce90-06e0-024f-9c9b-3f30b1a959b4"] ]
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404651 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:6320774722f1 created by task D:8d7d2a8f0ca6
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404652 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VM.update_allowed_operations D:e5300a15f1ea created by task D:c22abe907392
Jul 1 20:32:06 df-c01-node04 xapi: [ info|df-c01-node04|2404652 UNIX /var/lib/xcp/xapi|dispatch:VM.update_allowed_operations D:e5300a15f1ea|taskhelper] task VM.update_allowed_operations R:2852f822293e (uuid:ecd676c2-0a35-ad5c-7f38-48fc3e383d52) created (trackid=59cc70e9542b52fdcd1622725fa443c3) by task D:c22abe907392
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404652 UNIX /var/lib/xcp/xapi|VM.update_allowed_operations R:2852f822293e|audit] VM.update_allowed_operations: VM = 'f828ce90-06e0-024f-9c9b-3f30b1a959b4 (DF-server24.df-webhosting.de)'
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|22 |events|xenops_server] VM f828ce90-06e0-024f-9c9b-3f30b1a959b4 is not requesting any attention
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|22 |events|xenops_server] VM_DB.signal f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|22 |events|task_server] Task 599094 completed; duration = 0
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|22 ||xenops_server] TASK.signal 599094 (object deleted)
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|39 ||xenops_server] Queue.pop returned ["VM_check_state","f828ce90-06e0-024f-9c9b-3f30b1a959b4"]
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|39 |events|xenops_server] Task 599095 reference events: ["VM_check_state","f828ce90-06e0-024f-9c9b-3f30b1a959b4"]
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404653 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:e038fd45a92c created by task D:c22abe907392
Jul 1 20:32:06 df-c01-node04 xapi: [ info|df-c01-node04|2404653 UNIX /var/lib/xcp/xapi|session.logout D:624f223cdcff|xapi] Session.destroy trackid=59cc70e9542b52fdcd1622725fa443c3
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] Processing event: ["Vm","f828ce90-06e0-024f-9c9b-3f30b1a959b4"]
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] xenops event on VM f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|632725 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops_server] VM.stat f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|39 |events|xenops_server] VM f828ce90-06e0-024f-9c9b-3f30b1a959b4 is not requesting any attention
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|39 |events|xenops_server] VM_DB.signal f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|39 |events|task_server] Task 599095 completed; duration = 0
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|39 ||xenops_server] TASK.signal 599095 (object deleted)
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] xenopsd event: processing event for VM f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] Will update VM.allowed_operations because guest_agent has changed.
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] xenopsd event: Updating VM f828ce90-06e0-024f-9c9b-3f30b1a959b4 domid 9 guest_agent
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404654 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.slave_login D:3041ad7a5d1d created by task D:c22abe907392
Jul 1 20:32:06 df-c01-node04 xapi: [ info|df-c01-node04|2404654 UNIX /var/lib/xcp/xapi|session.slave_login D:fa655deb1721|xapi] Session.create trackid=aa48585d92a36631054ef9468218522c pool=true uname= originator=xapi is_local_superuser=true auth_user_sid= parent=trackid=9834f5af41c964e225f24279aefe4e49
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404655 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:pool.get_all D:a1e04c4c79aa created by task D:fa655deb1721
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1235 |xapi events D:f60b314e49a9|dummytaskhelper] task timeboxed_rpc D:98ed995dc8af created by task D:f60b314e49a9
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404656 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:event.from D:c9f2b3e839ca created by task D:f60b314e49a9
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404657 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:VM.update_allowed_operations D:4a10137d6bc3 created by task D:c22abe907392
Jul 1 20:32:06 df-c01-node04 xapi: [ info|df-c01-node04|2404657 UNIX /var/lib/xcp/xapi|dispatch:VM.update_allowed_operations D:4a10137d6bc3|taskhelper] task VM.update_allowed_operations R:e4f434e89b02 (uuid:3e5a0652-8d8f-b774-53d9-f325c4cc63a1) created (trackid=aa48585d92a36631054ef9468218522c) by task D:c22abe907392
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404657 UNIX /var/lib/xcp/xapi|VM.update_allowed_operations R:e4f434e89b02|audit] VM.update_allowed_operations: VM = 'f828ce90-06e0-024f-9c9b-3f30b1a959b4 (DF-server24.df-webhosting.de)'
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|2404658 UNIX /var/lib/xcp/xapi||dummytaskhelper] task dispatch:session.logout D:9eeefa7124f9 created by task D:c22abe907392
Jul 1 20:32:06 df-c01-node04 xapi: [ info|df-c01-node04|2404658 UNIX /var/lib/xcp/xapi|session.logout D:b558e450cae3|xapi] Session.destroy trackid=aa48585d92a36631054ef9468218522c
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] Processing event: ["Vm","f828ce90-06e0-024f-9c9b-3f30b1a959b4"]
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] xenops event on VM f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xenopsd-xc: [debug|df-c01-node04|632727 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops_server] VM.stat f828ce90-06e0-024f-9c9b-3f30b1a959b4
Jul 1 20:32:06 df-c01-node04 xapi: [debug|df-c01-node04|1230 |org.xen.xapi.xenops.classic events D:c22abe907392|xenops] xenopsd event: ignoring event for VM f828ce90-06e0-024f-9c9b-3f30b1a959b4: metadata has not changed
Jul 1 20:32:06 df-c01-node04 xcp-rrdd: [ warn|df-c01-node04|0 monitor|main|rrdd_server] setting skip-cycles-after-error for plugin tap-31773-25 to 256
Jul 1 20:32:06 df-c01-node04 xcp-rrdd: [ warn|df-c01-node04|0 monitor|main|rrdd_server] Failed to process plugin: tap-31773-25 (Rrd_protocol.Invalid_header_string)
[20:32 df-c01-node04 log]#
@karlisi said in Backup Continuous Replication hängt!:
Orphaned
must delete every day orphaned snapshots!
No solution for this Problem?
that means every pool-master has this problem
@karlisi said in Backup Continuous Replication hängt!:
I have the same problem since... I don't remember, a year perhaps. Orphaned snapshots are some way related to continuous replication, they appears randomly sometimes on almost all VMs, sometimes none. For me this is not a problem, I delete them as part of daily routine.
I am using XOCE, xo-server and xo-web 5.59.0.
This problem have only pool-master node, other node does not have the problem.
I do not understand!
Every day have pool-master orpharned disks and I delete it.
Why does that happen?
@stormi said in Logs Partition Full:
du -sh /var/log/*
[5.1M /var/log/audit.log
24M /var/log/audit.log.1
0 /var/log/audit.log.1.gz
3.5M /var/log/audit.log.2.gz
0 /var/log/audit.log.3.gz
9.3M /var/log/blktap
0 /var/log/boot.log
24K /var/log/boot.log.1
4.0K /var/log/boot.log.2.gz
4.0K /var/log/boot.log.3.gz
4.0K /var/log/boot.log.4.gz
4.0K /var/log/boot.log.5.gz
4.0K /var/log/btmp
4.0K /var/log/btmp.1
0 /var/log/btmp.1.gz
4.0K /var/log/cluster
8.0K /var/log/crit.log
0 /var/log/crit.log.1
0 /var/log/crit.log.1.gz
2.9M /var/log/cron
40K /var/log/cron.1
0 /var/log/cron.1.gz
4.0K /var/log/grubby_prune_debug
1004K /var/log/installer
4.0K /var/log/interface-rename.log
0 /var/log/interface-rename.log.1
0 /var/log/interface-rename.log.1.gz
1.3M /var/log/kern.log
20K /var/log/kern.log.1
0 /var/log/kern.log.1.gz
16K /var/log/lost+found
164K /var/log/maillog
4.0K /var/log/maillog.32.gz
0 /var/log/messages
0 /var/log/messages.1
0 /var/log/messages.1.gz
4.0K /var/log/ntpstats
4.0K /var/log/openvswitch
4.0K /var/log/ovs-ctl.log
4.0K /var/log/ovs-ctl.log.32.gz
4.0K /var/log/ovsdb-server.log
4.0K /var/log/ovsdb-server.log.1
0 /var/log/ovsdb-server.log.1.gz
60K /var/log/ovs-vswitchd.log
4.0K /var/log/ovs-vswitchd.log.1
0 /var/log/ovs-vswitchd.log.1.gz
4.0K /var/log/ovs-xapi-sync.log
4.0K /var/log/ovs-xapi-sync.log.32.gz
8.0K /var/log/pbis-open-install.log
4.0K /var/log/pyperthreading-plugin.log
114M /var/log/sa
8.0K /var/log/samba
37M /var/log/secure
5.6M /var/log/SMlog
29M /var/log/SMlog.1
0 /var/log/SMlog.1.gz
5.0M /var/log/SMlog.2.gz
0 /var/log/spooler
4.0K /var/log/spooler.32.gz
0 /var/log/tallylog
0 /var/log/updater-plugin.log
972K /var/log/user.log
4.0K /var/log/user.log.1
0 /var/log/user.log.1.gz
696K /var/log/VMSSlog
20K /var/log/VMSSlog.1
0 /var/log/VMSSlog.1.gz
68K /var/log/wtmp
8.0K /var/log/xcp-rrdd-plugins.log
13M /var/log/xcp-rrdd-plugins.log.1
0 /var/log/xcp-rrdd-plugins.log.1.gz
240K /var/log/xcp-rrdd-plugins.log.2.gz
0 /var/log/xcp-rrdd-plugins.log.3.gz
220K /var/log/xen
96M /var/log/xensource.log
18M /var/log/xensource.log.1
0 /var/log/xensource.log.1.gz
8.2M /var/log/xensource.log.2.gz
9.2M /var/log/xensource.log.3.gz
6.5M /var/log/xensource.log.4.gz
8.0M /var/log/xenstored-access.log
33M /var/log/xenstored-access.log.1
0 /var/log/xenstored-access.log.1.gz
2.3M /var/log/xenstored-access.log.2.gz
4.0K /var/log/yum.log
36K /var/log/yum.log.1
Filesystem Size Used Avail Use% Mounted on
devtmpfs 3.9G 112K 3.9G 1% /dev
tmpfs 3.9G 1004K 3.9G 1% /dev/shm
tmpfs 3.9G 12M 3.9G 1% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
/dev/sda1 18G 2.4G 15G 14% /
xenstore 3.9G 0 3.9G 0% /var/lib/xenstored
/dev/sda5 3.9G 3.9G 0 100% /var/log
Im delete daemon.log, but command df -h say partition is full, why?
/dev/sda5 3.9G 3.9G 0 100% /var/log
@x-rayd said in Logs Partition Full:
squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated
no idea?
last daemon.log
Jun 17 00:23:08 df-c01-node04 squeezed[1275]: [4150926.54] watch /control/feature-balloon <- 1
Jun 17 00:23:08 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /control/feature-balloon <- 1
Jun 17 00:23:08 df-c01-node04 squeezed[1275]: [4150926.58] watch /data/updated <- Wed Jun 17 00:23:08 2020
Jun 17 00:23:08 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:08 2020
Jun 17 00:23:08 df-c01-node04 squeezed[1275]: [4150926.75] watch /data/updated <- Wed Jun 17 00:23:08 CEST 2020
Jun 17 00:23:08 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:08 CEST 2020
Jun 17 00:23:10 df-c01-node04 squeezed[1275]: [4150928.15] watch /data/updated <- Wed Jun 17 00:23:10 2020
Jun 17 00:23:10 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:10 2020
Jun 17 00:23:10 df-c01-node04 squeezed[1275]: [4150928.15] watch /data/updated <- Wed Jun 17 00:23:10 2020
Jun 17 00:23:10 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:10 2020
Jun 17 00:23:16 df-c01-node04 squeezed[1275]: [4150934.19] watch /control/feature-balloon <- 1
Jun 17 00:23:16 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /control/feature-balloon <- 1
Jun 17 00:23:16 df-c01-node04 squeezed[1275]: [4150934.26] watch /data/updated <- Wed Jun 17 00:23:16 2020
Jun 17 00:23:16 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:16 2020
Jun 17 00:23:30 df-c01-node04 squeezed[1275]: [4150948.51] watch /data/updated <- Wed Jun 17 00:23:30 CEST 2020
Jun 17 00:23:30 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:30 CEST 2020
Jun 17 00:23:41 df-c01-node04 squeezed[1275]: [4150958.91] watch /data/updated <- 1
Jun 17 00:23:41 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- 1
Jun 17 00:23:41 df-c01-node04 squeezed[1275]: [4150959.51] watch /data/updated <- Wed Jun 17 00:23:41 CEST 2020
Jun 17 00:23:41 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:41 CEST 2020
Jun 17 00:23:45 df-c01-node04 squeezed[1275]: [4150963.15] watch /control/feature-balloon <- 1
Jun 17 00:23:45 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /control/feature-balloon <- 1
Jun 17 00:23:45 df-c01-node04 squeezed[1275]: [4150963.26] watch /data/updated <- Wed Jun 17 00:23:45 2020
Jun 17 00:23:45 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:45 2020
Jun 17 00:23:45 df-c01-node04 squeezed[1275]: [4150963.33] watch /data/updated <- Wed Jun 17 00:23:45 CEST 2020
Jun 17 00:23:45 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:45 CEST 2020
Jun 17 00:23:50 df-c01-node04 squeezed[1275]: [4150967.96] watch /data/updated <- Wed Jun 17 00:23:50 CEST 2020
Jun 17 00:23:50 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:50 CEST 2020
Jun 17 00:23:52 df-c01-node04 squeezed[1275]: [4150970.10] watch /data/updated <- Wed Jun 17 00:23:52 2020
Jun 17 00:23:52 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xenops] watch /data/updated <- Wed Jun 17 00:23:52 2020
Jun 17 00:23:53 df-c01-node04 squeezed[1275]: [4150971.35] watch /data/updated <- Wed Jun 17 00:23:53 2020
Jun 17 00:23:53 df-c01-node04 squeezed: [debug|df-c01-node04|3 ||xen[16:56 df-c01-node04 log]#
```~~~~