How can I determine which VM is causing this problem?

Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.830066:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.830198:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.830311:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.830458:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.830605:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.830712:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.830797:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.830885:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.831006:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.831094:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.831185:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.831260:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.831333:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.831407:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.831478:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.831551:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.884264:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.884519:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.884615:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.884721:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.923891:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.924054:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.924207:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.924319:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.924505:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.924675:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.924803:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.924913:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.927026:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.927185:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.927303:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.927436:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.927657:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.927772:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.927903:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.927993:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.928136:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.928255:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.928396:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail2
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.928479:xen_platform_log xen platform: XENBUS|CacheCreateObject: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.932786:xen_platform_log xen platform: XENVIF|__AllocatePages: fail1 (c0000017)
Nov  1 13:05:00 df-c01-node04 qemu-dm-25[29890]: 29890@1730462700.932906:xen_platform_log xen platform: XENVIF|ReceiverPacketCtor: fail1 (c0000017)

What is dm-12?
Virtual server? how can i check which one?

[1053612.719918] Buffer I/O error on dev dm-12, logical block 6567808, async page read
[1053646.789183] Buffer I/O error on dev dm-12, logical block 6567920, async page r
```ead

@Danp said in Logs Partition Full:

I could be wrong, but to me "I/O error on dev xxx" indicates a hardware issue. On top of that, you continue to have issues with full log partitions and snapshots that aren't properly removed. Therefore, I assumed that you are having an issue with your storage system.

All vm´s running on local SSD storage.
Only backup snapshots running on Equallogic.

@Danp said in Logs Partition Full:

Quoted the above as it seems pertinent to the issue at hand. Do you have support for your Equallogic system? If so, then I would recommend reaching out to the vendor for assistance.

why do you think is Equallogic problem?

What is problem?

[1041538.079027] block tdbb: sector-size: 512/512 capacity: 104857600
[1041614.490832] block tdag: sector-size: 512/512 capacity: 41943040
[1041665.112325] block tdah: sector-size: 512/512 capacity: 83886080
[1041682.920728] block tdal: sector-size: 512/512 capacity: 83886080
[1041687.977055] block tdan: sector-size: 512/512 capacity: 83886080
[1041775.983839] block tdar: sector-size: 512/512 capacity: 314572800
[1041784.022923] block tdat: sector-size: 512/512 capacity: 314572800
[1041788.477265] block tdau: sector-size: 512/512 capacity: 314572800
[1041797.083981] Buffer I/O error on dev dm-65, logical block 13134816, async page read
[1041975.964437] Buffer I/O error on dev dm-65, logical block 13134756, async page read
[1041987.007220] block tdad: sector-size: 512/512 capacity: 52428800
[1042011.762333] block tdad: sector-size: 512/512 capacity: 314572800
[1042025.513177] block tdal: sector-size: 512/512 capacity: 314572800
[1042030.608114] block tdal: sector-size: 512/512 capacity: 314572800
[1042046.800416] block tdad: sector-size: 512/512 capacity: 52428800
[1042051.052154] block tdah: sector-size: 512/512 capacity: 52428800
[1042060.751052] block tdal: sector-size: 512/512 capacity: 52428800
[1042075.720653] block tdan: sector-size: 512/512 capacity: 52428800
[1042092.955679] block tdbb: sector-size: 512/512 capacity: 52428800
[1042098.149160] block tdbe: sector-size: 512/512 capacity: 52428800
[1042163.718485] Buffer I/O error on dev dm-46, logical block 10508192, async page read
[1042234.057144] Buffer I/O error on dev dm-34, logical block 39400446, async page read
[1042533.212350] Buffer I/O error on dev dm-34, logical block 13134816, async page read
[1042721.551045] Buffer I/O error on dev dm-34, logical block 264073, async page read
[1044849.455053] Buffer I/O error on dev dm-12, logical block 39400062, async page read
[1046391.419666] Buffer I/O error on dev dm-12, logical block 3941302, async page read
[1049772.497399] Buffer I/O error on dev dm-12, logical block 6567872, async page read
[1049857.595545] Buffer I/O error on dev dm-12, logical block 6567550, async page read
[1049929.102838] Buffer I/O error on dev dm-12, logical block 6567822, async page read
[1050167.988714] Buffer I/O error on dev dm-12, logical block 26267563, async page read
[1050366.554847] Buffer I/O error on dev dm-12, logical block 6567862, async page read
[1050776.365052] Buffer I/O error on dev dm-12, logical block 65665963, async page read
[1051056.348013] Buffer I/O error on dev dm-12, logical block 5255136, async page read
[1051092.751391] Buffer I/O error on dev dm-12, logical block 13134724, async page read
[1051328.387483] Buffer I/O error on dev dm-12, logical block 3941344, async page read
[1051711.573576] Buffer I/O error on dev dm-12, logical block 13134752, async page read
[1051848.129739] Buffer I/O error on dev dm-12, logical block 6567844, async page read
[1051992.984716] Buffer I/O error on dev dm-12, logical block 105064334, async page read
[1052434.107654] Buffer I/O error on dev dm-12, logical block 39400326, async page read
[1052695.987730] Buffer I/O error on dev dm-12, logical block 13134724, async page read
[1052923.659130] Buffer I/O error on dev dm-12, logical block 13134726, async page read
[1053136.646307] Buffer I/O error on dev dm-12, logical block 64153536, async page read
[1053612.719918] Buffer I/O error on dev dm-12, logical block 6567808, async page read
[1053646.789183] Buffer I/O error on dev dm-12, logical block 6567920, async page read
[1053778.875359] Buffer I/O error on dev dm-12, logical block 6567808, async page read
[1053838.326806] Buffer I/O error on dev dm-12, logical block 10508203, async page read
[1054000.750328] Buffer I/O error on dev dm-12, logical block 5255076, async page read
[1054451.772637] Buffer I/O error on dev dm-12, logical block 6567814, async page read
[1103083.191699] device vif7.0 left promiscuous mode
[1103208.344685] block tdh: sector-size: 512/512 capacity: 104857600
[1103217.894245] block tdh: sector-size: 512/512 capacity: 104857600
[1103218.758711] device vif47.0 entered promiscuous mode
[1103219.717538] vif vif-47-0 vif47.0: Guest Rx ready
[1103286.054941] device vif11.0 left promiscuous mode
[1103306.933523] block tdk: sector-size: 512/512 capacity: 104857600
[1103316.719467] block tdk: sector-size: 512/512 capacity: 104857600
[1103317.536966] device vif48.0 entered promiscuous mode
[1103319.391521] vif vif-48-0 vif48.0: Guest Rx ready

Thanks, that help!!!

??

waht is command?

@stormi said in Logs Partition Full:

lsof +L1

[10:38 df-c01-node04 ~]# lsof +L1
COMMAND     PID USER   FD   TYPE DEVICE   SIZE/OFF NLINK     NODE NAME
rsyslogd   1070 root   11w   REG    8,5 1945839857     0       15 /var/log/xensource.log (deleted)
monitor    2002 root    7u   REG    8,1        141     0   180233 /tmp/tmpf6sKcHH (deleted)
ovsdb-ser  2003 root    7u   REG    8,1        141     0   180233 /tmp/tmpf6sKcHH (deleted)
stunnel   13458 root    2w   REG   0,19        861     0 97301510 /run/nonpersistent/forkexecd/stunnelcd2b39.log (deleted)
[17:21 df-c01-node04 ~]#

@stormi said in Logs Partition Full:

xe-toolstack-restart

no help! what can I do now?