Troubleshooting "TCP: out of memory" - Possible memory leak?
-
Hello
This is an early start of a thread for documenting a potential issue. I will add to it with findings accordinly.
I'm having some issues with on XOCE instance, commit
7a187. Three times this week, it has gotten quite unresponsive with network trafic. And looking at the console give clues as to why:
"TCP: out of memory -- consider tuning tcp_mem"
The weird part is that this is my most beefy XOCE instance of all.
4 cores, 8GiB, management agent 10 (Citrix), Rocky 10.2.In troubleshooting I tried restarting the XO-server service. And this seems to have resolved the issue for now. But TBH, I was looking at several point at the same time. So it is possible that it had resolved itself (unlikely but possible) whilst I was troubleshooting and reading up on
tcp_mem.My other XOCE instances are setup identically, and are actually a little bit smaller in the amount of RAM assigned. And they show no problems.
Failing XO instance connects to three separate 1Host pools.
Working XO instance connects to 1 HA-pool with three hosts.Installation is done by a port of https://github.com/cloudrootab/ansible_role_xoce, adjusted for Rocky10 instead of Ubuntu/Debian.
Edit: I should also add that this XOCE has shown the same behaviour on both a recently updated XCP-ng host, and one that is two months behind on updates. Running on Dell Inc. PowerEdge R670 for each of the hosts (RAM on hosts has plenty to spare, roughly 50% utilization)
-
Adding the initial memstats I was looking at whilst trying to troubleshoot
$ cat /proc/net/sockstat sockets: used 3478 TCP: inuse 37 orphan 0 tw 25 alloc 3149 mem 194631 UDP: inuse 5 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 $ cat /proc/sys/net/ipv4/tcp_mem 90147 120196 180294Output of
sudo ssNetid State Recv-Q Send-Q Local Address:Port Peer Address:Port u_str ESTAB 0 0 * 10292 * 10293 u_str ESTAB 0 0 /var/lib/sss/pipes/nss 3260817 * 3261568 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 12062 * 12838 u_str ESTAB 0 0 * 11638 * 11639 u_str ESTAB 0 0 * 13510 * 11079 u_str ESTAB 0 0 * 11624 * 11625 u_dgr ESTAB 0 768 * 10128 * 10127 u_dgr ESTAB 0 0 * 6593 * 6594 u_str ESTAB 0 0 * 5011 * 7454 u_str ESTAB 0 0 * 11692 * 11693 u_str ESTAB 0 0 * 11641 * 11642 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 11295 * 11294 u_dgr ESTAB 0 0 * 6309 * 5237 u_str ESTAB 0 0 * 3261580 * 3260236 u_str ESTAB 0 0 * 3259924 * 3259925 u_dgr ESTAB 0 0 * 1806084 * 1806083 u_str ESTAB 0 0 * 12904 * 11125 u_dgr ESTAB 0 0 * 3545 * 3544 u_str ESTAB 0 0 * 3259925 * 3259924 u_str ESTAB 0 0 * 11666 * 11665 u_str ESTAB 0 0 /run/systemd/journal/stdout 7622 * 6577 u_str ESTAB 0 0 * 11660 * 11659 u_str ESTAB 0 0 /run/systemd/journal/stdout 7991 * 9524 u_str ESTAB 0 0 * 1806062 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 7914 * 7035 u_str ESTAB 0 0 /run/systemd/journal/stdout 6995 * 9451 u_dgr ESTAB 0 0 * 1806086 * 1806085 u_str ESTAB 0 0 * 13479 * 13480 u_dgr ESTAB 0 0 * 4829 * 4828 u_str ESTAB 0 0 * 3261568 * 3260817 u_dgr ESTAB 0 0 * 11586 * 5236 u_dgr ESTAB 0 0 * 7352 * 3543 u_str ESTAB 0 0 * 11647 * 11648 u_str ESTAB 0 0 * 7891 * 7892 u_dgr ESTAB 0 0 /run/systemd/notify 3543 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 3258766 * 3259810 u_str ESTAB 0 0 /run/dbus/system_bus_socket 12873 * 12872 u_str ESTAB 0 0 * 10290 * 10291 u_dgr ESTAB 0 0 * 1806088 * 1806087 u_str ESTAB 0 0 * 11625 * 11624 u_dgr ESTAB 0 0 * 12335 * 5236 u_str ESTAB 0 0 * 11651 * 11650 u_dgr ESTAB 0 0 /run/systemd/journal/dev-log 5236 * 0 u_dgr ESTAB 0 0 /run/systemd/journal/socket 5237 * 0 u_dgr ESTAB 0 0 * 3546 * 3547 u_str ESTAB 0 0 * 11632 * 11631 u_str ESTAB 0 0 * 11648 * 11647 u_str ESTAB 0 0 * 6974 * 6973 u_str ESTAB 0 0 * 12111 * 0 u_str ESTAB 0 0 * 3259676 * 3258625 u_str ESTAB 0 0 * 11675 * 11674 u_str ESTAB 0 0 * 11674 * 11675 u_str ESTAB 0 0 /run/systemd/journal/stdout 9419 * 9418 u_str ESTAB 0 0 * 11622 * 11621 u_dgr ESTAB 0 0 * 10127 * 10128 u_str ESTAB 0 0 * 7908 * 7019 u_str ESTAB 0 0 * 3259922 * 3259923 u_str ESTAB 0 0 * 3259921 * 3259920 u_str ESTAB 0 0 /run/systemd/journal/stdout 11068 * 13495 u_dgr ESTAB 0 0 * 10469 * 5236 u_dgr ESTAB 0 0 * 5032 * 5237 u_str ESTAB 0 0 /run/dbus/system_bus_socket 1805048 * 1806090 u_str ESTAB 0 0 * 1805720 * 0 u_str ESTAB 0 0 * 10298 * 10299 u_str ESTAB 0 0 * 11689 * 11690 u_dgr ESTAB 0 0 * 1806087 * 1806088 u_str ESTAB 0 0 * 11290 * 11291 u_dgr ESTAB 0 0 * 11701 * 5236 u_str ESTAB 0 0 * 11693 * 11692 u_str ESTAB 0 0 * 13495 * 11068 u_dgr ESTAB 0 0 * 7081 * 5236 u_str ESTAB 0 0 * 11636 * 11635 u_str ESTAB 0 0 * 8975 * 9386 u_str ESTAB 0 0 @/run/rpcbind.sock 1805023 * 1805767 u_str ESTAB 0 0 /run/systemd/journal/stdout 9214 * 9661 u_dgr ESTAB 0 0 * 1806069 * 5237 u_str ESTAB 0 0 * 11621 * 11622 u_dgr ESTAB 0 0 /run/chrony/chronyd.sock 7086 * 0 u_str ESTAB 0 0 * 11680 * 11681 u_dgr ESTAB 0 0 * 3261581 * 5236 u_str ESTAB 0 0 /run/dbus/system_bus_socket 10159 * 10158 u_dgr ESTAB 0 0 * 6594 * 6593 u_str ESTAB 0 0 /var/lib/sss/pipes/pam 3260236 * 3261580 u_str ESTAB 0 0 * 10789 * 10790 u_str ESTAB 0 0 * 9451 * 6995 u_str ESTAB 0 0 * 11678 * 11677 u_str ESTAB 0 0 * 11668 * 11669 u_dgr ESTAB 0 0 * 6589 * 5237 u_str ESTAB 0 0 * 11644 * 11645 u_str ESTAB 0 0 * 11642 * 11641 u_dgr ESTAB 0 768 * 9007 * 9006 u_str ESTAB 0 0 * 1806581 * 1806580 u_str ESTAB 0 0 * 1806580 * 1806581 u_str ESTAB 0 0 /run/systemd/journal/stdout 10293 * 10292 u_str ESTAB 0 0 /run/systemd/journal/stdout 10291 * 10290 u_str ESTAB 0 0 * 3261597 * 3261598 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 12063 * 13475 u_str ESTAB 0 0 * 11662 * 11663 u_str ESTAB 0 0 * 7035 * 7914 u_str ESTAB 0 0 /run/systemd/journal/stdout 7454 * 5011 u_str ESTAB 0 0 * 1805796 * 0 u_str ESTAB 0 0 * 3260520 * 0 u_str ESTAB 0 0 * 11657 * 11656 u_dgr ESTAB 0 0 * 1806083 * 1806084 u_str ESTAB 0 0 * 12624 * 12623 u_str ESTAB 0 0 * 9418 * 9419 u_dgr ESTAB 0 0 * 3258338 * 5237 u_str ESTAB 0 0 * 1805715 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 10299 * 10298 u_str ESTAB 0 0 * 11695 * 11696 u_str ESTAB 0 0 * 11681 * 11680 u_str ESTAB 0 0 * 3259838 * 0 u_str ESTAB 0 0 * 11631 * 11632 u_dgr ESTAB 0 0 * 12856 * 5237 u_str ESTAB 0 0 * 11653 * 11654 u_str ESTAB 0 0 /run/systemd/journal/stdout 9386 * 8975 u_str ESTAB 0 0 * 11665 * 11666 u_str ESTAB 0 0 * 9489 * 7097 u_dgr ESTAB 0 0 * 4828 * 4829 u_dgr ESTAB 0 0 * 12905 * 5236 u_str ESTAB 0 0 * 13475 * 12063 u_str ESTAB 0 0 /run/dbus/system_bus_socket 6990 * 7893 u_str ESTAB 0 0 * 11650 * 11651 u_str ESTAB 0 0 * 9661 * 9214 u_str ESTAB 0 0 * 3258205 * 0 u_str ESTAB 0 0 * 11669 * 11668 u_str ESTAB 0 0 * 11294 * 11295 u_dgr ESTAB 0 0 * 9089 * 5236 u_str ESTAB 0 0 * 6577 * 7622 u_str ESTAB 0 0 * 3259920 * 3259921 u_dgr ESTAB 0 0 * 1806577 * 5236 u_str ESTAB 0 0 * 12623 * 12624 u_str ESTAB 0 0 * 11656 * 11657 u_str ESTAB 0 0 /run/systemd/journal/stdout 3258625 * 3259676 u_str ESTAB 0 0 * 11686 * 11687 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 11291 * 11290 u_str ESTAB 0 0 * 10790 * 10789 u_dgr ESTAB 0 0 * 9427 * 5237 u_str ESTAB 0 0 /run/dbus/system_bus_socket 6988 * 9429 u_str ESTAB 0 0 * 11683 * 11684 u_str ESTAB 0 0 * 3259923 * 3259922 u_str ESTAB 0 0 * 3259919 * 3259918 u_dgr ESTAB 0 0 * 3260669 * 5237 u_dgr ESTAB 0 0 * 1806061 * 5236 u_str ESTAB 0 0 /run/systemd/journal/stdout 11079 * 13510 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 13480 * 13479 u_str ESTAB 0 0 * 11639 * 11638 u_str ESTAB 0 0 * 11635 * 11636 u_dgr ESTAB 0 0 * 3544 * 3545 u_str ESTAB 0 0 * 11690 * 11689 u_str ESTAB 0 0 * 11672 * 11671 u_str ESTAB 0 0 /run/systemd/journal/stdout 11125 * 12904 u_dgr ESTAB 0 0 * 1806085 * 1806086 u_dgr ESTAB 0 0 * 3547 * 3546 u_str ESTAB 0 0 * 11696 * 11695 u_str ESTAB 0 0 * 3259810 * 3258766 u_dgr ESTAB 0 0 * 1795630 * 5236 u_str ESTAB 0 0 * 11654 * 11653 u_str ESTAB 0 0 * 1805767 * 1805023 u_str ESTAB 0 0 * 10688 * 10121 u_str ESTAB 0 0 * 11687 * 11686 u_str ESTAB 0 0 * 11671 * 11672 u_str ESTAB 0 0 * 12838 * 12062 u_str ESTAB 0 0 * 11629 * 11628 u_str ESTAB 0 0 * 9524 * 7991 u_str ESTAB 0 0 * 7892 * 7891 u_str ESTAB 0 0 * 11659 * 11660 u_str ESTAB 0 0 * 11645 * 11644 u_dgr ESTAB 0 0 * 9706 * 5236 u_dgr ESTAB 0 0 * 9006 * 9007 u_str ESTAB 0 0 * 10158 * 10159 u_dgr ESTAB 0 0 * 6972 * 5236 u_str ESTAB 0 0 /run/dbus/system_bus_socket 10121 * 10688 u_str ESTAB 0 0 * 1806041 * 1805036 u_str ESTAB 0 0 /run/systemd/journal/stdout 9213 * 7140 u_str ESTAB 0 0 /run/dbus/system_bus_socket 7097 * 9489 u_str ESTAB 0 0 /run/systemd/journal/stdout 7019 * 7908 u_str ESTAB 0 0 * 6973 * 6974 u_str ESTAB 0 0 * 11684 * 11683 u_str ESTAB 0 0 * 11663 * 11662 u_str ESTAB 0 0 * 1795634 * 0 u_dgr ESTAB 0 0 * 3260114 * 5237 u_str ESTAB 0 0 * 3261598 * 3261597 u_str ESTAB 0 0 * 3259918 * 3259919 u_str ESTAB 0 0 /run/systemd/journal/stdout 1805036 * 1806041 u_str ESTAB 0 0 * 11628 * 11629 u_str ESTAB 0 0 * 7140 * 9213 u_str ESTAB 0 0 * 11677 * 11678 u_str ESTAB 0 0 @f5c12eea9342a377/bus/systemd-logind/system 12872 * 12873 u_str ESTAB 0 0 @ed019b0aea426515/bus/systemd/bus-system 1806090 * 1805048 u_str ESTAB 0 0 @3cbcd21433f31955/bus/dbus-broker-lau/system 7893 * 6990 u_str ESTAB 0 0 @8aa9cf87f9fa122a/bus/systemd/bus-api-system 9429 * 6988 icmp6 UNCONN 0 0 *:ipv6-icmp *:* tcp ESTAB 0 0 <obfuscated>:952 <obfuscated>:nfs tcp ESTAB 0 36 <obfuscated>:ssh <obfuscated>:53277 tcp ESTAB 0 0 <obfuscated>:43916 <obfuscated>:ldaps tcp ESTAB 0 0 <obfuscated>:54296 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:40380 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:33806 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:39428 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:58586 <obfuscated>:30514 tcp ESTAB 0 0 [::1]:50086 [::1]:redis tcp ESTAB 0 0 [::1]:pcsync-https [::1]:35174 tcp ESTAB 0 0 [::ffff:<obfuscated>]:pcsync-https [::ffff:192.168.132.166]:50136 tcp ESTAB 0 0 [::1]:35174 [::1]:pcsync-https tcp CLOSE-WAIT 401224 0 [::1]:42184 [::1]:9004 tcp CLOSE-WAIT 339569 0 [::1]:34522 [::1]:9004 tcp CLOSE-WAIT 401329 0 [::1]:47524 [::1]:9004 tcp ESTAB 0 0 [::1]:redis [::1]:50086 tcp CLOSE-WAIT 339562 0 [::1]:56834 [::1]:9004 tcp ESTAB 0 0 [::ffff:<obfuscated>]:pcsync-https [::ffff:192.168.132.166]:54998 tcp ESTAB 0 0 [::1]:pcsync-https [::1]:35184 tcp ESTAB 0 0 [::1]:35184 [::1]:pcsync-https tcp FIN-WAIT-2 0 0 [::1]:9004 [::1]:42728 tcp CLOSE-WAIT 339529 0 [::1]:33512 [::1]:9004 tcp CLOSE-WAIT 401208 0 [::1]:42728 [::1]:9004After latest
dnf upgradepost service restart, and subsequent reboot for new kernel$ cat /proc/net/sockstat sockets: used 339 TCP: inuse 24 orphan 0 tw 36 alloc 47 mem 238 UDP: inuse 5 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 $ cat /proc/sys/net/ipv4/tcp_mem 90147 120196 180294 -
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login