Troubleshooting "TCP: out of memory" - Possible memory leak?
-
Adding the initial memstats I was looking at whilst trying to troubleshoot
$ cat /proc/net/sockstat sockets: used 3478 TCP: inuse 37 orphan 0 tw 25 alloc 3149 mem 194631 UDP: inuse 5 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 $ cat /proc/sys/net/ipv4/tcp_mem 90147 120196 180294Output of
sudo ssNetid State Recv-Q Send-Q Local Address:Port Peer Address:Port u_str ESTAB 0 0 * 10292 * 10293 u_str ESTAB 0 0 /var/lib/sss/pipes/nss 3260817 * 3261568 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 12062 * 12838 u_str ESTAB 0 0 * 11638 * 11639 u_str ESTAB 0 0 * 13510 * 11079 u_str ESTAB 0 0 * 11624 * 11625 u_dgr ESTAB 0 768 * 10128 * 10127 u_dgr ESTAB 0 0 * 6593 * 6594 u_str ESTAB 0 0 * 5011 * 7454 u_str ESTAB 0 0 * 11692 * 11693 u_str ESTAB 0 0 * 11641 * 11642 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 11295 * 11294 u_dgr ESTAB 0 0 * 6309 * 5237 u_str ESTAB 0 0 * 3261580 * 3260236 u_str ESTAB 0 0 * 3259924 * 3259925 u_dgr ESTAB 0 0 * 1806084 * 1806083 u_str ESTAB 0 0 * 12904 * 11125 u_dgr ESTAB 0 0 * 3545 * 3544 u_str ESTAB 0 0 * 3259925 * 3259924 u_str ESTAB 0 0 * 11666 * 11665 u_str ESTAB 0 0 /run/systemd/journal/stdout 7622 * 6577 u_str ESTAB 0 0 * 11660 * 11659 u_str ESTAB 0 0 /run/systemd/journal/stdout 7991 * 9524 u_str ESTAB 0 0 * 1806062 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 7914 * 7035 u_str ESTAB 0 0 /run/systemd/journal/stdout 6995 * 9451 u_dgr ESTAB 0 0 * 1806086 * 1806085 u_str ESTAB 0 0 * 13479 * 13480 u_dgr ESTAB 0 0 * 4829 * 4828 u_str ESTAB 0 0 * 3261568 * 3260817 u_dgr ESTAB 0 0 * 11586 * 5236 u_dgr ESTAB 0 0 * 7352 * 3543 u_str ESTAB 0 0 * 11647 * 11648 u_str ESTAB 0 0 * 7891 * 7892 u_dgr ESTAB 0 0 /run/systemd/notify 3543 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 3258766 * 3259810 u_str ESTAB 0 0 /run/dbus/system_bus_socket 12873 * 12872 u_str ESTAB 0 0 * 10290 * 10291 u_dgr ESTAB 0 0 * 1806088 * 1806087 u_str ESTAB 0 0 * 11625 * 11624 u_dgr ESTAB 0 0 * 12335 * 5236 u_str ESTAB 0 0 * 11651 * 11650 u_dgr ESTAB 0 0 /run/systemd/journal/dev-log 5236 * 0 u_dgr ESTAB 0 0 /run/systemd/journal/socket 5237 * 0 u_dgr ESTAB 0 0 * 3546 * 3547 u_str ESTAB 0 0 * 11632 * 11631 u_str ESTAB 0 0 * 11648 * 11647 u_str ESTAB 0 0 * 6974 * 6973 u_str ESTAB 0 0 * 12111 * 0 u_str ESTAB 0 0 * 3259676 * 3258625 u_str ESTAB 0 0 * 11675 * 11674 u_str ESTAB 0 0 * 11674 * 11675 u_str ESTAB 0 0 /run/systemd/journal/stdout 9419 * 9418 u_str ESTAB 0 0 * 11622 * 11621 u_dgr ESTAB 0 0 * 10127 * 10128 u_str ESTAB 0 0 * 7908 * 7019 u_str ESTAB 0 0 * 3259922 * 3259923 u_str ESTAB 0 0 * 3259921 * 3259920 u_str ESTAB 0 0 /run/systemd/journal/stdout 11068 * 13495 u_dgr ESTAB 0 0 * 10469 * 5236 u_dgr ESTAB 0 0 * 5032 * 5237 u_str ESTAB 0 0 /run/dbus/system_bus_socket 1805048 * 1806090 u_str ESTAB 0 0 * 1805720 * 0 u_str ESTAB 0 0 * 10298 * 10299 u_str ESTAB 0 0 * 11689 * 11690 u_dgr ESTAB 0 0 * 1806087 * 1806088 u_str ESTAB 0 0 * 11290 * 11291 u_dgr ESTAB 0 0 * 11701 * 5236 u_str ESTAB 0 0 * 11693 * 11692 u_str ESTAB 0 0 * 13495 * 11068 u_dgr ESTAB 0 0 * 7081 * 5236 u_str ESTAB 0 0 * 11636 * 11635 u_str ESTAB 0 0 * 8975 * 9386 u_str ESTAB 0 0 @/run/rpcbind.sock 1805023 * 1805767 u_str ESTAB 0 0 /run/systemd/journal/stdout 9214 * 9661 u_dgr ESTAB 0 0 * 1806069 * 5237 u_str ESTAB 0 0 * 11621 * 11622 u_dgr ESTAB 0 0 /run/chrony/chronyd.sock 7086 * 0 u_str ESTAB 0 0 * 11680 * 11681 u_dgr ESTAB 0 0 * 3261581 * 5236 u_str ESTAB 0 0 /run/dbus/system_bus_socket 10159 * 10158 u_dgr ESTAB 0 0 * 6594 * 6593 u_str ESTAB 0 0 /var/lib/sss/pipes/pam 3260236 * 3261580 u_str ESTAB 0 0 * 10789 * 10790 u_str ESTAB 0 0 * 9451 * 6995 u_str ESTAB 0 0 * 11678 * 11677 u_str ESTAB 0 0 * 11668 * 11669 u_dgr ESTAB 0 0 * 6589 * 5237 u_str ESTAB 0 0 * 11644 * 11645 u_str ESTAB 0 0 * 11642 * 11641 u_dgr ESTAB 0 768 * 9007 * 9006 u_str ESTAB 0 0 * 1806581 * 1806580 u_str ESTAB 0 0 * 1806580 * 1806581 u_str ESTAB 0 0 /run/systemd/journal/stdout 10293 * 10292 u_str ESTAB 0 0 /run/systemd/journal/stdout 10291 * 10290 u_str ESTAB 0 0 * 3261597 * 3261598 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 12063 * 13475 u_str ESTAB 0 0 * 11662 * 11663 u_str ESTAB 0 0 * 7035 * 7914 u_str ESTAB 0 0 /run/systemd/journal/stdout 7454 * 5011 u_str ESTAB 0 0 * 1805796 * 0 u_str ESTAB 0 0 * 3260520 * 0 u_str ESTAB 0 0 * 11657 * 11656 u_dgr ESTAB 0 0 * 1806083 * 1806084 u_str ESTAB 0 0 * 12624 * 12623 u_str ESTAB 0 0 * 9418 * 9419 u_dgr ESTAB 0 0 * 3258338 * 5237 u_str ESTAB 0 0 * 1805715 * 0 u_str ESTAB 0 0 /run/systemd/journal/stdout 10299 * 10298 u_str ESTAB 0 0 * 11695 * 11696 u_str ESTAB 0 0 * 11681 * 11680 u_str ESTAB 0 0 * 3259838 * 0 u_str ESTAB 0 0 * 11631 * 11632 u_dgr ESTAB 0 0 * 12856 * 5237 u_str ESTAB 0 0 * 11653 * 11654 u_str ESTAB 0 0 /run/systemd/journal/stdout 9386 * 8975 u_str ESTAB 0 0 * 11665 * 11666 u_str ESTAB 0 0 * 9489 * 7097 u_dgr ESTAB 0 0 * 4828 * 4829 u_dgr ESTAB 0 0 * 12905 * 5236 u_str ESTAB 0 0 * 13475 * 12063 u_str ESTAB 0 0 /run/dbus/system_bus_socket 6990 * 7893 u_str ESTAB 0 0 * 11650 * 11651 u_str ESTAB 0 0 * 9661 * 9214 u_str ESTAB 0 0 * 3258205 * 0 u_str ESTAB 0 0 * 11669 * 11668 u_str ESTAB 0 0 * 11294 * 11295 u_dgr ESTAB 0 0 * 9089 * 5236 u_str ESTAB 0 0 * 6577 * 7622 u_str ESTAB 0 0 * 3259920 * 3259921 u_dgr ESTAB 0 0 * 1806577 * 5236 u_str ESTAB 0 0 * 12623 * 12624 u_str ESTAB 0 0 * 11656 * 11657 u_str ESTAB 0 0 /run/systemd/journal/stdout 3258625 * 3259676 u_str ESTAB 0 0 * 11686 * 11687 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 11291 * 11290 u_str ESTAB 0 0 * 10790 * 10789 u_dgr ESTAB 0 0 * 9427 * 5237 u_str ESTAB 0 0 /run/dbus/system_bus_socket 6988 * 9429 u_str ESTAB 0 0 * 11683 * 11684 u_str ESTAB 0 0 * 3259923 * 3259922 u_str ESTAB 0 0 * 3259919 * 3259918 u_dgr ESTAB 0 0 * 3260669 * 5237 u_dgr ESTAB 0 0 * 1806061 * 5236 u_str ESTAB 0 0 /run/systemd/journal/stdout 11079 * 13510 u_str ESTAB 0 0 /var/lib/sss/pipes/private/sbus-master 13480 * 13479 u_str ESTAB 0 0 * 11639 * 11638 u_str ESTAB 0 0 * 11635 * 11636 u_dgr ESTAB 0 0 * 3544 * 3545 u_str ESTAB 0 0 * 11690 * 11689 u_str ESTAB 0 0 * 11672 * 11671 u_str ESTAB 0 0 /run/systemd/journal/stdout 11125 * 12904 u_dgr ESTAB 0 0 * 1806085 * 1806086 u_dgr ESTAB 0 0 * 3547 * 3546 u_str ESTAB 0 0 * 11696 * 11695 u_str ESTAB 0 0 * 3259810 * 3258766 u_dgr ESTAB 0 0 * 1795630 * 5236 u_str ESTAB 0 0 * 11654 * 11653 u_str ESTAB 0 0 * 1805767 * 1805023 u_str ESTAB 0 0 * 10688 * 10121 u_str ESTAB 0 0 * 11687 * 11686 u_str ESTAB 0 0 * 11671 * 11672 u_str ESTAB 0 0 * 12838 * 12062 u_str ESTAB 0 0 * 11629 * 11628 u_str ESTAB 0 0 * 9524 * 7991 u_str ESTAB 0 0 * 7892 * 7891 u_str ESTAB 0 0 * 11659 * 11660 u_str ESTAB 0 0 * 11645 * 11644 u_dgr ESTAB 0 0 * 9706 * 5236 u_dgr ESTAB 0 0 * 9006 * 9007 u_str ESTAB 0 0 * 10158 * 10159 u_dgr ESTAB 0 0 * 6972 * 5236 u_str ESTAB 0 0 /run/dbus/system_bus_socket 10121 * 10688 u_str ESTAB 0 0 * 1806041 * 1805036 u_str ESTAB 0 0 /run/systemd/journal/stdout 9213 * 7140 u_str ESTAB 0 0 /run/dbus/system_bus_socket 7097 * 9489 u_str ESTAB 0 0 /run/systemd/journal/stdout 7019 * 7908 u_str ESTAB 0 0 * 6973 * 6974 u_str ESTAB 0 0 * 11684 * 11683 u_str ESTAB 0 0 * 11663 * 11662 u_str ESTAB 0 0 * 1795634 * 0 u_dgr ESTAB 0 0 * 3260114 * 5237 u_str ESTAB 0 0 * 3261598 * 3261597 u_str ESTAB 0 0 * 3259918 * 3259919 u_str ESTAB 0 0 /run/systemd/journal/stdout 1805036 * 1806041 u_str ESTAB 0 0 * 11628 * 11629 u_str ESTAB 0 0 * 7140 * 9213 u_str ESTAB 0 0 * 11677 * 11678 u_str ESTAB 0 0 @f5c12eea9342a377/bus/systemd-logind/system 12872 * 12873 u_str ESTAB 0 0 @ed019b0aea426515/bus/systemd/bus-system 1806090 * 1805048 u_str ESTAB 0 0 @3cbcd21433f31955/bus/dbus-broker-lau/system 7893 * 6990 u_str ESTAB 0 0 @8aa9cf87f9fa122a/bus/systemd/bus-api-system 9429 * 6988 icmp6 UNCONN 0 0 *:ipv6-icmp *:* tcp ESTAB 0 0 <obfuscated>:952 <obfuscated>:nfs tcp ESTAB 0 36 <obfuscated>:ssh <obfuscated>:53277 tcp ESTAB 0 0 <obfuscated>:43916 <obfuscated>:ldaps tcp ESTAB 0 0 <obfuscated>:54296 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:40380 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:33806 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:39428 <obfuscated>:https tcp ESTAB 0 0 <obfuscated>:58586 <obfuscated>:30514 tcp ESTAB 0 0 [::1]:50086 [::1]:redis tcp ESTAB 0 0 [::1]:pcsync-https [::1]:35174 tcp ESTAB 0 0 [::ffff:<obfuscated>]:pcsync-https [::ffff:192.168.132.166]:50136 tcp ESTAB 0 0 [::1]:35174 [::1]:pcsync-https tcp CLOSE-WAIT 401224 0 [::1]:42184 [::1]:9004 tcp CLOSE-WAIT 339569 0 [::1]:34522 [::1]:9004 tcp CLOSE-WAIT 401329 0 [::1]:47524 [::1]:9004 tcp ESTAB 0 0 [::1]:redis [::1]:50086 tcp CLOSE-WAIT 339562 0 [::1]:56834 [::1]:9004 tcp ESTAB 0 0 [::ffff:<obfuscated>]:pcsync-https [::ffff:192.168.132.166]:54998 tcp ESTAB 0 0 [::1]:pcsync-https [::1]:35184 tcp ESTAB 0 0 [::1]:35184 [::1]:pcsync-https tcp FIN-WAIT-2 0 0 [::1]:9004 [::1]:42728 tcp CLOSE-WAIT 339529 0 [::1]:33512 [::1]:9004 tcp CLOSE-WAIT 401208 0 [::1]:42728 [::1]:9004After latest
dnf upgradepost service restart, and subsequent reboot for new kernel$ cat /proc/net/sockstat sockets: used 339 TCP: inuse 24 orphan 0 tw 36 alloc 47 mem 238 UDP: inuse 5 mem 0 UDPLITE: inuse 0 RAW: inuse 0 FRAG: inuse 0 memory 0 $ cat /proc/sys/net/ipv4/tcp_mem 90147 120196 180294 -
-
@Forza
Comparing against the weaker XOCEStrong:
net.ipv4.tcp_mem = 90147 120196 180294 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304Weak:
net.ipv4.tcp_mem = 41766 55688 83532 net.ipv4.tcp_rmem = 4096 131072 6291456 net.ipv4.tcp_wmem = 4096 16384 4194304 -
P poddingue marked this topic as a question
-
Did a much closer look and can indeed confirm that the network capabilities go up from kb/s -> MB/s immediately from xo_server restarts.
-
On the tuning suggestion, my guess, untested and purely from reading your paste, is that if something is genuinely holding sockets then raising
tcp_membuys headroom rather than stopping the accumulation, and your post#4 about throughput coming back on anxo_serverrestart points the same way.
It might be worth a mention to @Team-XO-Backend, since they would know whether
xo-serverkeeps sockets per pool. -
On the tuning suggestion, my guess, untested and purely from reading your paste, is that if something is genuinely holding sockets then raising
tcp_membuys headroom rather than stopping the accumulation, and your post#4 about throughput coming back on anxo_serverrestart points the same way.It might be worth a mention to @Team-XO-Backend, since they would know whether
xo-serverkeeps sockets per pool.This is why I'm not increasing the numbers quite yet. Since this would only mask a potential issue. If an underlying cause is responsible, then this would be of interest.
Thanks for chiming in!
-
Looking at it today again. Doing a
sudo ss -tpand noticing that there are alot ofCLOSE-WAIT. Maybe this is a red herring, or maybe not. I thought I would share the findings regardless.The output is also to long for me to be able to paste it in text. So I've attached it as a txt-file.
close-wait.txt -
@jr-m4 you can export the heap memory of the nodeJS process by doing
kill -SIGUSER2 <xoserverpid>onte that this will increase a lot the memory consumed by the xo process even when its done exporting the memory . This will help us know what xo is doing at the moment
-
-
@jr-m4 you can export the heap memory of the nodeJS process by doing
kill -SIGUSER2 <xoserverpid>onte that this will increase a lot the memory consumed by the xo process even when its done exporting the memory . This will help us know what xo is doing at the moment
Do you have somewhere I can upload the heapsnapshot? (148MB)
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login