Troubleshooting "TCP: out of memory" - Possible memory leak?

Forza

$ cat /proc/sys/net/ipv4/tcp_mem
90147   120196  180294

Consider increasing your tcp memory buffers.?

❯ sysctl -a |grep tcp|grep mem
net.ipv4.tcp_mem = 372396       496530  744792
net.ipv4.tcp_rmem = 4096        262144  268435456
net.ipv4.tcp_wmem = 4096        262144  268435456

jr-m4

@Forza
Comparing against the weaker XOCE

Strong:

net.ipv4.tcp_mem = 90147        120196  180294
net.ipv4.tcp_rmem = 4096        131072  6291456
net.ipv4.tcp_wmem = 4096        16384   4194304

Weak:

net.ipv4.tcp_mem = 41766        55688   83532
net.ipv4.tcp_rmem = 4096        131072  6291456
net.ipv4.tcp_wmem = 4096        16384   4194304

jr-m4

Did a much closer look and can indeed confirm that the network capabilities go up from kb/s -> MB/s immediately from xo_server restarts.

poddingue

On the tuning suggestion, my guess, untested and purely from reading your paste, is that if something is genuinely holding sockets then raising tcp_mem buys headroom rather than stopping the accumulation, and your post#4 about throughput coming back on an xo_server restart points the same way.

It might be worth a mention to @Team-XO-Backend, since they would know whether xo-server keeps sockets per pool.

jr-m4

@poddingue said:

On the tuning suggestion, my guess, untested and purely from reading your paste, is that if something is genuinely holding sockets then raising tcp_mem buys headroom rather than stopping the accumulation, and your post#4 about throughput coming back on an xo_server restart points the same way.

It might be worth a mention to @Team-XO-Backend, since they would know whether xo-server keeps sockets per pool.

This is why I'm not increasing the numbers quite yet. Since this would only mask a potential issue. If an underlying cause is responsible, then this would be of interest.

Thanks for chiming in!

jr-m4

Looking at it today again. Doing a sudo ss -tp and noticing that there are alot of CLOSE-WAIT. Maybe this is a red herring, or maybe not. I thought I would share the findings regardless.

The output is also to long for me to be able to paste it in text. So I've attached it as a txt-file.
close-wait.txt

florent

@jr-m4 you can export the heap memory of the nodeJS process by doing kill -SIGUSER2 <xoserverpid>

onte that this will increase a lot the memory consumed by the xo process even when its done exporting the memory . This will help us know what xo is doing at the moment

jr-m4

@florent

I'll try to do this when I notice that it is misbehaving again.

Thanks!

jr-m4

@florent said:

@jr-m4 you can export the heap memory of the nodeJS process by doing kill -SIGUSER2 <xoserverpid>

onte that this will increase a lot the memory consumed by the xo process even when its done exporting the memory . This will help us know what xo is doing at the moment

Do you have somewhere I can upload the heapsnapshot? (148MB)

Ping @poddingue as well, for visibility

jr-m4

@florent & @poddingue

I now have to heap snapshots. Could you provide somewhere to upload them, please? And I will do so as soon as I can.