<![CDATA[Troubleshooting "TCP: out of memory" - Possible memory leak?]]>Hello

This is an early start of a thread for documenting a potential issue. I will add to it with findings accordinly.

I'm having some issues with on XOCE instance, commit 7a187. Three times this week, it has gotten quite unresponsive with network trafic. And looking at the console give clues as to why:
"TCP: out of memory -- consider tuning tcp_mem"

0ec99c11-994b-4a9f-abc8-66d7aa1c74b0-image.jpeg

The weird part is that this is my most beefy XOCE instance of all.
4 cores, 8GiB, management agent 10 (Citrix), Rocky 10.2.

In troubleshooting I tried restarting the XO-server service. And this seems to have resolved the issue for now. But TBH, I was looking at several point at the same time. So it is possible that it had resolved itself (unlikely but possible) whilst I was troubleshooting and reading up on tcp_mem.

My other XOCE instances are setup identically, and are actually a little bit smaller in the amount of RAM assigned. And they show no problems.

Failing XO instance connects to three separate 1Host pools.
Working XO instance connects to 1 HA-pool with three hosts.

Installation is done by a port of https://github.com/cloudrootab/ansible_role_xoce, adjusted for Rocky10 instead of Ubuntu/Debian.

Edit: I should also add that this XOCE has shown the same behaviour on both a recently updated XCP-ng host, and one that is two months behind on updates. Running on Dell Inc. PowerEdge R670 for each of the hosts (RAM on hosts has plenty to spare, roughly 50% utilization)

]]>https://xcp-ng.org/forum/topic/12455/troubleshooting-tcp-out-of-memory-possible-memory-leakRSS for NodeFri, 04 Sep 2026 09:06:29 GMTFri, 04 Sep 2026 08:49:32 GMT60<![CDATA[Reply to Troubleshooting "TCP: out of memory" - Possible memory leak? on Fri, 04 Sep 2026 09:00:54 GMT]]>Adding the initial memstats I was looking at whilst trying to troubleshoot

$ cat /proc/net/sockstat
sockets: used 3478
TCP: inuse 37 orphan 0 tw 25 alloc 3149 mem 194631
UDP: inuse 5 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

$ cat /proc/sys/net/ipv4/tcp_mem
90147   120196  180294

Output of sudo ss

Netid State       Recv-Q  Send-Q                                  Local Address:Port                       Peer Address:Port
u_str ESTAB       0       0                                                   * 10292                                 * 10293
u_str ESTAB       0       0                              /var/lib/sss/pipes/nss 3260817                               * 3261568
u_str ESTAB       0       0              /var/lib/sss/pipes/private/sbus-master 12062                                 * 12838
u_str ESTAB       0       0                                                   * 11638                                 * 11639
u_str ESTAB       0       0                                                   * 13510                                 * 11079
u_str ESTAB       0       0                                                   * 11624                                 * 11625
u_dgr ESTAB       0       768                                                 * 10128                                 * 10127
u_dgr ESTAB       0       0                                                   * 6593                                  * 6594
u_str ESTAB       0       0                                                   * 5011                                  * 7454
u_str ESTAB       0       0                                                   * 11692                                 * 11693
u_str ESTAB       0       0                                                   * 11641                                 * 11642
u_str ESTAB       0       0              /var/lib/sss/pipes/private/sbus-master 11295                                 * 11294
u_dgr ESTAB       0       0                                                   * 6309                                  * 5237
u_str ESTAB       0       0                                                   * 3261580                               * 3260236
u_str ESTAB       0       0                                                   * 3259924                               * 3259925
u_dgr ESTAB       0       0                                                   * 1806084                               * 1806083
u_str ESTAB       0       0                                                   * 12904                                 * 11125
u_dgr ESTAB       0       0                                                   * 3545                                  * 3544
u_str ESTAB       0       0                                                   * 3259925                               * 3259924
u_str ESTAB       0       0                                                   * 11666                                 * 11665
u_str ESTAB       0       0                         /run/systemd/journal/stdout 7622                                  * 6577
u_str ESTAB       0       0                                                   * 11660                                 * 11659
u_str ESTAB       0       0                         /run/systemd/journal/stdout 7991                                  * 9524
u_str ESTAB       0       0                                                   * 1806062                               * 0
u_str ESTAB       0       0                         /run/systemd/journal/stdout 7914                                  * 7035
u_str ESTAB       0       0                         /run/systemd/journal/stdout 6995                                  * 9451
u_dgr ESTAB       0       0                                                   * 1806086                               * 1806085
u_str ESTAB       0       0                                                   * 13479                                 * 13480
u_dgr ESTAB       0       0                                                   * 4829                                  * 4828
u_str ESTAB       0       0                                                   * 3261568                               * 3260817
u_dgr ESTAB       0       0                                                   * 11586                                 * 5236
u_dgr ESTAB       0       0                                                   * 7352                                  * 3543
u_str ESTAB       0       0                                                   * 11647                                 * 11648
u_str ESTAB       0       0                                                   * 7891                                  * 7892
u_dgr ESTAB       0       0                                 /run/systemd/notify 3543                                  * 0
u_str ESTAB       0       0                         /run/systemd/journal/stdout 3258766                               * 3259810
u_str ESTAB       0       0                         /run/dbus/system_bus_socket 12873                                 * 12872
u_str ESTAB       0       0                                                   * 10290                                 * 10291
u_dgr ESTAB       0       0                                                   * 1806088                               * 1806087
u_str ESTAB       0       0                                                   * 11625                                 * 11624
u_dgr ESTAB       0       0                                                   * 12335                                 * 5236
u_str ESTAB       0       0                                                   * 11651                                 * 11650
u_dgr ESTAB       0       0                        /run/systemd/journal/dev-log 5236                                  * 0
u_dgr ESTAB       0       0                         /run/systemd/journal/socket 5237                                  * 0
u_dgr ESTAB       0       0                                                   * 3546                                  * 3547
u_str ESTAB       0       0                                                   * 11632                                 * 11631
u_str ESTAB       0       0                                                   * 11648                                 * 11647
u_str ESTAB       0       0                                                   * 6974                                  * 6973
u_str ESTAB       0       0                                                   * 12111                                 * 0
u_str ESTAB       0       0                                                   * 3259676                               * 3258625
u_str ESTAB       0       0                                                   * 11675                                 * 11674
u_str ESTAB       0       0                                                   * 11674                                 * 11675
u_str ESTAB       0       0                         /run/systemd/journal/stdout 9419                                  * 9418
u_str ESTAB       0       0                                                   * 11622                                 * 11621
u_dgr ESTAB       0       0                                                   * 10127                                 * 10128
u_str ESTAB       0       0                                                   * 7908                                  * 7019
u_str ESTAB       0       0                                                   * 3259922                               * 3259923
u_str ESTAB       0       0                                                   * 3259921                               * 3259920
u_str ESTAB       0       0                         /run/systemd/journal/stdout 11068                                 * 13495
u_dgr ESTAB       0       0                                                   * 10469                                 * 5236
u_dgr ESTAB       0       0                                                   * 5032                                  * 5237
u_str ESTAB       0       0                         /run/dbus/system_bus_socket 1805048                               * 1806090
u_str ESTAB       0       0                                                   * 1805720                               * 0
u_str ESTAB       0       0                                                   * 10298                                 * 10299
u_str ESTAB       0       0                                                   * 11689                                 * 11690
u_dgr ESTAB       0       0                                                   * 1806087                               * 1806088
u_str ESTAB       0       0                                                   * 11290                                 * 11291
u_dgr ESTAB       0       0                                                   * 11701                                 * 5236
u_str ESTAB       0       0                                                   * 11693                                 * 11692
u_str ESTAB       0       0                                                   * 13495                                 * 11068
u_dgr ESTAB       0       0                                                   * 7081                                  * 5236
u_str ESTAB       0       0                                                   * 11636                                 * 11635
u_str ESTAB       0       0                                                   * 8975                                  * 9386
u_str ESTAB       0       0                                  @/run/rpcbind.sock 1805023                               * 1805767
u_str ESTAB       0       0                         /run/systemd/journal/stdout 9214                                  * 9661
u_dgr ESTAB       0       0                                                   * 1806069                               * 5237
u_str ESTAB       0       0                                                   * 11621                                 * 11622
u_dgr ESTAB       0       0                            /run/chrony/chronyd.sock 7086                                  * 0
u_str ESTAB       0       0                                                   * 11680                                 * 11681
u_dgr ESTAB       0       0                                                   * 3261581                               * 5236
u_str ESTAB       0       0                         /run/dbus/system_bus_socket 10159                                 * 10158
u_dgr ESTAB       0       0                                                   * 6594                                  * 6593
u_str ESTAB       0       0                              /var/lib/sss/pipes/pam 3260236                               * 3261580
u_str ESTAB       0       0                                                   * 10789                                 * 10790
u_str ESTAB       0       0                                                   * 9451                                  * 6995
u_str ESTAB       0       0                                                   * 11678                                 * 11677
u_str ESTAB       0       0                                                   * 11668                                 * 11669
u_dgr ESTAB       0       0                                                   * 6589                                  * 5237
u_str ESTAB       0       0                                                   * 11644                                 * 11645
u_str ESTAB       0       0                                                   * 11642                                 * 11641
u_dgr ESTAB       0       768                                                 * 9007                                  * 9006
u_str ESTAB       0       0                                                   * 1806581                               * 1806580
u_str ESTAB       0       0                                                   * 1806580                               * 1806581
u_str ESTAB       0       0                         /run/systemd/journal/stdout 10293                                 * 10292
u_str ESTAB       0       0                         /run/systemd/journal/stdout 10291                                 * 10290
u_str ESTAB       0       0                                                   * 3261597                               * 3261598
u_str ESTAB       0       0              /var/lib/sss/pipes/private/sbus-master 12063                                 * 13475
u_str ESTAB       0       0                                                   * 11662                                 * 11663
u_str ESTAB       0       0                                                   * 7035                                  * 7914
u_str ESTAB       0       0                         /run/systemd/journal/stdout 7454                                  * 5011
u_str ESTAB       0       0                                                   * 1805796                               * 0
u_str ESTAB       0       0                                                   * 3260520                               * 0
u_str ESTAB       0       0                                                   * 11657                                 * 11656
u_dgr ESTAB       0       0                                                   * 1806083                               * 1806084
u_str ESTAB       0       0                                                   * 12624                                 * 12623
u_str ESTAB       0       0                                                   * 9418                                  * 9419
u_dgr ESTAB       0       0                                                   * 3258338                               * 5237
u_str ESTAB       0       0                                                   * 1805715                               * 0
u_str ESTAB       0       0                         /run/systemd/journal/stdout 10299                                 * 10298
u_str ESTAB       0       0                                                   * 11695                                 * 11696
u_str ESTAB       0       0                                                   * 11681                                 * 11680
u_str ESTAB       0       0                                                   * 3259838                               * 0
u_str ESTAB       0       0                                                   * 11631                                 * 11632
u_dgr ESTAB       0       0                                                   * 12856                                 * 5237
u_str ESTAB       0       0                                                   * 11653                                 * 11654
u_str ESTAB       0       0                         /run/systemd/journal/stdout 9386                                  * 8975
u_str ESTAB       0       0                                                   * 11665                                 * 11666
u_str ESTAB       0       0                                                   * 9489                                  * 7097
u_dgr ESTAB       0       0                                                   * 4828                                  * 4829
u_dgr ESTAB       0       0                                                   * 12905                                 * 5236
u_str ESTAB       0       0                                                   * 13475                                 * 12063
u_str ESTAB       0       0                         /run/dbus/system_bus_socket 6990                                  * 7893
u_str ESTAB       0       0                                                   * 11650                                 * 11651
u_str ESTAB       0       0                                                   * 9661                                  * 9214
u_str ESTAB       0       0                                                   * 3258205                               * 0
u_str ESTAB       0       0                                                   * 11669                                 * 11668
u_str ESTAB       0       0                                                   * 11294                                 * 11295
u_dgr ESTAB       0       0                                                   * 9089                                  * 5236
u_str ESTAB       0       0                                                   * 6577                                  * 7622
u_str ESTAB       0       0                                                   * 3259920                               * 3259921
u_dgr ESTAB       0       0                                                   * 1806577                               * 5236
u_str ESTAB       0       0                                                   * 12623                                 * 12624
u_str ESTAB       0       0                                                   * 11656                                 * 11657
u_str ESTAB       0       0                         /run/systemd/journal/stdout 3258625                               * 3259676
u_str ESTAB       0       0                                                   * 11686                                 * 11687
u_str ESTAB       0       0              /var/lib/sss/pipes/private/sbus-master 11291                                 * 11290
u_str ESTAB       0       0                                                   * 10790                                 * 10789
u_dgr ESTAB       0       0                                                   * 9427                                  * 5237
u_str ESTAB       0       0                         /run/dbus/system_bus_socket 6988                                  * 9429
u_str ESTAB       0       0                                                   * 11683                                 * 11684
u_str ESTAB       0       0                                                   * 3259923                               * 3259922
u_str ESTAB       0       0                                                   * 3259919                               * 3259918
u_dgr ESTAB       0       0                                                   * 3260669                               * 5237
u_dgr ESTAB       0       0                                                   * 1806061                               * 5236
u_str ESTAB       0       0                         /run/systemd/journal/stdout 11079                                 * 13510
u_str ESTAB       0       0              /var/lib/sss/pipes/private/sbus-master 13480                                 * 13479
u_str ESTAB       0       0                                                   * 11639                                 * 11638
u_str ESTAB       0       0                                                   * 11635                                 * 11636
u_dgr ESTAB       0       0                                                   * 3544                                  * 3545
u_str ESTAB       0       0                                                   * 11690                                 * 11689
u_str ESTAB       0       0                                                   * 11672                                 * 11671
u_str ESTAB       0       0                         /run/systemd/journal/stdout 11125                                 * 12904
u_dgr ESTAB       0       0                                                   * 1806085                               * 1806086
u_dgr ESTAB       0       0                                                   * 3547                                  * 3546
u_str ESTAB       0       0                                                   * 11696                                 * 11695
u_str ESTAB       0       0                                                   * 3259810                               * 3258766
u_dgr ESTAB       0       0                                                   * 1795630                               * 5236
u_str ESTAB       0       0                                                   * 11654                                 * 11653
u_str ESTAB       0       0                                                   * 1805767                               * 1805023
u_str ESTAB       0       0                                                   * 10688                                 * 10121
u_str ESTAB       0       0                                                   * 11687                                 * 11686
u_str ESTAB       0       0                                                   * 11671                                 * 11672
u_str ESTAB       0       0                                                   * 12838                                 * 12062
u_str ESTAB       0       0                                                   * 11629                                 * 11628
u_str ESTAB       0       0                                                   * 9524                                  * 7991
u_str ESTAB       0       0                                                   * 7892                                  * 7891
u_str ESTAB       0       0                                                   * 11659                                 * 11660
u_str ESTAB       0       0                                                   * 11645                                 * 11644
u_dgr ESTAB       0       0                                                   * 9706                                  * 5236
u_dgr ESTAB       0       0                                                   * 9006                                  * 9007
u_str ESTAB       0       0                                                   * 10158                                 * 10159
u_dgr ESTAB       0       0                                                   * 6972                                  * 5236
u_str ESTAB       0       0                         /run/dbus/system_bus_socket 10121                                 * 10688
u_str ESTAB       0       0                                                   * 1806041                               * 1805036
u_str ESTAB       0       0                         /run/systemd/journal/stdout 9213                                  * 7140
u_str ESTAB       0       0                         /run/dbus/system_bus_socket 7097                                  * 9489
u_str ESTAB       0       0                         /run/systemd/journal/stdout 7019                                  * 7908
u_str ESTAB       0       0                                                   * 6973                                  * 6974
u_str ESTAB       0       0                                                   * 11684                                 * 11683
u_str ESTAB       0       0                                                   * 11663                                 * 11662
u_str ESTAB       0       0                                                   * 1795634                               * 0
u_dgr ESTAB       0       0                                                   * 3260114                               * 5237
u_str ESTAB       0       0                                                   * 3261598                               * 3261597
u_str ESTAB       0       0                                                   * 3259918                               * 3259919
u_str ESTAB       0       0                         /run/systemd/journal/stdout 1805036                               * 1806041
u_str ESTAB       0       0                                                   * 11628                                 * 11629
u_str ESTAB       0       0                                                   * 7140                                  * 9213
u_str ESTAB       0       0                                                   * 11677                                 * 11678
u_str ESTAB       0       0         @f5c12eea9342a377/bus/systemd-logind/system 12872                                 * 12873
u_str ESTAB       0       0            @ed019b0aea426515/bus/systemd/bus-system 1806090                               * 1805048
u_str ESTAB       0       0        @3cbcd21433f31955/bus/dbus-broker-lau/system 7893                                  * 6990
u_str ESTAB       0       0        @8aa9cf87f9fa122a/bus/systemd/bus-api-system 9429                                  * 6988
icmp6 UNCONN      0       0                                                   *:ipv6-icmp                             *:*
tcp   ESTAB       0       0                                       <obfuscated>:952                      <obfuscated>:nfs
tcp   ESTAB       0       36                                      <obfuscated>:ssh                     <obfuscated>:53277
tcp   ESTAB       0       0                                       <obfuscated>:43916                    <obfuscated>:ldaps
tcp   ESTAB       0       0                                       <obfuscated>:54296                    <obfuscated>:https
tcp   ESTAB       0       0                                       <obfuscated>:40380                    <obfuscated>:https
tcp   ESTAB       0       0                                       <obfuscated>:33806                    <obfuscated>:https
tcp   ESTAB       0       0                                       <obfuscated>:39428                    <obfuscated>:https
tcp   ESTAB       0       0                                       <obfuscated>:58586                     <obfuscated>:30514
tcp   ESTAB       0       0                                               [::1]:50086                             [::1]:redis
tcp   ESTAB       0       0                                               [::1]:pcsync-https                      [::1]:35174
tcp   ESTAB       0       0                              [::ffff:<obfuscated>]:pcsync-https   [::ffff:192.168.132.166]:50136
tcp   ESTAB       0       0                                               [::1]:35174                             [::1]:pcsync-https
tcp   CLOSE-WAIT  401224  0                                               [::1]:42184                             [::1]:9004
tcp   CLOSE-WAIT  339569  0                                               [::1]:34522                             [::1]:9004
tcp   CLOSE-WAIT  401329  0                                               [::1]:47524                             [::1]:9004
tcp   ESTAB       0       0                                               [::1]:redis                             [::1]:50086
tcp   CLOSE-WAIT  339562  0                                               [::1]:56834                             [::1]:9004
tcp   ESTAB       0       0                              [::ffff:<obfuscated>]:pcsync-https   [::ffff:192.168.132.166]:54998
tcp   ESTAB       0       0                                               [::1]:pcsync-https                      [::1]:35184
tcp   ESTAB       0       0                                               [::1]:35184                             [::1]:pcsync-https
tcp   FIN-WAIT-2  0       0                                               [::1]:9004                              [::1]:42728
tcp   CLOSE-WAIT  339529  0                                               [::1]:33512                             [::1]:9004
tcp   CLOSE-WAIT  401208  0                                               [::1]:42728                             [::1]:9004

After latest dnf upgrade post service restart, and subsequent reboot for new kernel

$ cat /proc/net/sockstat
sockets: used 339
TCP: inuse 24 orphan 0 tw 36 alloc 47 mem 238
UDP: inuse 5 mem 0
UDPLITE: inuse 0
RAW: inuse 0
FRAG: inuse 0 memory 0

$ cat /proc/sys/net/ipv4/tcp_mem
90147   120196  180294
]]>https://xcp-ng.org/forum/post/108289https://xcp-ng.org/forum/post/108289Fri, 04 Sep 2026 09:00:54 GMT