Alert: Control Domain Memory Usage

stormi

So, @r1 has prepared a kernel RPM for XCP-ng 8.1 that enables kmemleak. If anyone wants to give it a try (on XCP-ng 8.1 only), you can install it with:

yum install http://koji.xcp-ng.org/kojifiles/work/tasks/7624/17624/kernel-4.19.19-6.0.12.1.1.kmemleak.xcpng8.1.x86_64.rpm
reboot

You can revert to the main kernel with:

# yum downgrade won't work for the kernel because it's a protected package, so let's use rpm
yumdownloader
rpm -Uv --oldpackage name-of-file.rpm
reboot

There will be some performance impact that I'm not able to quantify and I'm not yet able to tell you how to use it to debug memory leaks, but there's plenty of documentation on the internet about kmemleak.

delaf

Hello,

We have the same problem here on multiple servers.

We also have 10G interfaces. We use ext local SR.

(::bbxl0271) (2 running) [08:43 bbxl0271 ~]# lsmod  | sort -k 2 -n  -r
ipv6                  548864  313 nf_nat_ipv6
sunrpc                413696  18 lockd,nfsv3,nfs_acl,nfs
ixgbe                 380928  0
fscache               380928  1 nfs
nfs                   307200  2 nfsv3
libata                274432  2 libahci,ahci
xhci_hcd              258048  1 xhci_pci
scsi_mod              253952  15 fcoe,scsi_dh_emc,sd_mod,dm_multipath,scsi_dh_alua,scsi_transport_fc,usb_storage,libfc,bnx2fc,uas,megaraid_sas,libata,sg,scsi_dh_rdac,scsi_dh_hp_sw
aesni_intel           200704  0
megaraid_sas          167936  4
nf_conntrack          163840  6 xt_conntrack,nf_nat,nf_nat_ipv6,nf_nat_ipv4,openvswitch,nf_conncount
bnx2fc                159744  0
dm_mod                151552  5 dm_multipath
openvswitch           147456  12
libfc                 147456  3 fcoe,bnx2fc,libfcoe
hid                   122880  2 usbhid,hid_generic
mei                   114688  1 mei_me
lockd                 110592  2 nfsv3,nfs
cnic                   81920  1 bnx2fc
libfcoe                77824  2 fcoe,bnx2fc
usb_storage            73728  1 uas
scsi_transport_fc      69632  3 fcoe,libfc,bnx2fc
ipmi_si                65536  0
ipmi_msghandler        61440  2 ipmi_devintf,ipmi_si
usbhid                 57344  0
sd_mod                 53248  5
tun                    49152  0
nfsv3                  49152  1
x_tables               45056  6 xt_conntrack,iptable_filter,xt_multiport,xt_tcpudp,ipt_REJECT,ip_tables
mei_me                 45056  0
sg                     40960  0
libahci                40960  1 ahci
ahci                   40960  0
8021q                  40960  0
nf_nat                 36864  3 nf_nat_ipv6,nf_nat_ipv4,openvswitch
fcoe                   32768  0
dm_multipath           32768  0
uas                    28672  0
lpc_ich                28672  0
ip_tables              28672  2 iptable_filter
i2c_i801               28672  0
cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
uio                    20480  1 cnic
scsi_dh_alua           20480  0
nf_defrag_ipv6         20480  2 nf_conntrack,openvswitch
mrp                    20480  1 8021q
ipmi_devintf           20480  0
aes_x86_64             20480  1 aesni_intel
acpi_power_meter       20480  0
xt_tcpudp              16384  9
xt_multiport           16384  1
xt_conntrack           16384  5
xhci_pci               16384  0
stp                    16384  1 garp
skx_edac               16384  0
scsi_dh_rdac           16384  0
scsi_dh_hp_sw          16384  0
scsi_dh_emc            16384  0
pcbc                   16384  0
nsh                    16384  1 openvswitch
nfs_acl                16384  1 nfsv3
nf_reject_ipv4         16384  1 ipt_REJECT
nf_nat_ipv6            16384  1 openvswitch
nf_nat_ipv4            16384  1 openvswitch
nf_defrag_ipv4         16384  1 nf_conntrack
nf_conncount           16384  1 openvswitch
llc                    16384  2 stp,garp
libcrc32c              16384  3 nf_conntrack,nf_nat,openvswitch
ipt_REJECT             16384  3
iptable_filter         16384  1
intel_rapl_perf        16384  0
intel_powerclamp       16384  0
hid_generic            16384  0
grace                  16384  1 lockd
glue_helper            16384  1 aesni_intel
ghash_clmulni_intel    16384  0
garp                   16384  1 8021q
crypto_simd            16384  1 aesni_intel
crct10dif_pclmul       16384  0
crc_ccitt              16384  1 ipv6
crc32_pclmul           16384  0

I'll install the kmemleak kernel on one server today.

delaf

@stormi @r1

I tried to install the kernel as described, but I got an error :

# yum install http://koji.xcp-ng.org/kojifiles/work/tasks/7620/17620/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm
Loaded plugins: fastestmirror
kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm                                                     |  30 MB  00:00:03
Examining /var/tmp/yum-root-Uyd1Lb/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm: kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64
/var/tmp/yum-root-Uyd1Lb/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm: does not update installed package.
Error: Nothing to do

The host is a 8.1 up to date (latest patchs are installed but I did not reboot yep after).

delaf

FYI : On one host that has the problem I only have 1 Debian VM that does only one thing : netdata. It is a netdata that get flows from other netdata and that is polled by a prometheus server (which is on another host).

Screen Shot 2020-11-13 at 09.26.10.png

delaf

As soon as I stopped the last VM on the host, there is no more memleak.

delaf

More informations:

# rpm -ivh http://koji.xcp-ng.org/kojifiles/work/tasks/7620/17620/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm
Retrieving http://koji.xcp-ng.org/kojifiles/work/tasks/7620/17620/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm
Preparing...                          ################################# [100%]
    package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64 (which is newer than kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64) is already installed
    file /boot/System.map-4.19.0+1 from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
    file /boot/config-4.19.0+1 from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
    file /boot/vmlinuz-4.19.0+1 from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
    file /lib/modules/4.19.0+1/kernel/fs/nfs/nfs.ko from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
    file /lib/modules/4.19.0+1/kernel/net/netfilter/nf_conntrack.ko from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
    file /lib/modules/4.19.0+1/modules.order from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64

stormi

I built a new kernel with memleak, which should install correctly. I've updated my post with the instructions above.

delaf

@stormi Ok thank you. The installation works.

I made another test: use the kernel-alt 4.19.108 on one box. It seems I did not have the issue anymore!

delaf

On server with kmemleak kernel, I get this error:

# echo scan > /sys/kernel/debug/kmemleak 
-bash: echo: write error: Device or resource busy

Digging a litlle bit, and I found :

# dmesg | grep memleak
[    0.677307] kmemleak: Kernel memory leak detector disabled
[    2.701225] kmemleak: Early log buffer exceeded (5128), please increase DEBUG_KMEMLEAK_EARLY_LOG_SIZE

So kmemleak is still disabled as DEBUG_KMEMLEAK_EARLY_LOG_SIZE seems to be too small
@stormi Could you rebuild a kernel with an increased DEBUG_KMEMLEAK_EARLY_LOG_SIZE?

Thank you!

stormi

Pinging @r1 about this.

r1

@stormi @delaf will check this and do required changes.

r1

@delaf and others, you can download and install and update from link which should work fine.

After the system is running for some time, user can # echo scan > /sys/kernel/debug/kmemleak and then # cat /sys/kernel/debug/kmemleak to see if there are any unreferenced objects floating in memory.

delaf

@r1 installed and it works :

# dmesg | grep kmem
[    6.181218] kmemleak: Kernel memory leak detector initialized
[    6.181223] kmemleak: Automatic memory scanning thread started

I will check the leaks tomorrow.

delaf

@r1 @stormi for now I don't have any memleak detected :

# uname -a
Linux bb0272.aquaray.com 4.19.0+1 #1 SMP Thu Nov 19 15:15:04 CET 2020 x86_64 x86_64 x86_64 GNU/Linux
# yum list installed | grep kernel
kernel.x86_64                   4.19.19-6.0.12.1.2.kmemleak.xcpng8.1
# cat /sys/kernel/debug/kmemleak
# echo scan > /sys/kernel/debug/kmemleak
# cat /sys/kernel/debug/kmemleak

delaf

FYI: I confirm that on the server running the alt kernel I do not have anymore the memleak problem :

# yum list installed | grep kernel
kernel.x86_64                   4.19.19-6.0.12.1.1.kmemleak.xcpng8.1
kernel-alt.x86_64               4.19.108-3.xcpng8.1         @xcp-ng-base
# uname -a
Linux bb0266.aquaray.com 4.19.108 #1 SMP Mon Mar 23 15:04:05 CET 2020 x86_64 x86_64 x86_64 GNU/Linux
#

Memory graph from this server :
Screen Shot 2020-11-20 at 12.14.54.png

Obviously, the alt-kernel has been installed ~2020-11-13.

On the server running the memleak kernel (since yesterday) :
Screen Shot 2020-11-20 at 12.16.21.png

olivierlambert

@delaf Weird but interesting!

r1

@olivierlambert @delaf what we know from kmemleak so far is that it will only scan and report unreferenced objects. If any kernel module / kernel itself is still holding(referencing) the memory then it may not show up. We are evaluating other options to find this.

kernel-alt is more related to upstream, so either this issue is known and fixed in upstream or it might have been introduced from kernel updates.

The oldest kernel available is 4.19.19-6.0.10.1.xcpng8.1, is it possible to install it and see if the issue repeats?

stormi

kernel-alt also means that all the drivers used are buillt-in, no those from the driver packages that take precedence over the built-ins when running the main kernel.

r1

Yes, all drivers are stock kernel modules for kernel-alt. It would be interesting to see the behavior by disabling updates and override. I think we can try both. 1st check if the downgraded kernel shows same symptoms and then disabling update drivers.

stormi

Ok, so now we know for sure that it's related to the kernel or one of the drivers.

Let me summarize all the tests that users affected by the issue can do to help find what causes it:

Test with kmemleak, hoping that it may be able to detect something. No luck for now for @delaf who tried.
Test with the oldest kernel (4.19.19-6.0.10.1.xcpng8.1). I doubt it will yield results, but that would allow to be sure. If it does yield results, then it will allow to search towards a specific patch.
With the current kernel, give priority to built-in drivers. If this gives good results, this will mean that the leak is in one of the drivers that are provided through driver RPMs. Two ways:
- A bit riskier but we'd still be interested in the results: disable them all so that only built-in kernel drivers are used. For this, edit /etc/modprobe.d/dist.conf and change search override updates extra built-in weak-updates into search extra built-in weak-updates override updates, then run depmod -a and reboot. Don't forget to restore the original contents after the tests.
- Another way, that allows to select specific drivers one by one:
  - Identify a few drivers that you want to check in the output of lsmod. For example ixgbe.
  - Find where the currently used driver is on the filesystem: modinfo ixgbe | head -n 1
  - If the path contains "/updates/", it's not a kernel built-in. Rename the file to name_of_file.save. depmod -a. reboot. The kernel will then use its built-in driver.
  - If nothing changes, restore the file and try another.

I also intend to build a new ixgbe driver, just in case we're lucky and it's the culprit, since every affected user uses it.