Alert: Control Domain Memory Usage
-
So, @r1 has prepared a
kernelRPM for XCP-ng 8.1 that enables kmemleak. If anyone wants to give it a try (on XCP-ng 8.1 only), you can install it with:yum install http://koji.xcp-ng.org/kojifiles/work/tasks/7624/17624/kernel-4.19.19-6.0.12.1.1.kmemleak.xcpng8.1.x86_64.rpm rebootYou can revert to the main kernel with:
# yum downgrade won't work for the kernel because it's a protected package, so let's use rpm yumdownloader rpm -Uv --oldpackage name-of-file.rpm rebootThere will be some performance impact that I'm not able to quantify and I'm not yet able to tell you how to use it to debug memory leaks, but there's plenty of documentation on the internet about kmemleak.
-
Hello,
We have the same problem here on multiple servers.
We also have 10G interfaces. We use
extlocal SR.(::bbxl0271) (2 running) [08:43 bbxl0271 ~]# lsmod | sort -k 2 -n -r ipv6 548864 313 nf_nat_ipv6 sunrpc 413696 18 lockd,nfsv3,nfs_acl,nfs ixgbe 380928 0 fscache 380928 1 nfs nfs 307200 2 nfsv3 libata 274432 2 libahci,ahci xhci_hcd 258048 1 xhci_pci scsi_mod 253952 15 fcoe,scsi_dh_emc,sd_mod,dm_multipath,scsi_dh_alua,scsi_transport_fc,usb_storage,libfc,bnx2fc,uas,megaraid_sas,libata,sg,scsi_dh_rdac,scsi_dh_hp_sw aesni_intel 200704 0 megaraid_sas 167936 4 nf_conntrack 163840 6 xt_conntrack,nf_nat,nf_nat_ipv6,nf_nat_ipv4,openvswitch,nf_conncount bnx2fc 159744 0 dm_mod 151552 5 dm_multipath openvswitch 147456 12 libfc 147456 3 fcoe,bnx2fc,libfcoe hid 122880 2 usbhid,hid_generic mei 114688 1 mei_me lockd 110592 2 nfsv3,nfs cnic 81920 1 bnx2fc libfcoe 77824 2 fcoe,bnx2fc usb_storage 73728 1 uas scsi_transport_fc 69632 3 fcoe,libfc,bnx2fc ipmi_si 65536 0 ipmi_msghandler 61440 2 ipmi_devintf,ipmi_si usbhid 57344 0 sd_mod 53248 5 tun 49152 0 nfsv3 49152 1 x_tables 45056 6 xt_conntrack,iptable_filter,xt_multiport,xt_tcpudp,ipt_REJECT,ip_tables mei_me 45056 0 sg 40960 0 libahci 40960 1 ahci ahci 40960 0 8021q 40960 0 nf_nat 36864 3 nf_nat_ipv6,nf_nat_ipv4,openvswitch fcoe 32768 0 dm_multipath 32768 0 uas 28672 0 lpc_ich 28672 0 ip_tables 28672 2 iptable_filter i2c_i801 28672 0 cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel uio 20480 1 cnic scsi_dh_alua 20480 0 nf_defrag_ipv6 20480 2 nf_conntrack,openvswitch mrp 20480 1 8021q ipmi_devintf 20480 0 aes_x86_64 20480 1 aesni_intel acpi_power_meter 20480 0 xt_tcpudp 16384 9 xt_multiport 16384 1 xt_conntrack 16384 5 xhci_pci 16384 0 stp 16384 1 garp skx_edac 16384 0 scsi_dh_rdac 16384 0 scsi_dh_hp_sw 16384 0 scsi_dh_emc 16384 0 pcbc 16384 0 nsh 16384 1 openvswitch nfs_acl 16384 1 nfsv3 nf_reject_ipv4 16384 1 ipt_REJECT nf_nat_ipv6 16384 1 openvswitch nf_nat_ipv4 16384 1 openvswitch nf_defrag_ipv4 16384 1 nf_conntrack nf_conncount 16384 1 openvswitch llc 16384 2 stp,garp libcrc32c 16384 3 nf_conntrack,nf_nat,openvswitch ipt_REJECT 16384 3 iptable_filter 16384 1 intel_rapl_perf 16384 0 intel_powerclamp 16384 0 hid_generic 16384 0 grace 16384 1 lockd glue_helper 16384 1 aesni_intel ghash_clmulni_intel 16384 0 garp 16384 1 8021q crypto_simd 16384 1 aesni_intel crct10dif_pclmul 16384 0 crc_ccitt 16384 1 ipv6 crc32_pclmul 16384 0I'll install the
kmemleakkernel on one server today. -
I tried to install the kernel as described, but I got an error :
# yum install http://koji.xcp-ng.org/kojifiles/work/tasks/7620/17620/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm Loaded plugins: fastestmirror kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm | 30 MB 00:00:03 Examining /var/tmp/yum-root-Uyd1Lb/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm: kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 /var/tmp/yum-root-Uyd1Lb/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm: does not update installed package. Error: Nothing to doThe host is a 8.1 up to date (latest patchs are installed but I did not reboot yep after).
-
FYI : On one host that has the problem I only have 1 Debian VM that does only one thing : netdata. It is a netdata that get flows from other netdata and that is polled by a prometheus server (which is on another host).
-
As soon as I stopped the last VM on the host, there is no more memleak.
-
More informations:
# rpm -ivh http://koji.xcp-ng.org/kojifiles/work/tasks/7620/17620/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm Retrieving http://koji.xcp-ng.org/kojifiles/work/tasks/7620/17620/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm Preparing... ################################# [100%] package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64 (which is newer than kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64) is already installed file /boot/System.map-4.19.0+1 from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64 file /boot/config-4.19.0+1 from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64 file /boot/vmlinuz-4.19.0+1 from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64 file /lib/modules/4.19.0+1/kernel/fs/nfs/nfs.ko from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64 file /lib/modules/4.19.0+1/kernel/net/netfilter/nf_conntrack.ko from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64 file /lib/modules/4.19.0+1/modules.order from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64 -
I built a new kernel with memleak, which should install correctly. I've updated my post with the instructions above.
-
@stormi Ok thank you. The installation works.
I made another test: use the kernel-alt 4.19.108 on one box. It seems I did not have the issue anymore!
-
On server with kmemleak kernel, I get this error:
# echo scan > /sys/kernel/debug/kmemleak -bash: echo: write error: Device or resource busyDigging a litlle bit, and I found :
# dmesg | grep memleak [ 0.677307] kmemleak: Kernel memory leak detector disabled [ 2.701225] kmemleak: Early log buffer exceeded (5128), please increase DEBUG_KMEMLEAK_EARLY_LOG_SIZESo kmemleak is still disabled as DEBUG_KMEMLEAK_EARLY_LOG_SIZE seems to be too small
@stormi Could you rebuild a kernel with an increased DEBUG_KMEMLEAK_EARLY_LOG_SIZE?Thank you!
-
Pinging @r1 about this.
-
-
-
@r1 installed and it works :
# dmesg | grep kmem [ 6.181218] kmemleak: Kernel memory leak detector initialized [ 6.181223] kmemleak: Automatic memory scanning thread startedI will check the leaks tomorrow.
-
@r1 @stormi for now I don't have any memleak detected :
# uname -a Linux bb0272.aquaray.com 4.19.0+1 #1 SMP Thu Nov 19 15:15:04 CET 2020 x86_64 x86_64 x86_64 GNU/Linux # yum list installed | grep kernel kernel.x86_64 4.19.19-6.0.12.1.2.kmemleak.xcpng8.1 # cat /sys/kernel/debug/kmemleak # echo scan > /sys/kernel/debug/kmemleak # cat /sys/kernel/debug/kmemleak -
FYI: I confirm that on the server running the
altkernel I do not have anymore the memleak problem :# yum list installed | grep kernel kernel.x86_64 4.19.19-6.0.12.1.1.kmemleak.xcpng8.1 kernel-alt.x86_64 4.19.108-3.xcpng8.1 @xcp-ng-base # uname -a Linux bb0266.aquaray.com 4.19.108 #1 SMP Mon Mar 23 15:04:05 CET 2020 x86_64 x86_64 x86_64 GNU/Linux #Memory graph from this server :
Obviously, the alt-kernel has been installed ~2020-11-13.
On the server running the memleak kernel (since yesterday) :
-
@delaf Weird but interesting!
-
@olivierlambert @delaf what we know from
kmemleakso far is that it will only scan and report unreferenced objects. If any kernel module / kernel itself is still holding(referencing) the memory then it may not show up. We are evaluating other options to find this.kernel-altis more related to upstream, so either this issue is known and fixed in upstream or it might have been introduced from kernel updates.The oldest kernel available is
4.19.19-6.0.10.1.xcpng8.1, is it possible to install it and see if the issue repeats? -
kernel-altalso means that all the drivers used are buillt-in, no those from the driver packages that take precedence over the built-ins when running the main kernel. -
Yes, all drivers are stock kernel modules for
kernel-alt. It would be interesting to see the behavior by disablingupdatesandoverride. I think we can try both. 1st check if the downgraded kernel shows same symptoms and then disabling update drivers. -
Ok, so now we know for sure that it's related to the kernel or one of the drivers.
Let me summarize all the tests that users affected by the issue can do to help find what causes it:
- Test with kmemleak, hoping that it may be able to detect something. No luck for now for @delaf who tried.
- Test with the oldest kernel (
4.19.19-6.0.10.1.xcpng8.1). I doubt it will yield results, but that would allow to be sure. If it does yield results, then it will allow to search towards a specific patch. - With the current kernel, give priority to built-in drivers. If this gives good results, this will mean that the leak is in one of the drivers that are provided through driver RPMs. Two ways:
- A bit riskier but we'd still be interested in the results: disable them all so that only built-in kernel drivers are used. For this, edit
/etc/modprobe.d/dist.confand changesearch override updates extra built-in weak-updatesintosearch extra built-in weak-updates override updates, then rundepmod -aand reboot. Don't forget to restore the original contents after the tests. - Another way, that allows to select specific drivers one by one:
- Identify a few drivers that you want to check in the output of
lsmod. For exampleixgbe. - Find where the currently used driver is on the filesystem:
modinfo ixgbe | head -n 1 - If the path contains "/updates/", it's not a kernel built-in. Rename the file to
name_of_file.save.depmod -a.reboot. The kernel will then use its built-in driver. - If nothing changes, restore the file and try another.
- Identify a few drivers that you want to check in the output of
- A bit riskier but we'd still be interested in the results: disable them all so that only built-in kernel drivers are used. For this, edit
I also intend to build a new
ixgbedriver, just in case we're lucky and it's the culprit, since every affected user uses it.
