XCP-ng

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups

    Solved Alert: Control Domain Memory Usage

    Compute
    20
    193
    31693
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • delaf
      delaf @delaf last edited by

      @stormi @r1

      I tried to install the kernel as described, but I got an error :

      # yum install http://koji.xcp-ng.org/kojifiles/work/tasks/7620/17620/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm
      Loaded plugins: fastestmirror
      kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm                                                     |  30 MB  00:00:03
      Examining /var/tmp/yum-root-Uyd1Lb/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm: kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64
      /var/tmp/yum-root-Uyd1Lb/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm: does not update installed package.
      Error: Nothing to do
      

      The host is a 8.1 up to date (latest patchs are installed but I did not reboot yep after).

      delaf 1 Reply Last reply Reply Quote 0
      • delaf
        delaf @delaf last edited by

        FYI : On one host that has the problem I only have 1 Debian VM that does only one thing : netdata. It is a netdata that get flows from other netdata and that is polled by a prometheus server (which is on another host).

        Screen Shot 2020-11-13 at 09.26.10.png

        delaf 1 Reply Last reply Reply Quote 0
        • delaf
          delaf @delaf last edited by

          As soon as I stopped the last VM on the host, there is no more memleak.

          1 Reply Last reply Reply Quote 0
          • delaf
            delaf @delaf last edited by

            More informations:

            # rpm -ivh http://koji.xcp-ng.org/kojifiles/work/tasks/7620/17620/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm
            Retrieving http://koji.xcp-ng.org/kojifiles/work/tasks/7620/17620/kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64.rpm
            Preparing...                          ################################# [100%]
                package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64 (which is newer than kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64) is already installed
                file /boot/System.map-4.19.0+1 from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
                file /boot/config-4.19.0+1 from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
                file /boot/vmlinuz-4.19.0+1 from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
                file /lib/modules/4.19.0+1/kernel/fs/nfs/nfs.ko from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
                file /lib/modules/4.19.0+1/kernel/net/netfilter/nf_conntrack.ko from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
                file /lib/modules/4.19.0+1/modules.order from install of kernel-4.19.19-6.0.12.1.kmemleak.1.xcpng8.1.x86_64 conflicts with file from package kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
            1 Reply Last reply Reply Quote 0
            • stormi
              stormi Vates 🪐 XCP-ng Team 🚀 last edited by stormi

              I built a new kernel with memleak, which should install correctly. I've updated my post with the instructions above.

              delaf 2 Replies Last reply Reply Quote 0
              • delaf
                delaf @stormi last edited by

                @stormi Ok thank you. The installation works.

                I made another test: use the kernel-alt 4.19.108 on one box. It seems I did not have the issue anymore!

                1 Reply Last reply Reply Quote 0
                • delaf
                  delaf @stormi last edited by

                  On server with kmemleak kernel, I get this error:

                  # echo scan > /sys/kernel/debug/kmemleak 
                  -bash: echo: write error: Device or resource busy
                  

                  Digging a litlle bit, and I found :

                  # dmesg | grep memleak
                  [    0.677307] kmemleak: Kernel memory leak detector disabled
                  [    2.701225] kmemleak: Early log buffer exceeded (5128), please increase DEBUG_KMEMLEAK_EARLY_LOG_SIZE
                  

                  So kmemleak is still disabled as DEBUG_KMEMLEAK_EARLY_LOG_SIZE seems to be too small 😞
                  @stormi Could you rebuild a kernel with an increased DEBUG_KMEMLEAK_EARLY_LOG_SIZE?

                  Thank you!

                  1 Reply Last reply Reply Quote 0
                  • stormi
                    stormi Vates 🪐 XCP-ng Team 🚀 last edited by

                    Pinging @r1 about this.

                    R 1 Reply Last reply Reply Quote 0
                    • R
                      r1 XCP-ng Team 🚀 @stormi last edited by

                      @stormi @delaf will check this and do required changes.

                      R 1 Reply Last reply Reply Quote 0
                      • R
                        r1 XCP-ng Team 🚀 @r1 last edited by

                        @delaf and others, you can download and install and update from link which should work fine.

                        After the system is running for some time, user can # echo scan > /sys/kernel/debug/kmemleak and then # cat /sys/kernel/debug/kmemleak to see if there are any unreferenced objects floating in memory.

                        delaf 1 Reply Last reply Reply Quote 0
                        • delaf
                          delaf @r1 last edited by

                          @r1 installed and it works :

                          # dmesg | grep kmem
                          [    6.181218] kmemleak: Kernel memory leak detector initialized
                          [    6.181223] kmemleak: Automatic memory scanning thread started
                          

                          I will check the leaks tomorrow.

                          delaf 1 Reply Last reply Reply Quote 1
                          • delaf
                            delaf @delaf last edited by delaf

                            @r1 @stormi for now I don't have any memleak detected :

                            # uname -a
                            Linux bb0272.aquaray.com 4.19.0+1 #1 SMP Thu Nov 19 15:15:04 CET 2020 x86_64 x86_64 x86_64 GNU/Linux
                            # yum list installed | grep kernel
                            kernel.x86_64                   4.19.19-6.0.12.1.2.kmemleak.xcpng8.1
                            # cat /sys/kernel/debug/kmemleak
                            # echo scan > /sys/kernel/debug/kmemleak
                            # cat /sys/kernel/debug/kmemleak
                            delaf 1 Reply Last reply Reply Quote 0
                            • delaf
                              delaf @delaf last edited by

                              FYI: I confirm that on the server running the alt kernel I do not have anymore the memleak problem :

                              # yum list installed | grep kernel
                              kernel.x86_64                   4.19.19-6.0.12.1.1.kmemleak.xcpng8.1
                              kernel-alt.x86_64               4.19.108-3.xcpng8.1         @xcp-ng-base
                              # uname -a
                              Linux bb0266.aquaray.com 4.19.108 #1 SMP Mon Mar 23 15:04:05 CET 2020 x86_64 x86_64 x86_64 GNU/Linux
                              #
                              

                              Memory graph from this server :
                              Screen Shot 2020-11-20 at 12.14.54.png

                              Obviously, the alt-kernel has been installed ~2020-11-13.

                              On the server running the memleak kernel (since yesterday) :
                              Screen Shot 2020-11-20 at 12.16.21.png

                              olivierlambert 1 Reply Last reply Reply Quote 0
                              • olivierlambert
                                olivierlambert Vates 🪐 Founder & CEO 🦸 @delaf last edited by

                                @delaf Weird but interesting!

                                R 1 Reply Last reply Reply Quote 0
                                • R
                                  r1 XCP-ng Team 🚀 @olivierlambert last edited by

                                  @olivierlambert @delaf what we know from kmemleak so far is that it will only scan and report unreferenced objects. If any kernel module / kernel itself is still holding(referencing) the memory then it may not show up. We are evaluating other options to find this.

                                  kernel-alt is more related to upstream, so either this issue is known and fixed in upstream or it might have been introduced from kernel updates.

                                  The oldest kernel available is 4.19.19-6.0.10.1.xcpng8.1, is it possible to install it and see if the issue repeats?

                                  delaf 1 Reply Last reply Reply Quote 0
                                  • stormi
                                    stormi Vates 🪐 XCP-ng Team 🚀 last edited by

                                    kernel-alt also means that all the drivers used are buillt-in, no those from the driver packages that take precedence over the built-ins when running the main kernel.

                                    R 1 Reply Last reply Reply Quote 0
                                    • R
                                      r1 XCP-ng Team 🚀 @stormi last edited by r1

                                      Yes, all drivers are stock kernel modules for kernel-alt. It would be interesting to see the behavior by disabling updates and override. I think we can try both. 1st check if the downgraded kernel shows same symptoms and then disabling update drivers.

                                      1 Reply Last reply Reply Quote 0
                                      • stormi
                                        stormi Vates 🪐 XCP-ng Team 🚀 last edited by stormi

                                        Ok, so now we know for sure that it's related to the kernel or one of the drivers.

                                        Let me summarize all the tests that users affected by the issue can do to help find what causes it:

                                        • Test with kmemleak, hoping that it may be able to detect something. No luck for now for @delaf who tried.
                                        • Test with the oldest kernel (4.19.19-6.0.10.1.xcpng8.1). I doubt it will yield results, but that would allow to be sure. If it does yield results, then it will allow to search towards a specific patch.
                                        • With the current kernel, give priority to built-in drivers. If this gives good results, this will mean that the leak is in one of the drivers that are provided through driver RPMs. Two ways:
                                          • A bit riskier but we'd still be interested in the results: disable them all so that only built-in kernel drivers are used. For this, edit /etc/modprobe.d/dist.conf and change search override updates extra built-in weak-updates into search extra built-in weak-updates override updates, then run depmod -a and reboot. Don't forget to restore the original contents after the tests.
                                          • Another way, that allows to select specific drivers one by one:
                                            • Identify a few drivers that you want to check in the output of lsmod. For example ixgbe.
                                            • Find where the currently used driver is on the filesystem: modinfo ixgbe | head -n 1
                                            • If the path contains "/updates/", it's not a kernel built-in. Rename the file to name_of_file.save. depmod -a. reboot. The kernel will then use its built-in driver.
                                            • If nothing changes, restore the file and try another.

                                        I also intend to build a new ixgbe driver, just in case we're lucky and it's the culprit, since every affected user uses it.

                                        delaf 1 Reply Last reply Reply Quote 0
                                        • delaf
                                          delaf @r1 last edited by

                                          @r1 I do not manage to install the old kernel. Any idea?

                                          # yum downgrade "kernel == 4.19.19-6.0.10.1.xcpng8.1"
                                          Loaded plugins: fastestmirror
                                          Loading mirror speeds from cached hostfile
                                          Excluding mirror: updates.xcp-ng.org
                                           * xcp-ng-base: mirrors.xcp-ng.org
                                          Excluding mirror: updates.xcp-ng.org
                                           * xcp-ng-updates: mirrors.xcp-ng.org
                                          Resolving Dependencies
                                          --> Running transaction check
                                          ---> Package kernel.x86_64 0:4.19.19-6.0.10.1.xcpng8.1 will be a downgrade
                                          ---> Package kernel.x86_64 0:4.19.19-6.0.12.1.xcpng8.1 will be erased
                                          --> Finished Dependency Resolution
                                          Error: Trying to remove "kernel", which is protected
                                          
                                          1 Reply Last reply Reply Quote 0
                                          • Danp
                                            Danp Top contributor 💪 last edited by

                                            @delaf I ran into the same thing recently. See solution below --

                                            https://xcp-ng.org/forum/post/33291

                                            delaf 1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post