XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Alert: Control Domain Memory Usage

    Scheduled Pinned Locked Moved Solved Compute
    194 Posts 21 Posters 200.6k Views 16 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • stormiS Offline
      stormi Vates 🪐 XCP-ng Team
      last edited by

      Pinging @r1 about this.

      R 1 Reply Last reply Reply Quote 0
      • R Offline
        r1 XCP-ng Team @stormi
        last edited by

        @stormi @delaf will check this and do required changes.

        R 1 Reply Last reply Reply Quote 0
        • R Offline
          r1 XCP-ng Team @r1
          last edited by

          @delaf and others, you can download and install and update from link which should work fine.

          After the system is running for some time, user can # echo scan > /sys/kernel/debug/kmemleak and then # cat /sys/kernel/debug/kmemleak to see if there are any unreferenced objects floating in memory.

          delafD 1 Reply Last reply Reply Quote 0
          • delafD Offline
            delaf @r1
            last edited by

            @r1 installed and it works :

            # dmesg | grep kmem
            [    6.181218] kmemleak: Kernel memory leak detector initialized
            [    6.181223] kmemleak: Automatic memory scanning thread started
            

            I will check the leaks tomorrow.

            delafD 1 Reply Last reply Reply Quote 1
            • delafD Offline
              delaf @delaf
              last edited by delaf

              @r1 @stormi for now I don't have any memleak detected :

              # uname -a
              Linux bb0272.aquaray.com 4.19.0+1 #1 SMP Thu Nov 19 15:15:04 CET 2020 x86_64 x86_64 x86_64 GNU/Linux
              # yum list installed | grep kernel
              kernel.x86_64                   4.19.19-6.0.12.1.2.kmemleak.xcpng8.1
              # cat /sys/kernel/debug/kmemleak
              # echo scan > /sys/kernel/debug/kmemleak
              # cat /sys/kernel/debug/kmemleak
              delafD 1 Reply Last reply Reply Quote 0
              • delafD Offline
                delaf @delaf
                last edited by

                FYI: I confirm that on the server running the alt kernel I do not have anymore the memleak problem :

                # yum list installed | grep kernel
                kernel.x86_64                   4.19.19-6.0.12.1.1.kmemleak.xcpng8.1
                kernel-alt.x86_64               4.19.108-3.xcpng8.1         @xcp-ng-base
                # uname -a
                Linux bb0266.aquaray.com 4.19.108 #1 SMP Mon Mar 23 15:04:05 CET 2020 x86_64 x86_64 x86_64 GNU/Linux
                #
                

                Memory graph from this server :
                Screen Shot 2020-11-20 at 12.14.54.png

                Obviously, the alt-kernel has been installed ~2020-11-13.

                On the server running the memleak kernel (since yesterday) :
                Screen Shot 2020-11-20 at 12.16.21.png

                olivierlambertO 1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates 🪐 Co-Founder CEO @delaf
                  last edited by

                  @delaf Weird but interesting!

                  R 1 Reply Last reply Reply Quote 0
                  • R Offline
                    r1 XCP-ng Team @olivierlambert
                    last edited by

                    @olivierlambert @delaf what we know from kmemleak so far is that it will only scan and report unreferenced objects. If any kernel module / kernel itself is still holding(referencing) the memory then it may not show up. We are evaluating other options to find this.

                    kernel-alt is more related to upstream, so either this issue is known and fixed in upstream or it might have been introduced from kernel updates.

                    The oldest kernel available is 4.19.19-6.0.10.1.xcpng8.1, is it possible to install it and see if the issue repeats?

                    delafD 1 Reply Last reply Reply Quote 0
                    • stormiS Offline
                      stormi Vates 🪐 XCP-ng Team
                      last edited by

                      kernel-alt also means that all the drivers used are buillt-in, no those from the driver packages that take precedence over the built-ins when running the main kernel.

                      R 1 Reply Last reply Reply Quote 0
                      • R Offline
                        r1 XCP-ng Team @stormi
                        last edited by r1

                        Yes, all drivers are stock kernel modules for kernel-alt. It would be interesting to see the behavior by disabling updates and override. I think we can try both. 1st check if the downgraded kernel shows same symptoms and then disabling update drivers.

                        1 Reply Last reply Reply Quote 0
                        • stormiS Offline
                          stormi Vates 🪐 XCP-ng Team
                          last edited by stormi

                          Ok, so now we know for sure that it's related to the kernel or one of the drivers.

                          Let me summarize all the tests that users affected by the issue can do to help find what causes it:

                          • Test with kmemleak, hoping that it may be able to detect something. No luck for now for @delaf who tried.
                          • Test with the oldest kernel (4.19.19-6.0.10.1.xcpng8.1). I doubt it will yield results, but that would allow to be sure. If it does yield results, then it will allow to search towards a specific patch.
                          • With the current kernel, give priority to built-in drivers. If this gives good results, this will mean that the leak is in one of the drivers that are provided through driver RPMs. Two ways:
                            • A bit riskier but we'd still be interested in the results: disable them all so that only built-in kernel drivers are used. For this, edit /etc/modprobe.d/dist.conf and change search override updates extra built-in weak-updates into search extra built-in weak-updates override updates, then run depmod -a and reboot. Don't forget to restore the original contents after the tests.
                            • Another way, that allows to select specific drivers one by one:
                              • Identify a few drivers that you want to check in the output of lsmod. For example ixgbe.
                              • Find where the currently used driver is on the filesystem: modinfo ixgbe | head -n 1
                              • If the path contains "/updates/", it's not a kernel built-in. Rename the file to name_of_file.save. depmod -a. reboot. The kernel will then use its built-in driver.
                              • If nothing changes, restore the file and try another.

                          I also intend to build a new ixgbe driver, just in case we're lucky and it's the culprit, since every affected user uses it.

                          delafD 1 Reply Last reply Reply Quote 0
                          • delafD Offline
                            delaf @r1
                            last edited by

                            @r1 I do not manage to install the old kernel. Any idea?

                            # yum downgrade "kernel == 4.19.19-6.0.10.1.xcpng8.1"
                            Loaded plugins: fastestmirror
                            Loading mirror speeds from cached hostfile
                            Excluding mirror: updates.xcp-ng.org
                             * xcp-ng-base: mirrors.xcp-ng.org
                            Excluding mirror: updates.xcp-ng.org
                             * xcp-ng-updates: mirrors.xcp-ng.org
                            Resolving Dependencies
                            --> Running transaction check
                            ---> Package kernel.x86_64 0:4.19.19-6.0.10.1.xcpng8.1 will be a downgrade
                            ---> Package kernel.x86_64 0:4.19.19-6.0.12.1.xcpng8.1 will be erased
                            --> Finished Dependency Resolution
                            Error: Trying to remove "kernel", which is protected
                            
                            1 Reply Last reply Reply Quote 0
                            • DanpD Offline
                              Danp Pro Support Team
                              last edited by

                              @delaf I ran into the same thing recently. See solution below --

                              https://xcp-ng.org/forum/post/33291

                              delafD 1 Reply Last reply Reply Quote 1
                              • delafD Offline
                                delaf @stormi
                                last edited by

                                @stormi I have a server with only search extra built-in weak-updates override updates. We will see if it is better.

                                1 Reply Last reply Reply Quote 1
                                • delafD Offline
                                  delaf @Danp
                                  last edited by

                                  @danp Thank you. I have downgraded one server.

                                  @stormi So i have :

                                  • one server with 4.19.19-6.0.10.1.xcpng8.1
                                  • one server with kmemleak kernel
                                  • one server with search extra built-in weak-updates override updates
                                  delafD 1 Reply Last reply Reply Quote 1
                                  • delafD Offline
                                    delaf @delaf
                                    last edited by

                                    @stormi @r1
                                    Four days later, I get:

                                    • one server (266) with alt-kernel: still no problem
                                    • one server (268) with 4.19.19-6.0.10.1.xcpng8.1: no more problem!
                                    • one server (272) with kmemleak kernel: no memleak detected, but the problem is present
                                    • one server (273) with search extra built-in weak-updates override updates: problem still present
                                    R 1 Reply Last reply Reply Quote 1
                                    • stormiS Offline
                                      stormi Vates 🪐 XCP-ng Team
                                      last edited by

                                      @delaf Excellent news, especially if you can replicate on another host to be sure.

                                      This thread is older than kernel-4.19.19-6.0.10.1.xcpng8.1 (especially in @dave's case), but it wouldn't be unlikely that several distinct memory leak causes exist.

                                      1 Reply Last reply Reply Quote 0
                                      • R Offline
                                        r1 XCP-ng Team @delaf
                                        last edited by

                                        @delaf said in Alert: Control Domain Memory Usage:

                                        one server (268) with 4.19.19-6.0.10.1.xcpng8.1: no more problem!

                                        Yeah, we need to be sure that this is a stable kernel and somewhere after this, the memory leak seems to have introduced.

                                        1 Reply Last reply Reply Quote 0
                                        • daveD Offline
                                          dave
                                          last edited by

                                          I currently have:

                                          top - 13:35:31 up 59 days, 17:11,  1 user,  load average: 0.43, 0.36, 0.34
                                          Tasks: 646 total,   1 running, 436 sleeping,   0 stopped,   0 zombie
                                          %Cpu(s):  0.8 us,  1.1 sy,  0.0 ni, 97.5 id,  0.3 wa,  0.0 hi,  0.1 si,  0.2 st
                                          KiB Mem : 12205936 total,   149152 free, 10627080 used,  1429704 buff/cache
                                          KiB Swap:  1048572 total,  1048572 free,        0 used.  1153360 avail Mem
                                          
                                          
                                          top - 13:35:54 up 35 days, 17:29,  1 user,  load average: 0.54, 0.73, 0.77
                                          Tasks: 489 total,   1 running, 324 sleeping,   0 stopped,   0 zombie
                                          %Cpu(s):  3.5 us,  3.4 sy,  0.0 ni, 92.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.4 st
                                          KiB Mem : 12207996 total,   155084 free,  9388032 used,  2664880 buff/cache
                                          KiB Swap:  1048572 total,  1048572 free,        0 used.  2394220 avail Mem
                                          
                                          

                                          both with:

                                          # uname -a
                                          Linux xs01 4.19.0+1 #1 SMP Thu Jun 11 16:18:33 CEST 2020 x86_64 x86_64 x86_64 GNU/Linux
                                          # yum list installed | grep kernel
                                          kernel.x86_64                   4.19.19-6.0.11.1.xcpng8.1   @xcp-ng-updates
                                          
                                          

                                          shall i test something?

                                          1 Reply Last reply Reply Quote 0
                                          • U Offline
                                            umbradark
                                            last edited by

                                            I have a set of hosts on kernel-4.19.19-6.0.11.1.xcpng8.1 and I believe I'm hitting this as well. The OOM seems to kill openvswitch, which takes the host offline and in most cases, the VMs as well.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post