XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Alert: Control Domain Memory Usage

    Scheduled Pinned Locked Moved Solved Compute
    194 Posts 21 Posters 200.6k Views 16 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R Offline
      r1 XCP-ng Team @stormi
      last edited by r1

      Yes, all drivers are stock kernel modules for kernel-alt. It would be interesting to see the behavior by disabling updates and override. I think we can try both. 1st check if the downgraded kernel shows same symptoms and then disabling update drivers.

      1 Reply Last reply Reply Quote 0
      • stormiS Offline
        stormi Vates 🪐 XCP-ng Team
        last edited by stormi

        Ok, so now we know for sure that it's related to the kernel or one of the drivers.

        Let me summarize all the tests that users affected by the issue can do to help find what causes it:

        • Test with kmemleak, hoping that it may be able to detect something. No luck for now for delaf who tried.
        • Test with the oldest kernel (4.19.19-6.0.10.1.xcpng8.1). I doubt it will yield results, but that would allow to be sure. If it does yield results, then it will allow to search towards a specific patch.
        • With the current kernel, give priority to built-in drivers. If this gives good results, this will mean that the leak is in one of the drivers that are provided through driver RPMs. Two ways:
          • A bit riskier but we'd still be interested in the results: disable them all so that only built-in kernel drivers are used. For this, edit /etc/modprobe.d/dist.conf and change search override updates extra built-in weak-updates into search extra built-in weak-updates override updates, then run depmod -a and reboot. Don't forget to restore the original contents after the tests.
          • Another way, that allows to select specific drivers one by one:
            • Identify a few drivers that you want to check in the output of lsmod. For example ixgbe.
            • Find where the currently used driver is on the filesystem: modinfo ixgbe | head -n 1
            • If the path contains "/updates/", it's not a kernel built-in. Rename the file to name_of_file.save. depmod -a. reboot. The kernel will then use its built-in driver.
            • If nothing changes, restore the file and try another.

        I also intend to build a new ixgbe driver, just in case we're lucky and it's the culprit, since every affected user uses it.

        delafD 1 Reply Last reply Reply Quote 0
        • delafD Offline
          delaf @r1
          last edited by

          r1 I do not manage to install the old kernel. Any idea?

          # yum downgrade "kernel == 4.19.19-6.0.10.1.xcpng8.1"
          Loaded plugins: fastestmirror
          Loading mirror speeds from cached hostfile
          Excluding mirror: updates.xcp-ng.org
           * xcp-ng-base: mirrors.xcp-ng.org
          Excluding mirror: updates.xcp-ng.org
           * xcp-ng-updates: mirrors.xcp-ng.org
          Resolving Dependencies
          --> Running transaction check
          ---> Package kernel.x86_64 0:4.19.19-6.0.10.1.xcpng8.1 will be a downgrade
          ---> Package kernel.x86_64 0:4.19.19-6.0.12.1.xcpng8.1 will be erased
          --> Finished Dependency Resolution
          Error: Trying to remove "kernel", which is protected
          
          1 Reply Last reply Reply Quote 0
          • DanpD Offline
            Danp Pro Support Team
            last edited by

            delaf I ran into the same thing recently. See solution below --

            https://xcp-ng.org/forum/post/33291

            delafD 1 Reply Last reply Reply Quote 1
            • delafD Offline
              delaf @stormi
              last edited by

              stormi I have a server with only search extra built-in weak-updates override updates. We will see if it is better.

              1 Reply Last reply Reply Quote 1
              • delafD Offline
                delaf @Danp
                last edited by

                Danp Thank you. I have downgraded one server.

                stormi So i have :

                • one server with 4.19.19-6.0.10.1.xcpng8.1
                • one server with kmemleak kernel
                • one server with search extra built-in weak-updates override updates
                delafD 1 Reply Last reply Reply Quote 1
                • delafD Offline
                  delaf @delaf
                  last edited by

                  stormi r1
                  Four days later, I get:

                  • one server (266) with alt-kernel: still no problem
                  • one server (268) with 4.19.19-6.0.10.1.xcpng8.1: no more problem!
                  • one server (272) with kmemleak kernel: no memleak detected, but the problem is present
                  • one server (273) with search extra built-in weak-updates override updates: problem still present
                  R 1 Reply Last reply Reply Quote 1
                  • stormiS Offline
                    stormi Vates 🪐 XCP-ng Team
                    last edited by

                    delaf Excellent news, especially if you can replicate on another host to be sure.

                    This thread is older than kernel-4.19.19-6.0.10.1.xcpng8.1 (especially in dave's case), but it wouldn't be unlikely that several distinct memory leak causes exist.

                    1 Reply Last reply Reply Quote 0
                    • R Offline
                      r1 XCP-ng Team @delaf
                      last edited by

                      delaf said in Alert: Control Domain Memory Usage:

                      one server (268) with 4.19.19-6.0.10.1.xcpng8.1: no more problem!

                      Yeah, we need to be sure that this is a stable kernel and somewhere after this, the memory leak seems to have introduced.

                      1 Reply Last reply Reply Quote 0
                      • daveD Offline
                        dave
                        last edited by

                        I currently have:

                        top - 13:35:31 up 59 days, 17:11,  1 user,  load average: 0.43, 0.36, 0.34
                        Tasks: 646 total,   1 running, 436 sleeping,   0 stopped,   0 zombie
                        %Cpu(s):  0.8 us,  1.1 sy,  0.0 ni, 97.5 id,  0.3 wa,  0.0 hi,  0.1 si,  0.2 st
                        KiB Mem : 12205936 total,   149152 free, 10627080 used,  1429704 buff/cache
                        KiB Swap:  1048572 total,  1048572 free,        0 used.  1153360 avail Mem
                        
                        
                        top - 13:35:54 up 35 days, 17:29,  1 user,  load average: 0.54, 0.73, 0.77
                        Tasks: 489 total,   1 running, 324 sleeping,   0 stopped,   0 zombie
                        %Cpu(s):  3.5 us,  3.4 sy,  0.0 ni, 92.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.4 st
                        KiB Mem : 12207996 total,   155084 free,  9388032 used,  2664880 buff/cache
                        KiB Swap:  1048572 total,  1048572 free,        0 used.  2394220 avail Mem
                        
                        

                        both with:

                        # uname -a
                        Linux xs01 4.19.0+1 #1 SMP Thu Jun 11 16:18:33 CEST 2020 x86_64 x86_64 x86_64 GNU/Linux
                        # yum list installed | grep kernel
                        kernel.x86_64                   4.19.19-6.0.11.1.xcpng8.1   @xcp-ng-updates
                        
                        

                        shall i test something?

                        1 Reply Last reply Reply Quote 0
                        • U Offline
                          umbradark
                          last edited by

                          I have a set of hosts on kernel-4.19.19-6.0.11.1.xcpng8.1 and I believe I'm hitting this as well. The OOM seems to kill openvswitch, which takes the host offline and in most cases, the VMs as well.

                          1 Reply Last reply Reply Quote 0
                          • stormiS Offline
                            stormi Vates 🪐 XCP-ng Team
                            last edited by

                            So, the difference between 4.19.19-6.0.10.1.xcpng8.1 and 4.19.19-6.0.11.1.xcpng8.1 is two patches meant to reduce the performance overhead of the CROSSTalk vulnerability mitigations.

                            So, assuming from delaf's test results that one of those patches introduced the memory leak, I have built

                            Now here are the tests that you can do:

                            • Reproduce delaf's findings by running kernel-4.19.19-6.0.10.1.xcpng8.1: no more memory leaks?
                            • Test this kernel I built with patch 53 disabled: https://nextcloud.vates.fr/index.php/s/YXWCSEwo8SWkfAZ
                            • Test this kernel I built with patch 62 disabled: https://nextcloud.vates.fr/index.php/s/arj5YfdrkjMKbBy

                            If one of the patches is the cause of the memory leak, then one of the last two should still cause a memory leak and the other one not.

                            delafD 1 Reply Last reply Reply Quote 0
                            • delafD Offline
                              delaf @stormi
                              last edited by delaf

                              stormi I have installed the two kernels

                              272 ~]# yum list installed kernel | grep kernel
                              kernel.x86_64                   4.19.19-6.0.11.1.0.1.patch53disabled.xcpng8.1
                              
                              273 ~]# yum list installed kernel | grep kernel
                              kernel.x86_64                   4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1
                              

                              I have removed the modification in /etc/modprobe.d/dist.conf on server 273.

                              We have to wait a little bit now 😉

                              1 Reply Last reply Reply Quote 1
                              • stormiS Offline
                                stormi Vates 🪐 XCP-ng Team
                                last edited by

                                FYI, the kernel with kmemleak support did detect something for a user who has a support ticket related to dom0 memory usage.

                                delafD 1 Reply Last reply Reply Quote 0
                                • delafD Offline
                                  delaf @stormi
                                  last edited by

                                  stormi For the kernel-4.19.19-6.0.10.1.xcpng8.1 test, i'm not sure it solve the problem because I get a small memory increase. We have to wait a bit more 😕

                                  delafD 1 Reply Last reply Reply Quote 1
                                  • delafD Offline
                                    delaf @delaf
                                    last edited by olivierlambert

                                    stormi

                                    • server 266 with alt-kernel: still no problem.
                                      Screen Shot 2020-12-02 at 10.08.47.png

                                    • server 268 with 4.19.19-6.0.10.1.xcpng8.1: the problem has begun some days ago after some stable days.
                                      Screen Shot 2020-12-02 at 10.03.57.png

                                    • server 272 with 4.19.19-6.0.11.1.0.1.patch53disabled.xcpng8.1:
                                      Screen Shot 2020-12-02 at 10.05.47.png )

                                    • server 273 with 4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1:
                                      Screen Shot 2020-12-02 at 10.05.50.png

                                    It seems that 4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1 is more stable than 4.19.19-6.0.11.1.0.1.patch53disabled.xcpng8.1. But it is a but early to be sure.

                                    delafD 1 Reply Last reply Reply Quote 1
                                    • delafD Offline
                                      delaf @delaf
                                      last edited by delaf

                                      stormi r1 server 273 with 4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1 is still stable and 272 has the memory problem.

                                      • 272
                                        Screen Shot 2020-12-15 at 14.50.31.png

                                      • 273
                                        Screen Shot 2020-12-15 at 14.50.40.png

                                      1 Reply Last reply Reply Quote 0
                                      • stormiS Offline
                                        stormi Vates 🪐 XCP-ng Team
                                        last edited by

                                        Thanks. It looks like I'm doomed to see seemingly contradictory results for every kernel-related issue (this one, and an other one regarding network performance): you don't have any leaks without patch 62, but you had leaks with kernel 4.19.19-6.0.10.1.xcpng8.1 which doesn't have that patch either. So it's hard to conclude anything 😕

                                        1 Reply Last reply Reply Quote 0
                                        • R Offline
                                          rblvlvl
                                          last edited by rblvlvl

                                          Hey Guys,

                                          we are facing the same issue with xcp 8.1.
                                          We can't figure out what uses all this memory (8GB) or how to reduce it. Restarting the Toolstack did nothing and we can't afford a downtime because everything runs in production. Similar systems with same configurations don't show such a behavior.

                                          I can provide you with some output from our system, maybe you can see something or help us finding a solution.

                                          free -m

                                                        total        used        free      shared  buff/cache   available
                                          Mem:           7912        7595          82          33         234          62
                                          Swap:          1023         216         807
                                          

                                          xl top

                                                NAME  STATE   CPU(sec) CPU(%)     MEM(k) MEM(%)  MAXMEM(k) MAXMEM(%) VCPUS NETS NETTX(k) NETRX(k) VBDS   VBD_OO    VBD_RD     VBD_WR   VBD_RSECT   VBD_WSECT SSID
                                            Domain-0 -----r    7308446   52.1    8388608    3.1    8388608       3.1    16    0        0        0    0        0         0          0           0           0    0
                                          

                                          xe vm-param-list uuid | grep memory

                                                                   memory-actual ( RO): 8589934592
                                                                   memory-target ( RO): <unknown>
                                                                 memory-overhead ( RO): 84934656
                                                               memory-static-max ( RW): 8589934592
                                                              memory-dynamic-max ( RW): 8589934592
                                                              memory-dynamic-min ( RW): 8589934592
                                                               memory-static-min ( RW): 8589934592
                                                                          memory (MRO): <not in database>
                                          

                                          lsmod and grub.cfg
                                          lsmod.txt
                                          grub-cgf.txt

                                          top output
                                          Bildschirmfoto 2020-12-30 um 08.55.32.png

                                          Tell me if you need more information or if you have any idea. Thanks.

                                          1 Reply Last reply Reply Quote 0
                                          • olivierlambertO Online
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by

                                            We need more details on the host.

                                            1. Hardware detail (NICs, server model)
                                            2. If all your hardware is fully BIOS/firmware up to date
                                            3. The kind of storage used (iSCSI, FCoE, NFS?)

                                            So far, we couldn't find a real common thing between people, and that's make hard to find the root cause.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post