Group Details Private

XCP-ng Center Team

Developers of XCP-ng Center

Member List

  • RE: Alert: Control Domain Memory Usage

    @r1

    [02:27 xen19 ~]# uname -a
    Linux xen19 4.19.142 #1 SMP Tue Nov 3 11:27:36 CET 2020 x86_64 x86_64 x86_64 GNU/Linux
    
    [02:30 xen19 ~]# yum list installed | grep kernel
    kernel.x86_64                   4.19.19-7.0.9.1.xcpng8.2    @xcp-ng-updates     
    kernel-alt.x86_64               4.19.142-1.xcpng8.2         @xcp-ng-base 
    

    memory graph so far:

    ca10cb2d-fbfb-48b0-8edd-237fc56ff38f-grafik.png

    looking very good!!!

    posted in Compute
  • RE: Alert: Control Domain Memory Usage

    @r1 our complete setup:

    [FreeNAS NFS] <----shared-storage----> [Pool of 2 servers (xen22 + xen23)] ----XAPI---> [XO from sources] -----remote----> [FreeNAS NFS]

    All [servers] are real hardware servers, no VMs involved.

    Same chain of servers for xen19, execpt there are more pool members (and VMs).

    posted in Compute
  • RE: Alert: Control Domain Memory Usage

    @stormi I'm not complete sure if I did the iso upgrade or not... 😕

    But it's a good idea to reinstall the poolmaster from scratch...

    posted in Compute
  • RE: Alert: Control Domain Memory Usage

    Good news from the kernel-alt (server xen19): No RAM leaks so far 🙂

    b6ca8f80-8bef-4dfe-9064-5823f6aba907-grafik.png

    posted in Compute
  • RE: Alert: Control Domain Memory Usage

    @stormi on server xen19 I think so, on server xen22 I'm not sure

    I looked more close on my memory graphs and saw, that the memory baseline increases every night:

    "bump" every day:
    ca76f2fc-89aa-49fb-aa33-5e568070fbc4-grafik.png

    closer look in week 53:
    692a8c57-f2b5-4284-8cab-be3a045fb179-grafik.png

    Dez 31. - Jan 01.
    8a267bb0-97c3-481c-9da9-bd031093496e-grafik.png

    Our Backups run from 18.00 till 3 or 4 in the morning (including coalesce).

    --> maybe the heavy IO load leads to memory leaks "somewhere"?

    posted in Compute
  • RE: Alert: Control Domain Memory Usage

    @r1 yes, ever line "installed" in yum.log is an Upgrade from XCP-ng.
    Problems started with XCP-ng 8.x

    posted in Compute
  • RE: Alert: Control Domain Memory Usage

    @r1 in general we do not restart many of our VMs, its all very static, only manual operated

    xen19 is now rebootet (we need it in production) with kernel-alt - highest id is currently 4

    xen22 (pool master of another affected pool) - highest id is curently 30

    memory graphs of xen22
    61ef0b14-6ec2-46ea-96e6-6efbd30eb528-grafik.png

    yum.log of xen22 (Problem here also after installing kernel-4.19.19-6.0.10.1.xcpng8.1.x86_64)

    yum.log.5.gz:Dec 19 00:52:47 Updated: kernel-4.4.52-4.0.12.x86_6
    yum.log.3.gz:Nov 08 10:07:40 Updated: kernel-4.4.52-4.0.13.x86_64
    yum.log.1:Apr 10 20:31:01 Installed: kernel-4.19.19-6.0.10.1.xcpng8.1.x86_64
    yum.log.1:Aug 31 23:10:50 Updated: kernel-4.19.19-6.0.11.1.xcpng8.1.x86_64
    yum.log.1:Dec 11 18:00:54 Updated: kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
    yum.log.1:Dec 19 12:52:00 Updated: kernel-4.19.19-6.0.13.1.xcpng8.1.x86_64
    yum.log.1:Dec 19 12:54:13 Updated: kernel-4.19.19-7.0.9.1.xcpng8.2.x86_64
    
    posted in Compute
  • RE: Alert: Control Domain Memory Usage

    looked in my yum.log on this server (xen19):

    our problems startet exactly since "Apr 10 18:10:29 Installed: kernel-4.19.19-6.0.10.1.xcpng8.1.x86_64"

    yum.log.4.gz:Oct 03 17:35:54 Installed: kernel-4.4.52-4.0.7.1.x86_64
    yum.log.4.gz:Nov 20 18:29:29 Updated: kernel-4.4.52-4.0.12.x86_64
    yum.log.2.gz:Oct 10 20:19:31 Updated: kernel-4.4.52-4.0.13.x86_64
    yum.log.1:Apr 10 18:10:29 Installed: kernel-4.19.19-6.0.10.1.xcpng8.1.x86_64
    yum.log.1:Jul 07 17:46:34 Updated: kernel-4.19.19-6.0.11.1.xcpng8.1.x86_64
    yum.log.1:Dec 10 17:59:07 Updated: kernel-4.19.19-6.0.12.1.xcpng8.1.x86_64
    yum.log.1:Dec 19 13:53:39 Updated: kernel-4.19.19-6.0.13.1.xcpng8.1.x86_64
    yum.log.1:Dec 19 13:55:20 Updated: kernel-4.19.19-7.0.9.1.xcpng8.2.x86_64
    yum.log:Jan 18 17:35:07 Installed: kernel-alt-4.19.142-1.xcpng8.2.x86_64
    
    posted in Compute
  • RE: Alert: Control Domain Memory Usage

    I noticed in my monitoring graphs, that since we have this issue, the SWAP is not used like before the issue:

    b07111c8-7c21-4351-85fa-2acc7030bb2d-grafik.png

    posted in Compute
  • RE: Alert: Control Domain Memory Usage

    We also have an issue with growing control domain memory:

    • XCP-ng 8.2
    • NFS shared storage
    • the poolmaster (xen19 is one of them) are more affected than pool members

    d1fac6d1-997b-4a7c-898c-d4a9c398b566-grafik.png

    Today I install the alternate kernel on one of our poolmaster to see if that resolves our issue.

    posted in Compute