Alert: Control Domain Memory Usage



  • @olivierlambert said in Alert: Control Domain Memory Usage:

    Netdata

    Hello Oliver,
    thanks for your quick Response.

    Yes, htop is much prettier, but it doen`t see more the top, at least in this case:

    htop.JPG

    I think, Netdata will also just see those things?


  • XCP-ng Team

    1. Can you restart the toolstack and see if it changes memory footprint?
    2. Anything suspicious in dmesg?


    1. toolstack restart didn`t change much
    2. good idea: Indeed there are some entries in dmesg:
    [4286099.036105] Status code returned 0xc000006d STATUS_LOGON_FAILURE
    [4286099.036116] CIFS VFS: Send error in SessSetup = -13
    
    

    With "mount" i can see two old cifs iso libaries, which have allready been detached from the pool.

    I was able to unmount one of them.

    The other one is busy:

    umount: /run/sr-mount/43310179-cf17-1942-7845-59bf55cb3550: target is busy.
    

    I will have a look into it a little later and report.

    Thanks.



  • I was not able to find what was keeping the target busy.

    I did a lazy unmount "umount -l", since then i have no new dmesg lines, but the memory isn`t freed.

    So i`m not sure, if this is/was the reason for the Problem. Other Servers in the pool have this same old mount, but do not use this much of RAM. But i will check cifs mounts, the next time i see this Problem.

    I`m still looking for a way how to find out what is using this memory, when there is no process visible which uses it. Even though this may be a common Linux question 🙂



  • I had a look at another server in another pool this morning, which also had memory alerts.

    Gues what: There was an unresponsive CIFS ISO share too.

    The share looked like it was online form XCP-NG Center, but the containing isos were not shown.

    I could not detach it from XCP-NG Center, but after a few tries i could do a pbd-unplug.

    This share was on a smal NAS, which may have been restarted during the pool was running.

    So maybe something happens when you have a unresponsive CIFS ISO share, that eats up your RAM.

    If this is realy the reason, i think this behaviour changed in never versions om XCP-NG/Xenserver. I had dozens of servers, where the ISO store credentials changed or/and the CIFS servers has been restarted, without this Problem.



  • We're seeing the same kind of development on two servers. Nothing interesting in dmesg or so.

    Will have to resize for now I guess.



  • @dave said in Alert: Control Domain Memory Usage:

    @olivierlambert said in Alert: Control Domain Memory Usage:

    Netdata

    Hello Oliver,
    thanks for your quick Response.

    Yes, htop is much prettier, but it doen`t see more the top, at least in this case:

    htop.JPG

    I think, Netdata will also just see those things?

    Hi there! Hope it's okay that I answer regarding this topic....

    We've also this Error and in 'Notifications' in the XCP-NG Center I can see the following:

    d2b733ea-2117-43d9-818d-ecac95c9d40b-grafik.png

    Is there any possibility to resolve this error without restarting the hole host? How can I restart the toolstack? Can I do it without an interrupt of the business?
    We've also raised up the Memory for the Controller Domain from 4 --> 8GB.....


  • XCP-ng Team

    In Xen Orchestra, restarting the toolstack is clicking on a button. This won't affect running VMs (except if you are in the middle of an hypervisor operation, like migrating/exporting). Restarting the toolstack won't fundamentally solve your issue.

    Adding more memory into the dom0 is a good idea in general (depending on the total memory you have). 8GiB is a good start. This will require a host reboot in any case, to be taken into account.



  • @olivierlambert said in Alert: Control Domain Memory Usage:

    In Xen Orchestra, restarting the toolstack is clicking on a button. This won't affect running VMs (except if you are in the middle of an hypervisor operation, like migrating/exporting). Restarting the toolstack won't fundamentally solve your issue.

    Adding more memory into the dom0 is a good idea in general (depending on the total memory you have). 8GiB is a good start. This will require a host reboot in any case, to be taken into account.

    Hi Olivier, thanks for your fast reply. We've identified on our systems that the Openvswitch needs a lot of memory and therefore we wanna restart it.
    Is it possible to do this without restarting the hole host and without an downtime?


  • XCP-ng Team

    Just use systemd restart to do that (eg systemctl restart <whatever>).

    Regarding growing memory on your host, this will require a reboot though.


Log in to reply
 

XCP-ng Pro Support

XCP-ng Pro Support