Memory reporting incorrect values

fred974

Hi,

Could anyone please help me understand wich value I should go by when looking at CPU resources?
XOA:

XO Lite:

Server htop

According to Xen Orchestra, I am using almost all of my 8GB RAM where as in htop, I am only using 1.74G.

Which one should I trust?

Thank you

olivierlambert

Hi,

Check carefully your htop output. See the huge yellow bar? That's the buffer/cache used

edit: also please fix the title, your request is related to memory, not CPU

fred974

@olivierlambert said in CPU reporting incorrect values:

Check carefully your htop output. See the huge yellow bar? That's the buffer/cache used

So Xen Orchestra see cached value as ram in use and not the actual active used ram?

olivierlambert

It's not XO. It's what's reported by the tools.

fred974

@olivierlambert But all modern OS will use the RAM as cache for performance etc. So how can I ensure I get the correct info? At the moment I bombarded with email saying high RAM usage

planedrop

@fred974 This is normal behavior, most modern OSes will use up as much RAM as they possibly can for caching, unused RAM is wasted RAM.

I'd adjust your alert settings if you're getting high RAM usage alerts. You also could change things in the OS to prevent as much RAM being used for caching, but I'd avoid that if possible.

The OS reports, via the tools, to the hypervisor about it's RAM usage, so if the OS is caching a lot, and therefore using most of it's RAM, it will report that to the hypervisor.

I think, could be wrong, Windows doesn't report cached RAM via the tools so it may show "real" RAM usage, but this is OS specific stuff and not really XCP-ng or any hypervisor specific stuff.

olivierlambert

Adding @yann and @TeddyAstie in the convo (since we maybe need to report both, IDK)

TeddyAstie

It could be interesting to report all the information the guest can provide us, but there is no xenstore schema for that; and no support from xcp-rrdd either.

Aside that, I don't think xe-guest-utilities consider cache memory as used; but there may be corner cases.
What gives cat /proc/meminfo

jebrown

@planedrop I'm using "citrix" tools for windows and I haven't noticed any of mine reporting any cached ram. Yet
cached ram.png

planedrop

@jebrown OK good to know. This may be different on different OSes though, Windows may just report actually used RAM and leave out anything that is cached.

fred974

Thank you all for your feedback on this matter. The OS used in this case ins Ubuntu 22.04.5 LTS.

@TeddyAstie said in Memory reporting incorrect values:

What gives cat /proc/meminfo

cat /proc/meminfo
MemTotal:        8123972 kB
MemFree:          440556 kB
MemAvailable:    5419780 kB
Buffers:          410852 kB
Cached:          4761852 kB
SwapCached:          328 kB
Active:          2995088 kB
Inactive:        3405912 kB
Active(anon):     723800 kB
Inactive(anon):  1399348 kB
Active(file):    2271288 kB
Inactive(file):  2006564 kB
Unevictable:       27620 kB
Mlocked:           27620 kB
SwapTotal:       4194300 kB
SwapFree:        4191228 kB
Dirty:              1508 kB
Writeback:             0 kB
AnonPages:       1255676 kB
Mapped:           939728 kB
Shmem:            885788 kB
KReclaimable:     995468 kB
Slab:            1154076 kB
SReclaimable:     995468 kB
SUnreclaim:       158608 kB
KernelStack:        4656 kB
PageTables:        13848 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8256284 kB
Committed_AS:    5873420 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       18896 kB
VmallocChunk:          0 kB
Percpu:             4192 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      307200 kB
DirectMap2M:     8077312 kB
DirectMap1G:     1048576 kB

yann

@fred974 the tools should be reporting the value from MemAvailable, so buffer/cache should not be considered as in-use. What do you see (reported by the agent) in xenstore ?

From within the guest:

# xenstore ls data
meminfo_free = "324060"
meminfo_total = "973736"
os_distro = "Debian"
os_name = "Debian 11"
os_uname = "5.10.0-21-amd64"
updated = "1"

The meminfo_free should match what you see in /proc/meminfo.

fred974

@yann said in Memory reporting incorrect values:

xenstore ls data

Here is what I see:
cat /proc/meminfo

MemTotal:        8123972 kB
MemFree:          428388 kB
MemAvailable:    5419696 kB
Buffers:          411604 kB
Cached:          4771524 kB
SwapCached:          328 kB
Active:          3060024 kB
Inactive:        3339116 kB
Active(anon):     736300 kB
Inactive(anon):  1374628 kB
Active(file):    2323724 kB
Inactive(file):  1964488 kB
Unevictable:       27620 kB
Mlocked:           27620 kB
SwapTotal:       4194300 kB
SwapFree:        4191228 kB
Dirty:               284 kB
Writeback:             0 kB
AnonPages:       1243360 kB
Mapped:           919268 kB
Shmem:            885852 kB
KReclaimable:     997192 kB
Slab:            1155308 kB
SReclaimable:     997192 kB
SUnreclaim:       158116 kB
KernelStack:        4432 kB
PageTables:        13128 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     8256284 kB
Committed_AS:    5854320 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       18656 kB
VmallocChunk:          0 kB
Percpu:             4192 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:      307200 kB
DirectMap2M:     8077312 kB
DirectMap1G:     1048576 kB

xenstore ls data

meminfo_free = "422892"
meminfo_total = "8123972"
os_distro = "ubuntu"
os_majorver = "22"
os_minorver = "04"
os_name = "Ubuntu 22.04.5 LTS"
os_uname = "5.15.0-128-generic"
updated = "Mon Jan  6 11:52:45 2025"
volumes = ""
 0 = ""
  extents = ""
   0 = "xvda"
  filesystem = "ext4"
  free = "142059319296"
  mount_points = ""
   0 = "/"
  name = "/dev/xvda1(072768fc-9db5-4fa4-a6a1-c98afdd82d32)"
  size = "171682283008"
 1 = ""
  extents = ""
   0 = "xvda"
  free = "4138934272"
  name = "/dev/xvda14"
  size = "4194304"
 2 = ""
  extents = ""
   0 = "xvda"
  filesystem = "vfat"
  free = "103057408"
  mount_points = ""
   0 = "/boot/efi"
  name = "/dev/xvda15(D9A6-D2AA)"
  size = "111149056"
 3 = ""
  extents = ""
   0 = "xvdb"
  free = "4138934272"
  name = "/dev/xvdb1(355A-4FC2)"
  size = "10486272"

htop

XOA

XO Community

fred974

It would be great to have a bit for details on the memory so we can monitor the system better.

fred974

@yann said in Memory reporting incorrect values:

@fred974 the tools should be reporting the value from MemAvailable, so buffer/cache should not be considered as in-use.

Doesn't seem to be the case

yann

@fred974 right, it is reporting MemFree here.

What does xenstore ls attr/PVAddons show for this VM? And which kernel version is it using?

Old xe-guest-utilities before v7.30 indeed reported MemFree, which XenServer then improved in v8.4 to still report something for kernels older than 3.14 which did not have MemAvailable yet, but Ubuntu 22.04 ships a decently-recent kernel, so I suspect it would be rather due to an old xe-guest-utilities.

fred974

@yann said in Memory reporting incorrect values:

What does xenstore ls attr/PVAddons show for this VM? And which kernel version is it using?

BuildVersion = "0"
Installed = "1"
MajorVersion = "6"
MicroVersion = "80"
MinorVersion = "6"

hostnamectl status

   Static hostname: WEB1-OLS-QHUK
Transient hostname: web1-ols-qh-uk
         Icon name: computer-vm
           Chassis: vm
        Machine ID: 3b1da9b8e5cf4ecda845219bxxx
           Boot ID: 19308dbfd7d347c699985e82xxx
    Virtualization: xen
  Operating System: Ubuntu 22.04.5 LTS              
            Kernel: Linux 5.15.0-128-generic
      Architecture: x86-64
   Hardware Vendor: Xen
    Hardware Model: HVM domU

yann

@fred974 v6.80.6 is definitely very old, and did report MemFree, we ship v7.30 with XCP-ng 8.2. You will likely want to update it, from the "XCP-ng tools" SR.

fred974

@yann said in Memory reporting incorrect values:

@fred974 v6.80.6 is definitely very old, and did report MemFree, we ship v7.30 with XCP-ng 8.2. You will likely want to update it, from the "XCP-ng tools" SR.

Probably because I installed xe-guest-utilities from ububtu repo.. I tried to update from the CD and got the following error:
bash /mnt/Linux/install.sh

Detected `Ubuntu 22.04.5 LTS' (ubuntu version 22).

The following changes will be made to this Virtual Machine:
  * update arp_notify sysctl.
  * packages to be installed/upgraded:
    - xe-guest-utilities_7.30.0-12_amd64.deb

Continue? [y/n] y

(Reading database ... 136958 files and directories currently installed.)
Preparing to unpack .../xe-guest-utilities_7.30.0-12_amd64.deb ...
Unpacking xe-guest-utilities (7.30.0-12) over (7.20.2-0ubuntu1~22.04.2) ...
dpkg: error processing archive /mnt/Linux/xe-guest-utilities_7.30.0-12_amd64.deb (--install):
 trying to overwrite '/usr/bin/xenstore', which is also in package xenstore-utils 4.16.0-1~ubuntu2.1
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /mnt/Linux/xe-guest-utilities_7.30.0-12_amd64.deb

Could anyone advice of how to fix this issue? I tried apt remove xe-guest-utilities and got the same error

yann

@fred974 there is an undeclared conflict between the xe-guest-utilities in the ISO and xenstore-utils, you can install the former after removing the latter (but yes, it is a problem, the Go implementation of xenstore tools should not be in this package).