Here's an updated ixgbe driver package that is meant as an update candidate without updating to a higher version. If need all feedback we can get on this one, because this is the candidate for the official fix as would be delivered to everyone. Of course the intel-igb-alt will remain available for those who need a more recent driver, and I even moved it to the updates repository so that one doesn't need to add --enablerepo=xcp-ng-testing to install it anymore.

To test the official update candidate on XCP-ng 8.1 or 8.2.

• If you had previously installed intel-ixgbe-alt:
  • yum remove intel-ixgbe-alt -y
  • Check that the /lib/modules/4.19.0+1/override/ixgbe.ko file was properly deleted. I've seen, once, a situation where a .ko file from an -alt package wasn't deleted, so I'm being cautious here and ask you to report you see that it is still present. In theory, this is impossible, but as I've seen it once I don't know anymore
  • Run depmod -a
• Update the intel-ixgbe package from the testing repo: yum update intel-ixgbe --enablerepo=xcp-ng-testing
• Reboot

Note: I'm not 100% sure that I picked the right patch, nor that this patch alone is sufficient.

echo 3 > /proc/sys/vm/drop_caches

to release some of the cache again, without interfering with running processes.

[root@host2 ~]# free -m
total used free shared buff/cache available
Mem: 15958 3308 158 8 12491 2355
Swap: 1023 177 846
[root@host2 ~]# echo 3 > /proc/sys/vm/drop_caches
[root@host2 ~]# free -m
total used free shared buff/cache available
Mem: 15958 3308 2598 10 10051 2751
Swap: 1023 177 846

We made it. This is the end of this huge thread.

A big thank you to everyone involved in debugging the issue.

And this is not a :D.

$ cat get_network_drivers_info.sh
#!/bin/bash                                                                                                                                                                                                                                                                                                   

format="| %-13.13s | %-20.20s | %-20.20s | %-10.10s | %-7.7s | %-10.10s | %-30.30s | %-s \n"
printf "${format}" "date" "hostname" "OS" "interface" "driver" "version" "firmware" "yum"
printf "${format}" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------"

if [ $# -gt 0 ]; then
    servers=($(echo ${BASH_ARGV[*]}))
else
    servers=($(cat host.json | jq -r '.[] | .address' | egrep -v "^192.168.124.9$"))
fi

for line in ${servers[@]}; do
    scp get_network_drivers_info.sh.tpl ${line}:/tmp/get_network_drivers_info.sh  > /dev/null 2>&1;
    ssh -n ${line} bash /tmp/get_network_drivers_info.sh 2> /dev/null;
    if [ $? -ne 0 ]; then
        echo "${line} fail" >&2
    fi
done

$ cat get_network_drivers_info.sh.tpl
#!/bin/bash                                                                                                                                                                                                                                                                                                   

format="| %-13.13s | %-20.20s | %-20.20s | %-10.10s | %-7.7s | %-10.10s | %-30.30s | %-s \n"
d=$(date '+%Y%m%d-%H%M')
name=$(hostname)
cd  /sys/class/net/
for interface in $(ls -l /sys/class/net/ | awk '/\/pci/ {print $9}'); do
    version=$(ethtool -i ${interface} | awk '/^version:/ {$1=""; print}')
    firmware=$(ethtool -i ${interface} | awk '/^firmware-version:/ {$1=""; print}')
    driver=$(ethtool -i ${interface} | awk '/^driver:/ {$1=""; print}')
    YUM=$(which yum)
    if [ $? -eq 0 ]; then
        packages=$(yum list installed | awk '/ixgbe/ {print $1"@"$2}' | tr '\n' ',')
    else
        packages="NA"
    fi
    os_version=$(lsb_release -d | awk '{$1=""} 1' | sed 's/XenServer/XS/; s/ (xenenterprise)//; s/release //')
    printf "${format}" "${d}" "${name}" "${os_version}" "${interface}" "${driver}" "${version}" "${firmware}" "${packages}"
done

PS: host.json file is generated via : xo-cli --list-objects type=host

@olivierlambert you can change the retention method and keep much more data on netdata. There is also (since netdata 1.18 i think) a dbengine that allows you to store data on disk.

If you want longer metrics, you need to send the data in Prometheus/Grafana.

Does this fix has been released or is to be released ?

Probably plenty of Citrix customers were affected, but they would rather reboot on schedule than spend months working through the support process

What I still find really weird is the fact we had report of the issue far longer before Citrix. And we had roughly 10 people affected while Citrix got only 1 report

Official Citrix update has been posted: https://support.citrix.com/article/CTX306529

edit: thanks @fasterfourier for your feedback!

/opt/xensource/libexec/xen-cmdline --set-dom0 page_owner=on

They then confirmed the memory leak was from the NIC driver. They are intending to release a public hotfix for this issue.

Here's an updated ixgbe driver package that is meant as an update candidate without updating to a higher version. If need all feedback we can get on this one, because this is the candidate for the official fix as would be delivered to everyone. Of course the intel-igb-alt will remain available for those who need a more recent driver, and I even moved it to the updates repository so that one doesn't need to add --enablerepo=xcp-ng-testing to install it anymore.

To test the official update candidate on XCP-ng 8.1 or 8.2.

• If you had previously installed intel-ixgbe-alt:
  • yum remove intel-ixgbe-alt -y
  • Check that the /lib/modules/4.19.0+1/override/ixgbe.ko file was properly deleted. I've seen, once, a situation where a .ko file from an -alt package wasn't deleted, so I'm being cautious here and ask you to report you see that it is still present. In theory, this is impossible, but as I've seen it once I don't know anymore
  • Run depmod -a
• Update the intel-ixgbe package from the testing repo: yum update intel-ixgbe --enablerepo=xcp-ng-testing
• Reboot

Note: I'm not 100% sure that I picked the right patch, nor that this patch alone is sufficient.