@stormi
It seems to be good here!

@olivierlambert when changes are made (like IP change in the VM), it is now instantly reported by XO
I will be able to go back to my work to integrate a BGP deamon to announce locals IPs to the network (I started last year to work on the xe-deamon but never had the time to finish it, and some months after I saw you blog post about moving to rust for it).

Thank you!

PS: i'm using these 2 scripts to list all interfaces drivers version accross our servers :

$ cat get_network_drivers_info.sh
#!/bin/bash                                                                                                                                                                                                                                                                                                   

format="| %-13.13s | %-20.20s | %-20.20s | %-10.10s | %-7.7s | %-10.10s | %-30.30s | %-s \n"
printf "${format}" "date" "hostname" "OS" "interface" "driver" "version" "firmware" "yum"
printf "${format}" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------"

if [ $# -gt 0 ]; then
    servers=($(echo ${BASH_ARGV[*]}))
else
    servers=($(cat host.json | jq -r '.[] | .address' | egrep -v "^192.168.124.9$"))
fi

for line in ${servers[@]}; do
    scp get_network_drivers_info.sh.tpl ${line}:/tmp/get_network_drivers_info.sh  > /dev/null 2>&1;
    ssh -n ${line} bash /tmp/get_network_drivers_info.sh 2> /dev/null;
    if [ $? -ne 0 ]; then
        echo "${line} fail" >&2
    fi
done

$ cat get_network_drivers_info.sh.tpl
#!/bin/bash                                                                                                                                                                                                                                                                                                   

format="| %-13.13s | %-20.20s | %-20.20s | %-10.10s | %-7.7s | %-10.10s | %-30.30s | %-s \n"
d=$(date '+%Y%m%d-%H%M')
name=$(hostname)
cd  /sys/class/net/
for interface in $(ls -l /sys/class/net/ | awk '/\/pci/ {print $9}'); do
    version=$(ethtool -i ${interface} | awk '/^version:/ {$1=""; print}')
    firmware=$(ethtool -i ${interface} | awk '/^firmware-version:/ {$1=""; print}')
    driver=$(ethtool -i ${interface} | awk '/^driver:/ {$1=""; print}')
    YUM=$(which yum)
    if [ $? -eq 0 ]; then
        packages=$(yum list installed | awk '/ixgbe/ {print $1"@"$2}' | tr '\n' ',')
    else
        packages="NA"
    fi
    os_version=$(lsb_release -d | awk '{$1=""} 1' | sed 's/XenServer/XS/; s/ (xenenterprise)//; s/release //')
    printf "${format}" "${d}" "${name}" "${os_version}" "${interface}" "${driver}" "${version}" "${firmware}" "${packages}"
done

PS: host.json file is generated via : xo-cli --list-objects type=host

@stormi Hello, some week after, I can confirm that the problem is solved here by using intel-ixgbe.x86_64@5.5.2-2.1.xcpng8.1 or intel-ixgbe.x86_64@5.5.2-2.1.xcpng8.2

@stormi I have installed intel-ixgbe 5.5.2-2.1.xcpng8.2 on my server s0267. Let's wait a some days to check if the memleak is solved by this patch.

@stormi

The 2 servers have been reinstalled with an up to date 8.2. They host each 2 VMs that are doing the same thing (~100Mb/s of netdata stream).

The right one has the 5.9.4-1.xcpng8.2, the left one has 5.5.2-2.xcpng8.2.

The patch seem to be OK for me.

@stormi

• server 266 with alt-kernel: still no problem.
  

• server 268 with 4.19.19-6.0.10.1.xcpng8.1: the problem has begun some days ago after some stable days.
  

• server 272 with 4.19.19-6.0.11.1.0.1.patch53disabled.xcpng8.1:
  )

• server 273 with 4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1:
  

It seems that 4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1 is more stable than 4.19.19-6.0.11.1.0.1.patch53disabled.xcpng8.1. But it is a but early to be sure.

@stormi For the kernel-4.19.19-6.0.10.1.xcpng8.1 test, i'm not sure it solve the problem because I get a small memory increase. We have to wait a bit more

@stormi I have installed the two kernels

272 ~]# yum list installed kernel | grep kernel
kernel.x86_64                   4.19.19-6.0.11.1.0.1.patch53disabled.xcpng8.1

273 ~]# yum list installed kernel | grep kernel
kernel.x86_64                   4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1

I have removed the modification in /etc/modprobe.d/dist.conf on server 273.

We have to wait a little bit now

@stormi @r1
Four days later, I get:

• one server (266) with alt-kernel: still no problem
• one server (268) with 4.19.19-6.0.10.1.xcpng8.1: no more problem!
• one server (272) with kmemleak kernel: no memleak detected, but the problem is present
• one server (273) with search extra built-in weak-updates override updates: problem still present

@julien-f I think we should add this in the reverse proxy documentaion (https://xen-orchestra.com/docs/configuration.html#reverse-proxy). I have created https://github.com/vatesfr/xen-orchestra/pull/7289

gdelafond opened this pull request in vatesfr/xen-orchestra

closed docs(configuration): useForwardedHeaders #7289

@julien-f I have both case with a forwarding proxy on 127.0.0.1: useForwardedHeaders = ['127.0.0.1'] and useForwardedHeaders = ['10.0.0.0/8'] and all seems OK

@julien-f it works

As it is said in the doc, relying on x-forwarded-* should be done carefuly in a controlled network.
@julien-f FYI, some software (like mod_remoteip for apache) use another variable to know from which proxy it is safe to use x-forwarded-* (https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html#remoteiptrustedproxy).

BWT thank you!

@olivierlambert

• step 5, you get access to XO.
• step 6 (I did a curl on http://X.X.X.X:PORT/signin)

Request sent by HAProxy:

09:34:03.294200 IP 127.0.0.1.34134 > 127.0.0.1.8080: Flags [P.], seq 1:266, ack 1, win 512, options [nop,nop,TS val 2919287515 ecr 2919287515], length 265: HTTP: GET /signin HTTP/1.1
E..=^.@.@............V.PR,.5.........1.....
........GET /signin HTTP/1.1
host: xoau.ivy1.aquaray.com:8080
user-agent: curl/7.74.0
accept: */*
haproxy: yes
x-unique-id: AC1014F0:BCD6_AC1014F0:1F90_6576C97B_000A:1783DC
forwarded: proto=http;for=X.X.X.X
x-forwarded-for: X.XX.X
connection: close

Response from XO:

09:34:03.296545 IP 127.0.0.1.8080 > 127.0.0.1.34134: Flags [P.], seq 1:2009, ack 266, win 512, options [nop,nop,TS val 2919287517 ecr 2919287515], length 2008: HTTP: HTTP/1.1 200 OK
E...t.@.@..`.........P.V....R,.>...........
........HTTP/1.1 200 OK
X-DNS-Prefetch-Control: off
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: text/html; charset=utf-8
Content-Length: 1464
ETag: W/"5b8-OqUsZViW2KwDMOq1IfmEYkCzkN0"
Set-Cookie: connect.sid=s%3A3imvxT96Uq9L224R8NpArAHuW5Ho7jOS.ArGj0Ms2cIXEalxqdYSk95JS7J0ihftv%2FcURH53p07A; Path=/; HttpOnly
Vary: Accept-Encoding
Date: Mon, 11 Dec 2023 08:34:03 GMT
Connection: close

@olivierlambert any idea on what can I do to help to debug this issue?

Step to reproduce the issue :

1. Install xo and make it listening on 127.0.0.1:8080
2. Install HAProxy 2.8 on Debian 12 (check https://haproxy.debian.net/) :

# curl https://haproxy.debian.net/bernat.debian.org.gpg | gpg --dearmor > /usr/share/keyrings/haproxy.debian.net.gpg
# echo "deb [signed-by=/usr/share/keyrings/haproxy.debian.net.gpg] http://haproxy.debian.net bookworm-backports-2.8 main" > /etc/apt/sources.list.d/haproxy.list
# apt-get update
# apt-get install haproxy=2.8.\*
# systemctl stop haproxy

3. Configure HAProxy
   Very small HAProxy config (/etc/haproxy/haproxy.cfg), update the bind line to listen on the IP:PORT you want. XO is listening on 127.0.0.1:8080.

global
  log /dev/log local0
  log /dev/log local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s

  user haproxy
  group haproxy
  daemon

defaults
  mode http
  log global
  log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Td=%Tt %ST %U %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[last_rule_file]:%[last_rule_line] %ID"
  error-log-format "%ci:%cp [%tr] %ft %ac/%fc %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc %{+Q}[fc_err_str]"
  option dontlognull
  option redispatch
  timeout connect 5s
  timeout client 50s
  timeout server 50s
  unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
  unique-id-header X-Unique-ID

frontend ft_public
  bind IP:PORT
  mode http
  default_backend bk_xo

backend bk_xo
  mode http
  option forwarded
  option forwardfor
  option http-server-close
  http-request add-header HAProxy yes
  server xo 127.0.0.1:8080 check

4. Start HAProxy: systemctl start haproxy

5. Test it http://IP:PORT

6. Check on the network the headers sent by HAProxy to XO tcpdump -ni lo port 8080

@Gheppy ok, here even with the Forwarded header (set by an HAProxy 2.8), XO does not log in right IP (in the audit log).

@Gheppy The RFC says that Forwarded must (when active) be set on HTTP requets but not HTTP response. It use HTTP for HTTP and HTTPS in this context, that is confirmed by :

   Information passed in this header field can be, for example, the source IP
   address of the request, the IP address of the incoming interface on
   the proxy, or whether HTTP or HTTPS was used

The syntax describe by mozilla (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) :

Syntax: Forwarded: by=<identifier>;for=<identifier>;host=<host>;proto=<http|https>

@john-c said in Reverse Proxy Configuration - HAProxy:

Well in the case of forward-for it is fully implementing RFC7239 ("forwarded") which maintains information which may be altered or lost when proxies are involved. This implementation is for both processing and generation. See this article for more information on the involvement of forwarded headers (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded), then this article is for how it affects the HAProxy specifically (https://www.haproxy.com/blog/announcing-haproxy-2-8).

Oh now, I remember I saw that when the 2.8 was released. Thank you for the link!

BTW, even with option forwarded (and not option forward) I did not get the real IP in XO logs. Do you have any idea what I should do to debug this on the XO side?
PS: in a tcpdump trace, I see the rights headers sent to XO: forwarded: proto=https;for=X.X.X.X and my old x-forwarded-for: X.X.X.X.

@john-c Could you explain what has changed in 2.8 that fix a forward-for problem? I did not see any related changes in 2.8 changelog.
There is nothing parlicular to do to in HAProxy to support websocket.

@ tbigs2012 I think you should add option forwardfor to add the X-Forwarded-For` header.

@olivierlambert AFAIK There is no specific configuration for websockets to make HAProxy compatible with XO.

@olivierlambert I have a similar config, but it seems that XO does not take the content of the X-Forwarded-For header. I know that some software have a IP whitelist to know when they are allowed to look into x-forwarded-for for the client IP. Is there some config like this in XO?

PS: I do not have "login/logout" actions with the related IP in the audit log. I think it could be a nice addon