Reverse Proxy Configuration - HAProxy

john.c

@delaf @olivierlambert Maybe worth investigating this to get this corrected so it will log the right IP address in the audit log.

delaf

@olivierlambert any idea on what can I do to help to debug this issue?

Step to reproduce the issue :

Install xo and make it listening on 127.0.0.1:8080
Install HAProxy 2.8 on Debian 12 (check https://haproxy.debian.net/) :

# curl https://haproxy.debian.net/bernat.debian.org.gpg | gpg --dearmor > /usr/share/keyrings/haproxy.debian.net.gpg
# echo "deb [signed-by=/usr/share/keyrings/haproxy.debian.net.gpg] http://haproxy.debian.net bookworm-backports-2.8 main" > /etc/apt/sources.list.d/haproxy.list
# apt-get update
# apt-get install haproxy=2.8.\*
# systemctl stop haproxy

Configure HAProxy
Very small HAProxy config (/etc/haproxy/haproxy.cfg), update the bind line to listen on the IP:PORT you want. XO is listening on 127.0.0.1:8080.

global
  log /dev/log local0
  log /dev/log local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s

  user haproxy
  group haproxy
  daemon

defaults
  mode http
  log global
  log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Td=%Tt %ST %U %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[last_rule_file]:%[last_rule_line] %ID"
  error-log-format "%ci:%cp [%tr] %ft %ac/%fc %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc %{+Q}[fc_err_str]"
  option dontlognull
  option redispatch
  timeout connect 5s
  timeout client 50s
  timeout server 50s
  unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
  unique-id-header X-Unique-ID

frontend ft_public
  bind IP:PORT
  mode http
  default_backend bk_xo

backend bk_xo
  mode http
  option forwarded
  option forwardfor
  option http-server-close
  http-request add-header HAProxy yes
  server xo 127.0.0.1:8080 check

Start HAProxy: systemctl start haproxy
Test it http://IP:PORT
Check on the network the headers sent by HAProxy to XO tcpdump -ni lo port 8080

olivierlambert

What's happening on step 5 and what's an example of sent stuff on 6?

delaf

@olivierlambert

step 5, you get access to XO.
step 6 (I did a curl on http://X.X.X.X:PORT/signin)

Request sent by HAProxy:

09:34:03.294200 IP 127.0.0.1.34134 > 127.0.0.1.8080: Flags [P.], seq 1:266, ack 1, win 512, options [nop,nop,TS val 2919287515 ecr 2919287515], length 265: HTTP: GET /signin HTTP/1.1
E..=^.@.@............V.PR,.5.........1.....
........GET /signin HTTP/1.1
host: xoau.ivy1.aquaray.com:8080
user-agent: curl/7.74.0
accept: */*
haproxy: yes
x-unique-id: AC1014F0:BCD6_AC1014F0:1F90_6576C97B_000A:1783DC
forwarded: proto=http;for=X.X.X.X
x-forwarded-for: X.XX.X
connection: close

Response from XO:

09:34:03.296545 IP 127.0.0.1.8080 > 127.0.0.1.34134: Flags [P.], seq 1:2009, ack 266, win 512, options [nop,nop,TS val 2919287517 ecr 2919287515], length 2008: HTTP: HTTP/1.1 200 OK
E...t.@.@..`.........P.V....R,.>...........
........HTTP/1.1 200 OK
X-DNS-Prefetch-Control: off
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: text/html; charset=utf-8
Content-Length: 1464
ETag: W/"5b8-OqUsZViW2KwDMOq1IfmEYkCzkN0"
Set-Cookie: connect.sid=s%3A3imvxT96Uq9L224R8NpArAHuW5Ho7jOS.ArGj0Ms2cIXEalxqdYSk95JS7J0ihftv%2FcURH53p07A; Path=/; HttpOnly
Vary: Accept-Encoding
Date: Mon, 11 Dec 2023 08:34:03 GMT
Connection: close

olivierlambert

Does it ring a bell @julien-f ?

julien-f

@delaf Can you test the PR https://github.com/vatesfr/xen-orchestra/pull/7233?

You will need to set http.useForwardedHeaders to true in your xo-server's config.

Let me know if that helps

julien-f opened this pull request in vatesfr/xen-orchestra

closed feat(xo-server): http.useForwardedHeaders #7233

delaf

@julien-f it works

As it is said in the doc, relying on x-forwarded-* should be done carefuly in a controlled network.
@julien-f FYI, some software (like mod_remoteip for apache) use another variable to know from which proxy it is safe to use x-forwarded-* (https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html#remoteiptrustedproxy).

BWT thank you!

julien-f

@delaf We could add the support for an array of IP addresses for useForwardedHeaders if necessary in the future.

Do you think it's required to do this now?

julien-f

@delaf I've added the support for trusted addresses, if you can test it that would be great

delaf

@julien-f I have both case with a forwarding proxy on 127.0.0.1: useForwardedHeaders = ['127.0.0.1'] and useForwardedHeaders = ['10.0.0.0/8'] and all seems OK

julien-f

@delaf Great! Thank you so much!

delaf

@julien-f I think we should add this in the reverse proxy documentaion (https://xen-orchestra.com/docs/configuration.html#reverse-proxy). I have created https://github.com/vatesfr/xen-orchestra/pull/7289

gdelafond opened this pull request in vatesfr/xen-orchestra

closed docs(configuration): useForwardedHeaders #7289

julien-f

@delaf Thank you!