Reverse Proxy Configuration - HAProxy

Gheppy

@olivierlambert
It is HTTPS ( not HTTP ) and proxy-protocol-v2, and as far as I know the "Forwarded" directive is not supported on HTTPS, only on HTTP

delaf

@Gheppy The RFC says that Forwarded must (when active) be set on HTTP requets but not HTTP response. It use HTTP for HTTP and HTTPS in this context, that is confirmed by :

   Information passed in this header field can be, for example, the source IP
   address of the request, the IP address of the incoming interface on
   the proxy, or whether HTTP or HTTPS was used

The syntax describe by mozilla (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) :

Syntax: Forwarded: by=<identifier>;for=<identifier>;host=<host>;proto=<http|https>

Gheppy

@delaf
Forwarded it seams that is supported with HAProxy 2.8, I use 2.6. I'll install 2.8 from source to see if is ok

delaf

@Gheppy ok, here even with the Forwarded header (set by an HAProxy 2.8), XO does not log in right IP (in the audit log).

john.c

@delaf @olivierlambert Maybe worth investigating this to get this corrected so it will log the right IP address in the audit log.

delaf

@olivierlambert any idea on what can I do to help to debug this issue?

Step to reproduce the issue :

Install xo and make it listening on 127.0.0.1:8080
Install HAProxy 2.8 on Debian 12 (check https://haproxy.debian.net/) :

# curl https://haproxy.debian.net/bernat.debian.org.gpg | gpg --dearmor > /usr/share/keyrings/haproxy.debian.net.gpg
# echo "deb [signed-by=/usr/share/keyrings/haproxy.debian.net.gpg] http://haproxy.debian.net bookworm-backports-2.8 main" > /etc/apt/sources.list.d/haproxy.list
# apt-get update
# apt-get install haproxy=2.8.\*
# systemctl stop haproxy

Configure HAProxy
Very small HAProxy config (/etc/haproxy/haproxy.cfg), update the bind line to listen on the IP:PORT you want. XO is listening on 127.0.0.1:8080.

global
  log /dev/log local0
  log /dev/log local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s

  user haproxy
  group haproxy
  daemon

defaults
  mode http
  log global
  log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Td=%Tt %ST %U %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[last_rule_file]:%[last_rule_line] %ID"
  error-log-format "%ci:%cp [%tr] %ft %ac/%fc %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc %{+Q}[fc_err_str]"
  option dontlognull
  option redispatch
  timeout connect 5s
  timeout client 50s
  timeout server 50s
  unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
  unique-id-header X-Unique-ID

frontend ft_public
  bind IP:PORT
  mode http
  default_backend bk_xo

backend bk_xo
  mode http
  option forwarded
  option forwardfor
  option http-server-close
  http-request add-header HAProxy yes
  server xo 127.0.0.1:8080 check

Start HAProxy: systemctl start haproxy
Test it http://IP:PORT
Check on the network the headers sent by HAProxy to XO tcpdump -ni lo port 8080

olivierlambert

What's happening on step 5 and what's an example of sent stuff on 6?

delaf

@olivierlambert

step 5, you get access to XO.
step 6 (I did a curl on http://X.X.X.X:PORT/signin)

Request sent by HAProxy:

09:34:03.294200 IP 127.0.0.1.34134 > 127.0.0.1.8080: Flags [P.], seq 1:266, ack 1, win 512, options [nop,nop,TS val 2919287515 ecr 2919287515], length 265: HTTP: GET /signin HTTP/1.1
E..=^.@.@............V.PR,.5.........1.....
........GET /signin HTTP/1.1
host: xoau.ivy1.aquaray.com:8080
user-agent: curl/7.74.0
accept: */*
haproxy: yes
x-unique-id: AC1014F0:BCD6_AC1014F0:1F90_6576C97B_000A:1783DC
forwarded: proto=http;for=X.X.X.X
x-forwarded-for: X.XX.X
connection: close

Response from XO:

09:34:03.296545 IP 127.0.0.1.8080 > 127.0.0.1.34134: Flags [P.], seq 1:2009, ack 266, win 512, options [nop,nop,TS val 2919287517 ecr 2919287515], length 2008: HTTP: HTTP/1.1 200 OK
E...t.@.@..`.........P.V....R,.>...........
........HTTP/1.1 200 OK
X-DNS-Prefetch-Control: off
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: text/html; charset=utf-8
Content-Length: 1464
ETag: W/"5b8-OqUsZViW2KwDMOq1IfmEYkCzkN0"
Set-Cookie: connect.sid=s%3A3imvxT96Uq9L224R8NpArAHuW5Ho7jOS.ArGj0Ms2cIXEalxqdYSk95JS7J0ihftv%2FcURH53p07A; Path=/; HttpOnly
Vary: Accept-Encoding
Date: Mon, 11 Dec 2023 08:34:03 GMT
Connection: close

olivierlambert

Does it ring a bell @julien-f ?

julien-f

@delaf Can you test the PR https://github.com/vatesfr/xen-orchestra/pull/7233?

You will need to set http.useForwardedHeaders to true in your xo-server's config.

Let me know if that helps

julien-f opened this pull request in vatesfr/xen-orchestra

closed feat(xo-server): http.useForwardedHeaders #7233

delaf

@julien-f it works

As it is said in the doc, relying on x-forwarded-* should be done carefuly in a controlled network.
@julien-f FYI, some software (like mod_remoteip for apache) use another variable to know from which proxy it is safe to use x-forwarded-* (https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html#remoteiptrustedproxy).

BWT thank you!

julien-f

@delaf We could add the support for an array of IP addresses for useForwardedHeaders if necessary in the future.

Do you think it's required to do this now?

julien-f

@delaf I've added the support for trusted addresses, if you can test it that would be great

delaf

@julien-f I have both case with a forwarding proxy on 127.0.0.1: useForwardedHeaders = ['127.0.0.1'] and useForwardedHeaders = ['10.0.0.0/8'] and all seems OK

julien-f

@delaf Great! Thank you so much!

delaf

@julien-f I think we should add this in the reverse proxy documentaion (https://xen-orchestra.com/docs/configuration.html#reverse-proxy). I have created https://github.com/vatesfr/xen-orchestra/pull/7289

gdelafond opened this pull request in vatesfr/xen-orchestra

closed docs(configuration): useForwardedHeaders #7289

julien-f

@delaf Thank you!