Reverse Proxy Configuration - HAProxy

john.c

@delaf said in Reverse Proxy Configuration - HAProxy:

@john-c Could you explain what has changed in 2.8 that fix a forward-for problem? I did not see any related changes in 2.8 changelog.
There is nothing parlicular to do to in HAProxy to support websocket.

Well in the case of forward-for it is fully implementing RFC7239 ("forwarded") which maintains information which may be altered or lost when proxies are involved. This implementation is for both processing and generation. See this article for more information on the involvement of forwarded headers (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded), then this article is for how it affects the HAProxy specifically (https://www.haproxy.com/blog/announcing-haproxy-2-8).

Actually there's certain parameters which are required to enable WebSocket support in HAProxy unless the defaults have changed since the articles. There's several items which are required in the backend section of the HAProxy configuration file to enable Websocket support:-

backend http_back
option forward
option forwardfor
option http-server-close

Without these and having HAProxy use the http backend by default for the connection it will have difficulty with the WebSocket functionality.

delaf

@john-c said in Reverse Proxy Configuration - HAProxy:

Well in the case of forward-for it is fully implementing RFC7239 ("forwarded") which maintains information which may be altered or lost when proxies are involved. This implementation is for both processing and generation. See this article for more information on the involvement of forwarded headers (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded), then this article is for how it affects the HAProxy specifically (https://www.haproxy.com/blog/announcing-haproxy-2-8).

Oh now, I remember I saw that when the 2.8 was released. Thank you for the link!

BTW, even with option forwarded (and not option forward) I did not get the real IP in XO logs. Do you have any idea what I should do to debug this on the XO side?
PS: in a tcpdump trace, I see the rights headers sent to XO: forwarded: proto=https;for=X.X.X.X and my old x-forwarded-for: X.X.X.X.

Gheppy

With HAProxy to get real IP on backend server you need to

Add to backend server
:443 send-proxy
Enable Proxy Protocol on XO (if is supported):
RemoteIPProxyProtocol On
RemoteIPHeader X-Forwarded-For
RemoteIPProxyProtocolExceptions 127.0.0.1

Examples are from HAProxy and Apache

olivierlambert

If you want to share a working config, feel free, we can add it to our XO doc

Gheppy

I don't use HAProxy with XO, but I'll do the test and I'll came with more info

john.c

@delaf Some configuration is required on the XO side involving the backend port to 443.

Gheppy

@olivierlambert
Let say that you have this:

HAProxy server :
- External IP 10.1.60.130
- Internal IP 172.20.100.1
XO server
- IP 172.20.100.2
for HAProxy to work it is needed to do this

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2
    #log         /dev/log local2 debug
    #log         /dev/log local2 notice
    #log         127.0.0.1 local2 info
    #log         127.0.0.1 local2 notice

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    #maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
    stats timeout 30s

    # utilize system-wide crypto-policies
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------

defaults
    mode                    http
    log                     global
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    #maxconn                 3000
    #errorfile 400 /opt/haproxy/share/err/400.http
    #errorfile 403 /opt/haproxy/share/err/403.http
    #errorfile 408 /opt/haproxy/share/err/408.http
    #errorfile 500 /opt/haproxy/share/err/500.http
    #errorfile 502 /opt/haproxy/share/err/502.http
    #errorfile 503 /opt/haproxy/share/err/503.http
    #errorfile 504 /opt/haproxy/share/err/504.http



#---------------------------------------------------------------------
# Port 443
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# frontend
#---------------------------------------------------------------------

frontend tb2_443
    bind 10.1.60.130:443
    mode tcp
    option tcplog
    acl tls req.ssl_hello_type 1
    tcp-request inspect-delay 5s
    tcp-request content accept if tls
    default_backend xoce
    
#---------------------------------------------------------------------
# backend
#---------------------------------------------------------------------	
backend xoce
    mode tcp
    option ssl-hello-chk
    server xoce_srv         172.20.100.2:443 send-proxy check

but ...

It seams that XO don't support Proxy Protocol

Secure Connection Failed

An error occurred during a connection to 10.1.60.130. PR_END_OF_FILE_ERROR

Error code: PR_END_OF_FILE_ERROR

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.

In XO code it must be included proxy-protocol ( npm install proxy-protocol ), to have this possibility.
This is the code:

const http = require('http');
const ProxyProtocol = require('proxy-protocol');

const server = http.createServer((req, res) => {
    // Parse Proxy Protocol headers
    const proxy = new ProxyProtocol();
    const proxyData = proxy.parse(req);

    // Extract client IP and Port from the parsed headers
    const clientIP = proxyData && proxyData.address ? proxyData.address : req.connection.remoteAddress;
    const clientPort = proxyData && proxyData.port ? proxyData.port : req.connection.remotePort;

    // Handle the request
    // ...
});

server.listen(3000);

olivierlambert

Weird, it shouldn't require anything baked into XO to support a reverse proxy Any comment/idea @julien-f ?

edit: it shouldn't be needed since HTTP protocol supports the "Forwarded" directive (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded)

Gheppy

@olivierlambert
It is HTTPS ( not HTTP ) and proxy-protocol-v2, and as far as I know the "Forwarded" directive is not supported on HTTPS, only on HTTP

delaf

@Gheppy The RFC says that Forwarded must (when active) be set on HTTP requets but not HTTP response. It use HTTP for HTTP and HTTPS in this context, that is confirmed by :

   Information passed in this header field can be, for example, the source IP
   address of the request, the IP address of the incoming interface on
   the proxy, or whether HTTP or HTTPS was used

The syntax describe by mozilla (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) :

Syntax: Forwarded: by=<identifier>;for=<identifier>;host=<host>;proto=<http|https>

Gheppy

@delaf
Forwarded it seams that is supported with HAProxy 2.8, I use 2.6. I'll install 2.8 from source to see if is ok

delaf

@Gheppy ok, here even with the Forwarded header (set by an HAProxy 2.8), XO does not log in right IP (in the audit log).

john.c

@delaf @olivierlambert Maybe worth investigating this to get this corrected so it will log the right IP address in the audit log.

delaf

@olivierlambert any idea on what can I do to help to debug this issue?

Step to reproduce the issue :

Install xo and make it listening on 127.0.0.1:8080
Install HAProxy 2.8 on Debian 12 (check https://haproxy.debian.net/) :

# curl https://haproxy.debian.net/bernat.debian.org.gpg | gpg --dearmor > /usr/share/keyrings/haproxy.debian.net.gpg
# echo "deb [signed-by=/usr/share/keyrings/haproxy.debian.net.gpg] http://haproxy.debian.net bookworm-backports-2.8 main" > /etc/apt/sources.list.d/haproxy.list
# apt-get update
# apt-get install haproxy=2.8.\*
# systemctl stop haproxy

Configure HAProxy
Very small HAProxy config (/etc/haproxy/haproxy.cfg), update the bind line to listen on the IP:PORT you want. XO is listening on 127.0.0.1:8080.

global
  log /dev/log local0
  log /dev/log local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s

  user haproxy
  group haproxy
  daemon

defaults
  mode http
  log global
  log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Td=%Tt %ST %U %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[last_rule_file]:%[last_rule_line] %ID"
  error-log-format "%ci:%cp [%tr] %ft %ac/%fc %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc %{+Q}[fc_err_str]"
  option dontlognull
  option redispatch
  timeout connect 5s
  timeout client 50s
  timeout server 50s
  unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
  unique-id-header X-Unique-ID

frontend ft_public
  bind IP:PORT
  mode http
  default_backend bk_xo

backend bk_xo
  mode http
  option forwarded
  option forwardfor
  option http-server-close
  http-request add-header HAProxy yes
  server xo 127.0.0.1:8080 check

Start HAProxy: systemctl start haproxy
Test it http://IP:PORT
Check on the network the headers sent by HAProxy to XO tcpdump -ni lo port 8080

olivierlambert

What's happening on step 5 and what's an example of sent stuff on 6?

delaf

@olivierlambert

step 5, you get access to XO.
step 6 (I did a curl on http://X.X.X.X:PORT/signin)

Request sent by HAProxy:

09:34:03.294200 IP 127.0.0.1.34134 > 127.0.0.1.8080: Flags [P.], seq 1:266, ack 1, win 512, options [nop,nop,TS val 2919287515 ecr 2919287515], length 265: HTTP: GET /signin HTTP/1.1
E..=^.@.@............V.PR,.5.........1.....
........GET /signin HTTP/1.1
host: xoau.ivy1.aquaray.com:8080
user-agent: curl/7.74.0
accept: */*
haproxy: yes
x-unique-id: AC1014F0:BCD6_AC1014F0:1F90_6576C97B_000A:1783DC
forwarded: proto=http;for=X.X.X.X
x-forwarded-for: X.XX.X
connection: close

Response from XO:

09:34:03.296545 IP 127.0.0.1.8080 > 127.0.0.1.34134: Flags [P.], seq 1:2009, ack 266, win 512, options [nop,nop,TS val 2919287517 ecr 2919287515], length 2008: HTTP: HTTP/1.1 200 OK
E...t.@.@..`.........P.V....R,.>...........
........HTTP/1.1 200 OK
X-DNS-Prefetch-Control: off
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: text/html; charset=utf-8
Content-Length: 1464
ETag: W/"5b8-OqUsZViW2KwDMOq1IfmEYkCzkN0"
Set-Cookie: connect.sid=s%3A3imvxT96Uq9L224R8NpArAHuW5Ho7jOS.ArGj0Ms2cIXEalxqdYSk95JS7J0ihftv%2FcURH53p07A; Path=/; HttpOnly
Vary: Accept-Encoding
Date: Mon, 11 Dec 2023 08:34:03 GMT
Connection: close

olivierlambert

Does it ring a bell @julien-f ?

julien-f

@delaf Can you test the PR https://github.com/vatesfr/xen-orchestra/pull/7233?

You will need to set http.useForwardedHeaders to true in your xo-server's config.

Let me know if that helps

julien-f opened this pull request in vatesfr/xen-orchestra

closed feat(xo-server): http.useForwardedHeaders #7233

delaf

@julien-f it works

As it is said in the doc, relying on x-forwarded-* should be done carefuly in a controlled network.
@julien-f FYI, some software (like mod_remoteip for apache) use another variable to know from which proxy it is safe to use x-forwarded-* (https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html#remoteiptrustedproxy).

BWT thank you!

julien-f

@delaf We could add the support for an array of IP addresses for useForwardedHeaders if necessary in the future.

Do you think it's required to do this now?