Reverse Proxy Configuration - HAProxy

tbigs2012

Hello,
Recently I have been trying to further secure my servers and while looking at Xen Orchestra I noticed my proxy servers IP is being captured instead of my host system's IP. I'd like to capture the real IP address. I did look through the documentation and I did find the reverse-proxy page but it doesn't include anything for HAProxy.
https://xen-orchestra.com/docs/configuration.html#reverse-proxy

Has anyone had success capturing the real IP address through HAProxy?

Right now I have HAProxy working in terms of being functional. I can use XO without an issue at all except the IP addresses in the log shows as my reverse proxy IP no matter who device makes changes.

I've tried adding some other configuration options shown below but it's still not seeing the real IP address.

http-request add-header X-Forwarded-Host %[req.hdr(Host)]
http-request add-header X-Forwarded-Proto https

This is what my backend looks like:

backend xoce.local_ipvANY
	mode			http
	id			102
	log			global
	timeout connect		30000
	timeout server		30000
	retries			3
	load-server-state-from-file	global
	http-request add-header X-Forwarded-Host %[req.hdr(Host)]
	http-request add-header X-Forwarded-Proto https
	acl			ip_whitelist	src 192.168.220.0/24 192.168.110.0/24 172.16.100.0/24
	acl			restrict_login	var(txn.txnhost) -m str -i xoce.assurecs.net
	http-request set-var(txn.txnhost) hdr(host)
	http-request deny deny_status 403  if  restrict_login !ip_whitelist 
	server			xoce.local 192.168.110.250:443 id 103 ssl check inter 1000  verify none crt /var/etc/haproxy/server_clientcert_64207b5c00cc3.pem

delaf

@julien-f I have both case with a forwarding proxy on 127.0.0.1: useForwardedHeaders = ['127.0.0.1'] and useForwardedHeaders = ['10.0.0.0/8'] and all seems OK

olivierlambert

Hi,

I do not know HA Proxy, but is there a specific config for web sockets?

delaf

@ tbigs2012 I think you should add option forwardfor to add the X-Forwarded-For` header.

@olivierlambert AFAIK There is no specific configuration for websockets to make HAProxy compatible with XO.

@olivierlambert I have a similar config, but it seems that XO does not take the content of the X-Forwarded-For header. I know that some software have a IP whitelist to know when they are allowed to look into x-forwarded-for for the client IP. Is there some config like this in XO?

PS: I do not have "login/logout" actions with the related IP in the audit log. I think it could be a nice addon

john.c

@delaf @olivierlambert @tbigs2012 Actually there is WebSocket support in HAProxy which could be used for XO compatibility.

In order to ensure that it functions properly when using forward-for configuration options, please download and install from the 2.8 branch of HAProxy, unless its features have been back ported by your Linux distribution. Also the QUIC feature has reached production readiness with the 2.8 branch. The 2.8 branch is an LTS branch.

Then follow the instructions from either of these articles adapting as appropriate (based on the parameters from the XO documentation) taking into account secure configuration principals.

delaf

@john-c Could you explain what has changed in 2.8 that fix a forward-for problem? I did not see any related changes in 2.8 changelog.
There is nothing parlicular to do to in HAProxy to support websocket.

john.c

@delaf said in Reverse Proxy Configuration - HAProxy:

@john-c Could you explain what has changed in 2.8 that fix a forward-for problem? I did not see any related changes in 2.8 changelog.
There is nothing parlicular to do to in HAProxy to support websocket.

Well in the case of forward-for it is fully implementing RFC7239 ("forwarded") which maintains information which may be altered or lost when proxies are involved. This implementation is for both processing and generation. See this article for more information on the involvement of forwarded headers (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded), then this article is for how it affects the HAProxy specifically (https://www.haproxy.com/blog/announcing-haproxy-2-8).

Actually there's certain parameters which are required to enable WebSocket support in HAProxy unless the defaults have changed since the articles. There's several items which are required in the backend section of the HAProxy configuration file to enable Websocket support:-

backend http_back
option forward
option forwardfor
option http-server-close

Without these and having HAProxy use the http backend by default for the connection it will have difficulty with the WebSocket functionality.

delaf

@john-c said in Reverse Proxy Configuration - HAProxy:

Well in the case of forward-for it is fully implementing RFC7239 ("forwarded") which maintains information which may be altered or lost when proxies are involved. This implementation is for both processing and generation. See this article for more information on the involvement of forwarded headers (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded), then this article is for how it affects the HAProxy specifically (https://www.haproxy.com/blog/announcing-haproxy-2-8).

Oh now, I remember I saw that when the 2.8 was released. Thank you for the link!

BTW, even with option forwarded (and not option forward) I did not get the real IP in XO logs. Do you have any idea what I should do to debug this on the XO side?
PS: in a tcpdump trace, I see the rights headers sent to XO: forwarded: proto=https;for=X.X.X.X and my old x-forwarded-for: X.X.X.X.

Gheppy

With HAProxy to get real IP on backend server you need to

Add to backend server
:443 send-proxy
Enable Proxy Protocol on XO (if is supported):
RemoteIPProxyProtocol On
RemoteIPHeader X-Forwarded-For
RemoteIPProxyProtocolExceptions 127.0.0.1

Examples are from HAProxy and Apache

olivierlambert

If you want to share a working config, feel free, we can add it to our XO doc

Gheppy

I don't use HAProxy with XO, but I'll do the test and I'll came with more info

john.c

@delaf Some configuration is required on the XO side involving the backend port to 443.

Gheppy

@olivierlambert
Let say that you have this:

HAProxy server :
- External IP 10.1.60.130
- Internal IP 172.20.100.1
XO server
- IP 172.20.100.2
for HAProxy to work it is needed to do this

#---------------------------------------------------------------------
# Global settings
#---------------------------------------------------------------------
global
    # to have these messages end up in /var/log/haproxy.log you will
    # need to:
    #
    # 1) configure syslog to accept network log events.  This is done
    #    by adding the '-r' option to the SYSLOGD_OPTIONS in
    #    /etc/sysconfig/syslog
    #
    # 2) configure local2 events to go to the /var/log/haproxy.log
    #   file. A line like the following can be added to
    #   /etc/sysconfig/syslog
    #
    #    local2.*                       /var/log/haproxy.log
    #
    log         127.0.0.1 local2
    #log         /dev/log local2 debug
    #log         /dev/log local2 notice
    #log         127.0.0.1 local2 info
    #log         127.0.0.1 local2 notice

    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    #maxconn     4000
    user        haproxy
    group       haproxy
    daemon

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats
    stats timeout 30s

    # utilize system-wide crypto-policies
    ssl-default-bind-ciphers PROFILE=SYSTEM
    ssl-default-server-ciphers PROFILE=SYSTEM

#---------------------------------------------------------------------
# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
#---------------------------------------------------------------------

defaults
    mode                    http
    log                     global
    option                  dontlognull
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    #maxconn                 3000
    #errorfile 400 /opt/haproxy/share/err/400.http
    #errorfile 403 /opt/haproxy/share/err/403.http
    #errorfile 408 /opt/haproxy/share/err/408.http
    #errorfile 500 /opt/haproxy/share/err/500.http
    #errorfile 502 /opt/haproxy/share/err/502.http
    #errorfile 503 /opt/haproxy/share/err/503.http
    #errorfile 504 /opt/haproxy/share/err/504.http



#---------------------------------------------------------------------
# Port 443
#---------------------------------------------------------------------

#---------------------------------------------------------------------
# frontend
#---------------------------------------------------------------------

frontend tb2_443
    bind 10.1.60.130:443
    mode tcp
    option tcplog
    acl tls req.ssl_hello_type 1
    tcp-request inspect-delay 5s
    tcp-request content accept if tls
    default_backend xoce
    
#---------------------------------------------------------------------
# backend
#---------------------------------------------------------------------	
backend xoce
    mode tcp
    option ssl-hello-chk
    server xoce_srv         172.20.100.2:443 send-proxy check

but ...

It seams that XO don't support Proxy Protocol

Secure Connection Failed

An error occurred during a connection to 10.1.60.130. PR_END_OF_FILE_ERROR

Error code: PR_END_OF_FILE_ERROR

    The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
    Please contact the website owners to inform them of this problem.

In XO code it must be included proxy-protocol ( npm install proxy-protocol ), to have this possibility.
This is the code:

const http = require('http');
const ProxyProtocol = require('proxy-protocol');

const server = http.createServer((req, res) => {
    // Parse Proxy Protocol headers
    const proxy = new ProxyProtocol();
    const proxyData = proxy.parse(req);

    // Extract client IP and Port from the parsed headers
    const clientIP = proxyData && proxyData.address ? proxyData.address : req.connection.remoteAddress;
    const clientPort = proxyData && proxyData.port ? proxyData.port : req.connection.remotePort;

    // Handle the request
    // ...
});

server.listen(3000);

olivierlambert

Weird, it shouldn't require anything baked into XO to support a reverse proxy Any comment/idea @julien-f ?

edit: it shouldn't be needed since HTTP protocol supports the "Forwarded" directive (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded)

Gheppy

@olivierlambert
It is HTTPS ( not HTTP ) and proxy-protocol-v2, and as far as I know the "Forwarded" directive is not supported on HTTPS, only on HTTP

delaf

@Gheppy The RFC says that Forwarded must (when active) be set on HTTP requets but not HTTP response. It use HTTP for HTTP and HTTPS in this context, that is confirmed by :

   Information passed in this header field can be, for example, the source IP
   address of the request, the IP address of the incoming interface on
   the proxy, or whether HTTP or HTTPS was used

The syntax describe by mozilla (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) :

Syntax: Forwarded: by=<identifier>;for=<identifier>;host=<host>;proto=<http|https>

Gheppy

@delaf
Forwarded it seams that is supported with HAProxy 2.8, I use 2.6. I'll install 2.8 from source to see if is ok

delaf

@Gheppy ok, here even with the Forwarded header (set by an HAProxy 2.8), XO does not log in right IP (in the audit log).

john.c

@delaf @olivierlambert Maybe worth investigating this to get this corrected so it will log the right IP address in the audit log.

delaf

@olivierlambert any idea on what can I do to help to debug this issue?

Step to reproduce the issue :

Install xo and make it listening on 127.0.0.1:8080
Install HAProxy 2.8 on Debian 12 (check https://haproxy.debian.net/) :

# curl https://haproxy.debian.net/bernat.debian.org.gpg | gpg --dearmor > /usr/share/keyrings/haproxy.debian.net.gpg
# echo "deb [signed-by=/usr/share/keyrings/haproxy.debian.net.gpg] http://haproxy.debian.net bookworm-backports-2.8 main" > /etc/apt/sources.list.d/haproxy.list
# apt-get update
# apt-get install haproxy=2.8.\*
# systemctl stop haproxy

Configure HAProxy
Very small HAProxy config (/etc/haproxy/haproxy.cfg), update the bind line to listen on the IP:PORT you want. XO is listening on 127.0.0.1:8080.

global
  log /dev/log local0
  log /dev/log local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s

  user haproxy
  group haproxy
  daemon

defaults
  mode http
  log global
  log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Td=%Tt %ST %U %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[last_rule_file]:%[last_rule_line] %ID"
  error-log-format "%ci:%cp [%tr] %ft %ac/%fc %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc %{+Q}[fc_err_str]"
  option dontlognull
  option redispatch
  timeout connect 5s
  timeout client 50s
  timeout server 50s
  unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
  unique-id-header X-Unique-ID

frontend ft_public
  bind IP:PORT
  mode http
  default_backend bk_xo

backend bk_xo
  mode http
  option forwarded
  option forwardfor
  option http-server-close
  http-request add-header HAProxy yes
  server xo 127.0.0.1:8080 check

Start HAProxy: systemctl start haproxy
Test it http://IP:PORT
Check on the network the headers sent by HAProxy to XO tcpdump -ni lo port 8080

olivierlambert

What's happening on step 5 and what's an example of sent stuff on 6?