XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Reverse Proxy Configuration - HAProxy

    Scheduled Pinned Locked Moved Solved Xen Orchestra
    30 Posts 6 Posters 5.8k Views 7 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J Offline
      john.c @delaf
      last edited by john.c

      @delaf @olivierlambert @tbigs2012 Actually there is WebSocket support in HAProxy which could be used for XO compatibility.

      In order to ensure that it functions properly when using forward-for configuration options, please download and install from the 2.8 branch of HAProxy, unless its features have been back ported by your Linux distribution. Also the QUIC feature has reached production readiness with the 2.8 branch. The 2.8 branch is an LTS branch.

      Then follow the instructions from either of these articles adapting as appropriate (based on the parameters from the XO documentation) taking into account secure configuration principals.

      • https://www.haproxy.com/documentation/haproxy-configuration-tutorials/load-balancing/websocket/
      • https://webhostinggeeks.com/howto/how-to-configure-haproxy-for-websocket-applications/#:~:text=How to Configure HAProxy for WebSocket Applications 1,WebSocket connections. ... 3 Step 3%3A Restart HAProxy
      delafD 1 Reply Last reply Reply Quote 0
      • delafD Offline
        delaf @john.c
        last edited by

        @john-c Could you explain what has changed in 2.8 that fix a forward-for problem? I did not see any related changes in 2.8 changelog.
        There is nothing parlicular to do to in HAProxy to support websocket.

        J 1 Reply Last reply Reply Quote 0
        • J Offline
          john.c @delaf
          last edited by john.c

          @delaf said in Reverse Proxy Configuration - HAProxy:

          @john-c Could you explain what has changed in 2.8 that fix a forward-for problem? I did not see any related changes in 2.8 changelog.
          There is nothing parlicular to do to in HAProxy to support websocket.

          Well in the case of forward-for it is fully implementing RFC7239 ("forwarded") which maintains information which may be altered or lost when proxies are involved. This implementation is for both processing and generation. See this article for more information on the involvement of forwarded headers (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded), then this article is for how it affects the HAProxy specifically (https://www.haproxy.com/blog/announcing-haproxy-2-8).

          Actually there's certain parameters which are required to enable WebSocket support in HAProxy unless the defaults have changed since the articles. There's several items which are required in the backend section of the HAProxy configuration file to enable Websocket support:-

          backend http_back
          option forward
          option forwardfor
          option http-server-close

          Without these and having HAProxy use the http backend by default for the connection it will have difficulty with the WebSocket functionality.

          delafD 1 Reply Last reply Reply Quote 0
          • delafD Offline
            delaf @john.c
            last edited by

            @john-c said in Reverse Proxy Configuration - HAProxy:

            Well in the case of forward-for it is fully implementing RFC7239 ("forwarded") which maintains information which may be altered or lost when proxies are involved. This implementation is for both processing and generation. See this article for more information on the involvement of forwarded headers (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded), then this article is for how it affects the HAProxy specifically (https://www.haproxy.com/blog/announcing-haproxy-2-8).

            Oh now, I remember I saw that when the 2.8 was released. Thank you for the link!

            BTW, even with option forwarded (and not option forward) I did not get the real IP in XO logs. Do you have any idea what I should do to debug this on the XO side?
            PS: in a tcpdump trace, I see the rights headers sent to XO: forwarded: proto=https;for=X.X.X.X and my old x-forwarded-for: X.X.X.X.

            J 1 Reply Last reply Reply Quote 0
            • GheppyG Offline
              Gheppy
              last edited by Gheppy

              With HAProxy to get real IP on backend server you need to

              • Add to backend server
                :443 send-proxy

              • Enable Proxy Protocol on XO (if is supported):
                RemoteIPProxyProtocol On
                RemoteIPHeader X-Forwarded-For
                RemoteIPProxyProtocolExceptions 127.0.0.1

              Examples are from HAProxy and Apache

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                If you want to share a working config, feel free, we can add it to our XO doc 🙂

                GheppyG 1 Reply Last reply Reply Quote 0
                • GheppyG Offline
                  Gheppy
                  last edited by

                  I don't use HAProxy with XO, but I'll do the test and I'll came with more info

                  1 Reply Last reply Reply Quote 0
                  • J Offline
                    john.c @delaf
                    last edited by

                    @delaf Some configuration is required on the XO side involving the backend port to 443.

                    1 Reply Last reply Reply Quote 0
                    • GheppyG Offline
                      Gheppy @olivierlambert
                      last edited by Gheppy

                      @olivierlambert
                      Let say that you have this:

                      • HAProxy server :
                        - External IP 10.1.60.130
                        - Internal IP 172.20.100.1
                      • XO server
                        - IP 172.20.100.2
                        for HAProxy to work it is needed to do this
                      #---------------------------------------------------------------------
                      # Global settings
                      #---------------------------------------------------------------------
                      global
                          # to have these messages end up in /var/log/haproxy.log you will
                          # need to:
                          #
                          # 1) configure syslog to accept network log events.  This is done
                          #    by adding the '-r' option to the SYSLOGD_OPTIONS in
                          #    /etc/sysconfig/syslog
                          #
                          # 2) configure local2 events to go to the /var/log/haproxy.log
                          #   file. A line like the following can be added to
                          #   /etc/sysconfig/syslog
                          #
                          #    local2.*                       /var/log/haproxy.log
                          #
                          log         127.0.0.1 local2
                          #log         /dev/log local2 debug
                          #log         /dev/log local2 notice
                          #log         127.0.0.1 local2 info
                          #log         127.0.0.1 local2 notice
                      
                          chroot      /var/lib/haproxy
                          pidfile     /var/run/haproxy.pid
                          #maxconn     4000
                          user        haproxy
                          group       haproxy
                          daemon
                      
                          # turn on stats unix socket
                          stats socket /var/lib/haproxy/stats
                          stats timeout 30s
                      
                          # utilize system-wide crypto-policies
                          ssl-default-bind-ciphers PROFILE=SYSTEM
                          ssl-default-server-ciphers PROFILE=SYSTEM
                      
                      #---------------------------------------------------------------------
                      # common defaults that all the 'listen' and 'backend' sections will
                      # use if not designated in their block
                      #---------------------------------------------------------------------
                      
                      defaults
                          mode                    http
                          log                     global
                          option                  dontlognull
                          option                  redispatch
                          retries                 3
                          timeout http-request    10s
                          timeout queue           1m
                          timeout connect         10s
                          timeout client          1m
                          timeout server          1m
                          timeout http-keep-alive 10s
                          timeout check           10s
                          #maxconn                 3000
                          #errorfile 400 /opt/haproxy/share/err/400.http
                          #errorfile 403 /opt/haproxy/share/err/403.http
                          #errorfile 408 /opt/haproxy/share/err/408.http
                          #errorfile 500 /opt/haproxy/share/err/500.http
                          #errorfile 502 /opt/haproxy/share/err/502.http
                          #errorfile 503 /opt/haproxy/share/err/503.http
                          #errorfile 504 /opt/haproxy/share/err/504.http
                      
                      
                      
                      #---------------------------------------------------------------------
                      # Port 443
                      #---------------------------------------------------------------------
                      
                      #---------------------------------------------------------------------
                      # frontend
                      #---------------------------------------------------------------------
                      
                      frontend tb2_443
                          bind 10.1.60.130:443
                          mode tcp
                          option tcplog
                          acl tls req.ssl_hello_type 1
                          tcp-request inspect-delay 5s
                          tcp-request content accept if tls
                          default_backend xoce
                          
                      #---------------------------------------------------------------------
                      # backend
                      #---------------------------------------------------------------------	
                      backend xoce
                          mode tcp
                          option ssl-hello-chk
                          server xoce_srv         172.20.100.2:443 send-proxy check
                      
                      

                      but ...

                      It seams that XO don't support Proxy Protocol

                      Secure Connection Failed
                      
                      An error occurred during a connection to 10.1.60.130. PR_END_OF_FILE_ERROR
                      
                      Error code: PR_END_OF_FILE_ERROR
                      
                          The page you are trying to view cannot be shown because the authenticity of the received data could not be verified.
                          Please contact the website owners to inform them of this problem.
                      

                      In XO code it must be included proxy-protocol ( npm install proxy-protocol ), to have this possibility.
                      This is the code:

                      const http = require('http');
                      const ProxyProtocol = require('proxy-protocol');
                      
                      const server = http.createServer((req, res) => {
                          // Parse Proxy Protocol headers
                          const proxy = new ProxyProtocol();
                          const proxyData = proxy.parse(req);
                      
                          // Extract client IP and Port from the parsed headers
                          const clientIP = proxyData && proxyData.address ? proxyData.address : req.connection.remoteAddress;
                          const clientPort = proxyData && proxyData.port ? proxyData.port : req.connection.remotePort;
                      
                          // Handle the request
                          // ...
                      });
                      
                      server.listen(3000);
                      
                      1 Reply Last reply Reply Quote 0
                      • olivierlambertO Offline
                        olivierlambert Vates 🪐 Co-Founder CEO
                        last edited by olivierlambert

                        🤔 Weird, it shouldn't require anything baked into XO to support a reverse proxy 🤔 Any comment/idea @julien-f ?

                        edit: it shouldn't be needed since HTTP protocol supports the "Forwarded" directive (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded)

                        GheppyG 1 Reply Last reply Reply Quote 0
                        • GheppyG Offline
                          Gheppy @olivierlambert
                          last edited by Gheppy

                          @olivierlambert
                          It is HTTPS ( not HTTP ) and proxy-protocol-v2, and as far as I know the "Forwarded" directive is not supported on HTTPS, only on HTTP

                          delafD 1 Reply Last reply Reply Quote 0
                          • delafD Offline
                            delaf @Gheppy
                            last edited by

                            @Gheppy The RFC says that Forwarded must (when active) be set on HTTP requets but not HTTP response. It use HTTP for HTTP and HTTPS in this context, that is confirmed by :

                               Information passed in this header field can be, for example, the source IP
                               address of the request, the IP address of the incoming interface on
                               the proxy, or whether HTTP or HTTPS was used
                            

                            The syntax describe by mozilla (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) :

                            Syntax: Forwarded: by=<identifier>;for=<identifier>;host=<host>;proto=<http|https>
                            
                            GheppyG 1 Reply Last reply Reply Quote 0
                            • GheppyG Offline
                              Gheppy @delaf
                              last edited by Gheppy

                              @delaf
                              Forwarded it seams that is supported with HAProxy 2.8, I use 2.6. I'll install 2.8 from source to see if is ok

                              delafD 1 Reply Last reply Reply Quote 0
                              • delafD Offline
                                delaf @Gheppy
                                last edited by

                                @Gheppy ok, here even with the Forwarded header (set by an HAProxy 2.8), XO does not log in right IP (in the audit log).

                                J 1 Reply Last reply Reply Quote 0
                                • J Offline
                                  john.c @delaf
                                  last edited by john.c

                                  @delaf @olivierlambert Maybe worth investigating this to get this corrected so it will log the right IP address in the audit log.

                                  delafD 1 Reply Last reply Reply Quote 0
                                  • delafD Offline
                                    delaf @john.c
                                    last edited by delaf

                                    @olivierlambert any idea on what can I do to help to debug this issue?

                                    Step to reproduce the issue :

                                    1. Install xo and make it listening on 127.0.0.1:8080
                                    2. Install HAProxy 2.8 on Debian 12 (check https://haproxy.debian.net/) :
                                    # curl https://haproxy.debian.net/bernat.debian.org.gpg | gpg --dearmor > /usr/share/keyrings/haproxy.debian.net.gpg
                                    # echo "deb [signed-by=/usr/share/keyrings/haproxy.debian.net.gpg] http://haproxy.debian.net bookworm-backports-2.8 main" > /etc/apt/sources.list.d/haproxy.list
                                    # apt-get update
                                    # apt-get install haproxy=2.8.\*
                                    # systemctl stop haproxy
                                    
                                    1. Configure HAProxy
                                      Very small HAProxy config (/etc/haproxy/haproxy.cfg), update the bind line to listen on the IP:PORT you want. XO is listening on 127.0.0.1:8080.
                                    global
                                      log /dev/log local0
                                      log /dev/log local1 notice
                                      chroot /var/lib/haproxy
                                      stats socket /run/haproxy/admin.sock mode 660 level admin
                                      stats timeout 30s
                                    
                                      user haproxy
                                      group haproxy
                                      daemon
                                    
                                    defaults
                                      mode http
                                      log global
                                      log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Td=%Tt %ST %U %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[last_rule_file]:%[last_rule_line] %ID"
                                      error-log-format "%ci:%cp [%tr] %ft %ac/%fc %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc %{+Q}[fc_err_str]"
                                      option dontlognull
                                      option redispatch
                                      timeout connect 5s
                                      timeout client 50s
                                      timeout server 50s
                                      unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
                                      unique-id-header X-Unique-ID
                                    
                                    frontend ft_public
                                      bind IP:PORT
                                      mode http
                                      default_backend bk_xo
                                    
                                    backend bk_xo
                                      mode http
                                      option forwarded
                                      option forwardfor
                                      option http-server-close
                                      http-request add-header HAProxy yes
                                      server xo 127.0.0.1:8080 check
                                    
                                    1. Start HAProxy: systemctl start haproxy

                                    2. Test it http://IP:PORT 😉

                                    3. Check on the network the headers sent by HAProxy to XO tcpdump -ni lo port 8080

                                    1 Reply Last reply Reply Quote 0
                                    • olivierlambertO Offline
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by

                                      What's happening on step 5 and what's an example of sent stuff on 6?

                                      delafD 1 Reply Last reply Reply Quote 0
                                      • delafD Offline
                                        delaf @olivierlambert
                                        last edited by delaf

                                        @olivierlambert

                                        • step 5, you get access to XO.
                                        • step 6 (I did a curl on http://X.X.X.X:PORT/signin)

                                        Request sent by HAProxy:

                                        09:34:03.294200 IP 127.0.0.1.34134 > 127.0.0.1.8080: Flags [P.], seq 1:266, ack 1, win 512, options [nop,nop,TS val 2919287515 ecr 2919287515], length 265: HTTP: GET /signin HTTP/1.1
                                        E..=^.@.@............V.PR,.5.........1.....
                                        ........GET /signin HTTP/1.1
                                        host: xoau.ivy1.aquaray.com:8080
                                        user-agent: curl/7.74.0
                                        accept: */*
                                        haproxy: yes
                                        x-unique-id: AC1014F0:BCD6_AC1014F0:1F90_6576C97B_000A:1783DC
                                        forwarded: proto=http;for=X.X.X.X
                                        x-forwarded-for: X.XX.X
                                        connection: close
                                        

                                        Response from XO:

                                        09:34:03.296545 IP 127.0.0.1.8080 > 127.0.0.1.34134: Flags [P.], seq 1:2009, ack 266, win 512, options [nop,nop,TS val 2919287517 ecr 2919287515], length 2008: HTTP: HTTP/1.1 200 OK
                                        E...t.@.@..`.........P.V....R,.>...........
                                        ........HTTP/1.1 200 OK
                                        X-DNS-Prefetch-Control: off
                                        X-Frame-Options: SAMEORIGIN
                                        Strict-Transport-Security: max-age=15552000; includeSubDomains
                                        X-Download-Options: noopen
                                        X-Content-Type-Options: nosniff
                                        X-XSS-Protection: 1; mode=block
                                        Content-Type: text/html; charset=utf-8
                                        Content-Length: 1464
                                        ETag: W/"5b8-OqUsZViW2KwDMOq1IfmEYkCzkN0"
                                        Set-Cookie: connect.sid=s%3A3imvxT96Uq9L224R8NpArAHuW5Ho7jOS.ArGj0Ms2cIXEalxqdYSk95JS7J0ihftv%2FcURH53p07A; Path=/; HttpOnly
                                        Vary: Accept-Encoding
                                        Date: Mon, 11 Dec 2023 08:34:03 GMT
                                        Connection: close
                                        
                                        1 Reply Last reply Reply Quote 0
                                        • olivierlambertO Offline
                                          olivierlambert Vates 🪐 Co-Founder CEO
                                          last edited by

                                          Does it ring a bell @julien-f ?

                                          julien-fJ 1 Reply Last reply Reply Quote 0
                                          • julien-fJ Offline
                                            julien-f Vates 🪐 Co-Founder XO Team @olivierlambert
                                            last edited by

                                            @delaf Can you test the PR https://github.com/vatesfr/xen-orchestra/pull/7233?

                                            You will need to set http.useForwardedHeaders to true in your xo-server's config.

                                            Let me know if that helps 🙂

                                            julien-f opened this pull request in vatesfr/xen-orchestra

                                            closed feat(xo-server): http.useForwardedHeaders #7233

                                            delafD 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post