@julien-f I think we should add this in the reverse proxy documentaion (https://xen-orchestra.com/docs/configuration.html#reverse-proxy). I have created https://github.com/vatesfr/xen-orchestra/pull/7289

gdelafond opened this pull request in vatesfr/xen-orchestra

closed docs(configuration): useForwardedHeaders #7289

@julien-f I have both case with a forwarding proxy on 127.0.0.1: useForwardedHeaders = ['127.0.0.1'] and useForwardedHeaders = ['10.0.0.0/8'] and all seems OK

@julien-f it works

As it is said in the doc, relying on x-forwarded-* should be done carefuly in a controlled network.
@julien-f FYI, some software (like mod_remoteip for apache) use another variable to know from which proxy it is safe to use x-forwarded-* (https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html#remoteiptrustedproxy).

BWT thank you!

@olivierlambert

• step 5, you get access to XO.
• step 6 (I did a curl on http://X.X.X.X:PORT/signin)

Request sent by HAProxy:

09:34:03.294200 IP 127.0.0.1.34134 > 127.0.0.1.8080: Flags [P.], seq 1:266, ack 1, win 512, options [nop,nop,TS val 2919287515 ecr 2919287515], length 265: HTTP: GET /signin HTTP/1.1
E..=^.@.@............V.PR,.5.........1.....
........GET /signin HTTP/1.1
host: xoau.ivy1.aquaray.com:8080
user-agent: curl/7.74.0
accept: */*
haproxy: yes
x-unique-id: AC1014F0:BCD6_AC1014F0:1F90_6576C97B_000A:1783DC
forwarded: proto=http;for=X.X.X.X
x-forwarded-for: X.XX.X
connection: close

Response from XO:

09:34:03.296545 IP 127.0.0.1.8080 > 127.0.0.1.34134: Flags [P.], seq 1:2009, ack 266, win 512, options [nop,nop,TS val 2919287517 ecr 2919287515], length 2008: HTTP: HTTP/1.1 200 OK
E...t.@.@..`.........P.V....R,.>...........
........HTTP/1.1 200 OK
X-DNS-Prefetch-Control: off
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Content-Type: text/html; charset=utf-8
Content-Length: 1464
ETag: W/"5b8-OqUsZViW2KwDMOq1IfmEYkCzkN0"
Set-Cookie: connect.sid=s%3A3imvxT96Uq9L224R8NpArAHuW5Ho7jOS.ArGj0Ms2cIXEalxqdYSk95JS7J0ihftv%2FcURH53p07A; Path=/; HttpOnly
Vary: Accept-Encoding
Date: Mon, 11 Dec 2023 08:34:03 GMT
Connection: close

@olivierlambert any idea on what can I do to help to debug this issue?

Step to reproduce the issue :

1. Install xo and make it listening on 127.0.0.1:8080
2. Install HAProxy 2.8 on Debian 12 (check https://haproxy.debian.net/) :

# curl https://haproxy.debian.net/bernat.debian.org.gpg | gpg --dearmor > /usr/share/keyrings/haproxy.debian.net.gpg
# echo "deb [signed-by=/usr/share/keyrings/haproxy.debian.net.gpg] http://haproxy.debian.net bookworm-backports-2.8 main" > /etc/apt/sources.list.d/haproxy.list
# apt-get update
# apt-get install haproxy=2.8.\*
# systemctl stop haproxy

3. Configure HAProxy
   Very small HAProxy config (/etc/haproxy/haproxy.cfg), update the bind line to listen on the IP:PORT you want. XO is listening on 127.0.0.1:8080.

global
  log /dev/log local0
  log /dev/log local1 notice
  chroot /var/lib/haproxy
  stats socket /run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s

  user haproxy
  group haproxy
  daemon

defaults
  mode http
  log global
  log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Td=%Tt %ST %U %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[last_rule_file]:%[last_rule_line] %ID"
  error-log-format "%ci:%cp [%tr] %ft %ac/%fc %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc %{+Q}[fc_err_str]"
  option dontlognull
  option redispatch
  timeout connect 5s
  timeout client 50s
  timeout server 50s
  unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
  unique-id-header X-Unique-ID

frontend ft_public
  bind IP:PORT
  mode http
  default_backend bk_xo

backend bk_xo
  mode http
  option forwarded
  option forwardfor
  option http-server-close
  http-request add-header HAProxy yes
  server xo 127.0.0.1:8080 check

4. Start HAProxy: systemctl start haproxy

5. Test it http://IP:PORT

6. Check on the network the headers sent by HAProxy to XO tcpdump -ni lo port 8080

@Gheppy ok, here even with the Forwarded header (set by an HAProxy 2.8), XO does not log in right IP (in the audit log).

@Gheppy The RFC says that Forwarded must (when active) be set on HTTP requets but not HTTP response. It use HTTP for HTTP and HTTPS in this context, that is confirmed by :

   Information passed in this header field can be, for example, the source IP
   address of the request, the IP address of the incoming interface on
   the proxy, or whether HTTP or HTTPS was used

The syntax describe by mozilla (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) :

Syntax: Forwarded: by=<identifier>;for=<identifier>;host=<host>;proto=<http|https>

@john-c said in Reverse Proxy Configuration - HAProxy:

Well in the case of forward-for it is fully implementing RFC7239 ("forwarded") which maintains information which may be altered or lost when proxies are involved. This implementation is for both processing and generation. See this article for more information on the involvement of forwarded headers (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded), then this article is for how it affects the HAProxy specifically (https://www.haproxy.com/blog/announcing-haproxy-2-8).

Oh now, I remember I saw that when the 2.8 was released. Thank you for the link!

BTW, even with option forwarded (and not option forward) I did not get the real IP in XO logs. Do you have any idea what I should do to debug this on the XO side?
PS: in a tcpdump trace, I see the rights headers sent to XO: forwarded: proto=https;for=X.X.X.X and my old x-forwarded-for: X.X.X.X.

@john-c Could you explain what has changed in 2.8 that fix a forward-for problem? I did not see any related changes in 2.8 changelog.
There is nothing parlicular to do to in HAProxy to support websocket.

@ tbigs2012 I think you should add option forwardfor to add the X-Forwarded-For` header.

@olivierlambert AFAIK There is no specific configuration for websockets to make HAProxy compatible with XO.

@olivierlambert I have a similar config, but it seems that XO does not take the content of the X-Forwarded-For header. I know that some software have a IP whitelist to know when they are allowed to look into x-forwarded-for for the client IP. Is there some config like this in XO?

PS: I do not have "login/logout" actions with the related IP in the audit log. I think it could be a nice addon

@olivierlambert when changes are made (like IP change in the VM), it is now instantly reported by XO
I will be able to go back to my work to integrate a BGP deamon to announce locals IPs to the network (I started last year to work on the xe-deamon but never had the time to finish it, and some months after I saw you blog post about moving to rust for it).

Thank you!

@olivierlambert all seems OK here too on a Debian11.

@tcorp8310 for information you can use the xe template-export template-uuid= filename= command to export a template into a file.

@olivierlambert I have the same problem, it is even not 2s.
Firefox 106.0.5 on macOS.

PS: it never worked well for me (ie it is not related to the latest XO versions).

@olivierlambert said in REST API:

For the last point, I invite you stay tuned being read carefully each XO release blog post. It's available in XO 5.72 since last June, see https://xen-orchestra.com/blog/xen-orchestra-5-72/#-rest-api-token-generation

I forgot it or miss it
Thank you!

@olivierlambert said in REST API:

However, there's a big gap with POST actions (doing "actions" in short). Since almost all operations are async, like disk or VM creation, REST needs to be tuned for that. I would say it's the "hard" part in REST vs a connected API like xo-cli with websockets and its persistent connection. We will probably use a "queue" system you can request to get the status of the task and the answer. Eg the UUID of the freshly created VM. All in all, it's more complex and can't be done quickly like adding "GET" actions

I hope you will find something that it is not too complicated to implement in your side!

@olivierlambert said in REST API:

Finally, the permissions IMHO are even more complex, we need to discuss that with @julien-f

Yes permissions are always a big thing.

@olivierlambert
For now, we are using the REST API to integrate information about VM into our ERP. We are tracking changes over time and are providing an up to date inventory to our customers.

Next step for us, would be able to create VM directly through the REST API (and not via xo-cli, or xe).
But on the security side, we will need to be able to restrict API users to some objects as the system must not be able to view/modify some objects (mostly do not be able to get/set info for some pools). Even further, we could imagine that each customer has 2 API KEY (one read, the other read/write) that will be restricted to the customer pool or customer VM.

To help people to use the REST API, it would be nice if xo-web can generate the API KEY.

Regards,

Hello,

We start using the REST API.
As I did not see evolution on it since some months, I would like to know what is the status and what are the plan of this API?

Today the most problematic part I have is the lack of rights in the API : the user must be an admin and have read/write access (even if for now the write access is quite limited as it is only "VDI Import"), and so have access to all ressources.

We are only using Collections for now to get VM list, and VM info (ram, disk, cpu), and so far so good

Regards,

PS: i'm using these 2 scripts to list all interfaces drivers version accross our servers :

$ cat get_network_drivers_info.sh
#!/bin/bash                                                                                                                                                                                                                                                                                                   

format="| %-13.13s | %-20.20s | %-20.20s | %-10.10s | %-7.7s | %-10.10s | %-30.30s | %-s \n"
printf "${format}" "date" "hostname" "OS" "interface" "driver" "version" "firmware" "yum"
printf "${format}" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------"

if [ $# -gt 0 ]; then
    servers=($(echo ${BASH_ARGV[*]}))
else
    servers=($(cat host.json | jq -r '.[] | .address' | egrep -v "^192.168.124.9$"))
fi

for line in ${servers[@]}; do
    scp get_network_drivers_info.sh.tpl ${line}:/tmp/get_network_drivers_info.sh  > /dev/null 2>&1;
    ssh -n ${line} bash /tmp/get_network_drivers_info.sh 2> /dev/null;
    if [ $? -ne 0 ]; then
        echo "${line} fail" >&2
    fi
done

$ cat get_network_drivers_info.sh.tpl
#!/bin/bash                                                                                                                                                                                                                                                                                                   

format="| %-13.13s | %-20.20s | %-20.20s | %-10.10s | %-7.7s | %-10.10s | %-30.30s | %-s \n"
d=$(date '+%Y%m%d-%H%M')
name=$(hostname)
cd  /sys/class/net/
for interface in $(ls -l /sys/class/net/ | awk '/\/pci/ {print $9}'); do
    version=$(ethtool -i ${interface} | awk '/^version:/ {$1=""; print}')
    firmware=$(ethtool -i ${interface} | awk '/^firmware-version:/ {$1=""; print}')
    driver=$(ethtool -i ${interface} | awk '/^driver:/ {$1=""; print}')
    YUM=$(which yum)
    if [ $? -eq 0 ]; then
        packages=$(yum list installed | awk '/ixgbe/ {print $1"@"$2}' | tr '\n' ',')
    else
        packages="NA"
    fi
    os_version=$(lsb_release -d | awk '{$1=""} 1' | sed 's/XenServer/XS/; s/ (xenenterprise)//; s/release //')
    printf "${format}" "${d}" "${name}" "${os_version}" "${interface}" "${driver}" "${version}" "${firmware}" "${packages}"
done

PS: host.json file is generated via : xo-cli --list-objects type=host

@stormi Hello, some week after, I can confirm that the problem is solved here by using intel-ixgbe.x86_64@5.5.2-2.1.xcpng8.1 or intel-ixgbe.x86_64@5.5.2-2.1.xcpng8.2

@stormi oh I did not know that as I never use it: I only know that it exists