XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    1. Home
    2. delaf
    Offline
    • Profile
    • Following 0
    • Followers 0
    • Topics 1
    • Posts 44
    • Groups 0

    delaf

    @delaf

    17
    Reputation
    54
    Profile views
    44
    Posts
    0
    Followers
    0
    Following
    Joined
    Last Online

    delaf Unfollow Follow

    Best posts made by delaf

    • RE: Alert: Control Domain Memory Usage

      stormi
      It seems to be good here!

      Screenshot 2021-03-09 at 08.36.50.png

      posted in Compute
      delafD
      delaf
    • RE: New Rust Xen guest tools

      olivierlambert when changes are made (like IP change in the VM), it is now instantly reported by XO 🙂
      I will be able to go back to my work to integrate a BGP deamon to announce locals IPs to the network (I started last year to work on the xe-deamon but never had the time to finish it, and some months after I saw you blog post about moving to rust for it).

      Thank you!

      posted in Development
      delafD
      delaf
    • RE: Alert: Control Domain Memory Usage

      PS: i'm using these 2 scripts to list all interfaces drivers version accross our servers :

      $ cat get_network_drivers_info.sh
      #!/bin/bash                                                                                                                                                                                                                                                                                                   
      
      format="| %-13.13s | %-20.20s | %-20.20s | %-10.10s | %-7.7s | %-10.10s | %-30.30s | %-s \n"
      printf "${format}" "date" "hostname" "OS" "interface" "driver" "version" "firmware" "yum"
      printf "${format}" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------" "----------------------------"
      
      if [ $# -gt 0 ]; then
          servers=($(echo ${BASH_ARGV[*]}))
      else
          servers=($(cat host.json | jq -r '.[] | .address' | egrep -v "^192.168.124.9$"))
      fi
      
      for line in ${servers[@]}; do
          scp get_network_drivers_info.sh.tpl ${line}:/tmp/get_network_drivers_info.sh  > /dev/null 2>&1;
          ssh -n ${line} bash /tmp/get_network_drivers_info.sh 2> /dev/null;
          if [ $? -ne 0 ]; then
              echo "${line} fail" >&2
          fi
      done
      
      $ cat get_network_drivers_info.sh.tpl
      #!/bin/bash                                                                                                                                                                                                                                                                                                   
      
      format="| %-13.13s | %-20.20s | %-20.20s | %-10.10s | %-7.7s | %-10.10s | %-30.30s | %-s \n"
      d=$(date '+%Y%m%d-%H%M')
      name=$(hostname)
      cd  /sys/class/net/
      for interface in $(ls -l /sys/class/net/ | awk '/\/pci/ {print $9}'); do
          version=$(ethtool -i ${interface} | awk '/^version:/ {$1=""; print}')
          firmware=$(ethtool -i ${interface} | awk '/^firmware-version:/ {$1=""; print}')
          driver=$(ethtool -i ${interface} | awk '/^driver:/ {$1=""; print}')
          YUM=$(which yum)
          if [ $? -eq 0 ]; then
              packages=$(yum list installed | awk '/ixgbe/ {print $1"@"$2}' | tr '\n' ',')
          else
              packages="NA"
          fi
          os_version=$(lsb_release -d | awk '{$1=""} 1' | sed 's/XenServer/XS/; s/ (xenenterprise)//; s/release //')
          printf "${format}" "${d}" "${name}" "${os_version}" "${interface}" "${driver}" "${version}" "${firmware}" "${packages}"
      done
      

      PS: host.json file is generated via : xo-cli --list-objects type=host

      posted in Compute
      delafD
      delaf
    • RE: Alert: Control Domain Memory Usage

      stormi Hello, some week after, I can confirm that the problem is solved here by using intel-ixgbe.x86_64@5.5.2-2.1.xcpng8.1 or intel-ixgbe.x86_64@5.5.2-2.1.xcpng8.2

      posted in Compute
      delafD
      delaf
    • RE: Alert: Control Domain Memory Usage

      stormi I have installed intel-ixgbe 5.5.2-2.1.xcpng8.2 on my server s0267. Let's wait a some days to check if the memleak is solved by this patch.

      posted in Compute
      delafD
      delaf
    • RE: Alert: Control Domain Memory Usage

      stormi
      Screenshot 2021-03-04 at 21.21.16.png
      The 2 servers have been reinstalled with an up to date 8.2. They host each 2 VMs that are doing the same thing (~100Mb/s of netdata stream).

      The right one has the 5.9.4-1.xcpng8.2, the left one has 5.5.2-2.xcpng8.2.

      The patch seem to be OK for me.

      posted in Compute
      delafD
      delaf
    • RE: Alert: Control Domain Memory Usage

      stormi

      • server 266 with alt-kernel: still no problem.
        Screen Shot 2020-12-02 at 10.08.47.png

      • server 268 with 4.19.19-6.0.10.1.xcpng8.1: the problem has begun some days ago after some stable days.
        Screen Shot 2020-12-02 at 10.03.57.png

      • server 272 with 4.19.19-6.0.11.1.0.1.patch53disabled.xcpng8.1:
        Screen Shot 2020-12-02 at 10.05.47.png )

      • server 273 with 4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1:
        Screen Shot 2020-12-02 at 10.05.50.png

      It seems that 4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1 is more stable than 4.19.19-6.0.11.1.0.1.patch53disabled.xcpng8.1. But it is a but early to be sure.

      posted in Compute
      delafD
      delaf
    • RE: Alert: Control Domain Memory Usage

      stormi For the kernel-4.19.19-6.0.10.1.xcpng8.1 test, i'm not sure it solve the problem because I get a small memory increase. We have to wait a bit more 😕

      posted in Compute
      delafD
      delaf
    • RE: Alert: Control Domain Memory Usage

      stormi I have installed the two kernels

      272 ~]# yum list installed kernel | grep kernel
      kernel.x86_64                   4.19.19-6.0.11.1.0.1.patch53disabled.xcpng8.1
      
      273 ~]# yum list installed kernel | grep kernel
      kernel.x86_64                   4.19.19-6.0.11.1.0.1.patch62disabled.xcpng8.1
      

      I have removed the modification in /etc/modprobe.d/dist.conf on server 273.

      We have to wait a little bit now 😉

      posted in Compute
      delafD
      delaf
    • RE: Alert: Control Domain Memory Usage

      stormi r1
      Four days later, I get:

      • one server (266) with alt-kernel: still no problem
      • one server (268) with 4.19.19-6.0.10.1.xcpng8.1: no more problem!
      • one server (272) with kmemleak kernel: no memleak detected, but the problem is present
      • one server (273) with search extra built-in weak-updates override updates: problem still present
      posted in Compute
      delafD
      delaf

    Latest posts made by delaf

    • RE: Reverse Proxy Configuration - HAProxy

      julien-f I think we should add this in the reverse proxy documentaion (https://xen-orchestra.com/docs/configuration.html#reverse-proxy). I have created https://github.com/vatesfr/xen-orchestra/pull/7289

      gdelafond opened this pull request in vatesfr/xen-orchestra

      closed docs(configuration): useForwardedHeaders #7289

      posted in Xen Orchestra
      delafD
      delaf
    • RE: Reverse Proxy Configuration - HAProxy

      julien-f I have both case with a forwarding proxy on 127.0.0.1: useForwardedHeaders = ['127.0.0.1'] and useForwardedHeaders = ['10.0.0.0/8'] and all seems OK 🙂

      posted in Xen Orchestra
      delafD
      delaf
    • RE: Reverse Proxy Configuration - HAProxy

      julien-f it works 🙂

      As it is said in the doc, relying on x-forwarded-* should be done carefuly in a controlled network.
      julien-f FYI, some software (like mod_remoteip for apache) use another variable to know from which proxy it is safe to use x-forwarded-* (https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html#remoteiptrustedproxy).

      BWT thank you!

      posted in Xen Orchestra
      delafD
      delaf
    • RE: Reverse Proxy Configuration - HAProxy

      olivierlambert

      • step 5, you get access to XO.
      • step 6 (I did a curl on http://X.X.X.X:PORT/signin)

      Request sent by HAProxy:

      09:34:03.294200 IP 127.0.0.1.34134 > 127.0.0.1.8080: Flags [P.], seq 1:266, ack 1, win 512, options [nop,nop,TS val 2919287515 ecr 2919287515], length 265: HTTP: GET /signin HTTP/1.1
      E..=^.@.@............V.PR,.5.........1.....
      ........GET /signin HTTP/1.1
      host: xoau.ivy1.aquaray.com:8080
      user-agent: curl/7.74.0
      accept: */*
      haproxy: yes
      x-unique-id: AC1014F0:BCD6_AC1014F0:1F90_6576C97B_000A:1783DC
      forwarded: proto=http;for=X.X.X.X
      x-forwarded-for: X.XX.X
      connection: close
      

      Response from XO:

      09:34:03.296545 IP 127.0.0.1.8080 > 127.0.0.1.34134: Flags [P.], seq 1:2009, ack 266, win 512, options [nop,nop,TS val 2919287517 ecr 2919287515], length 2008: HTTP: HTTP/1.1 200 OK
      E...t.@.@..`.........P.V....R,.>...........
      ........HTTP/1.1 200 OK
      X-DNS-Prefetch-Control: off
      X-Frame-Options: SAMEORIGIN
      Strict-Transport-Security: max-age=15552000; includeSubDomains
      X-Download-Options: noopen
      X-Content-Type-Options: nosniff
      X-XSS-Protection: 1; mode=block
      Content-Type: text/html; charset=utf-8
      Content-Length: 1464
      ETag: W/"5b8-OqUsZViW2KwDMOq1IfmEYkCzkN0"
      Set-Cookie: connect.sid=s%3A3imvxT96Uq9L224R8NpArAHuW5Ho7jOS.ArGj0Ms2cIXEalxqdYSk95JS7J0ihftv%2FcURH53p07A; Path=/; HttpOnly
      Vary: Accept-Encoding
      Date: Mon, 11 Dec 2023 08:34:03 GMT
      Connection: close
      
      posted in Xen Orchestra
      delafD
      delaf
    • RE: Reverse Proxy Configuration - HAProxy

      olivierlambert any idea on what can I do to help to debug this issue?

      Step to reproduce the issue :

      1. Install xo and make it listening on 127.0.0.1:8080
      2. Install HAProxy 2.8 on Debian 12 (check https://haproxy.debian.net/) :
      # curl https://haproxy.debian.net/bernat.debian.org.gpg | gpg --dearmor > /usr/share/keyrings/haproxy.debian.net.gpg
      # echo "deb [signed-by=/usr/share/keyrings/haproxy.debian.net.gpg] http://haproxy.debian.net bookworm-backports-2.8 main" > /etc/apt/sources.list.d/haproxy.list
      # apt-get update
      # apt-get install haproxy=2.8.\*
      # systemctl stop haproxy
      
      1. Configure HAProxy
        Very small HAProxy config (/etc/haproxy/haproxy.cfg), update the bind line to listen on the IP:PORT you want. XO is listening on 127.0.0.1:8080.
      global
        log /dev/log local0
        log /dev/log local1 notice
        chroot /var/lib/haproxy
        stats socket /run/haproxy/admin.sock mode 660 level admin
        stats timeout 30s
      
        user haproxy
        group haproxy
        daemon
      
      defaults
        mode http
        log global
        log-format "%ci:%cp [%t] %ft %b/%s %Th/%Ti/%TR/%Tw/%Tc/%Tr/%Td=%Tt %ST %U %B %CC %CS %tsc %ac/%fc/%bc/%sc/%rc %sq/%bq %hr %hs %{+Q}r %sslv %sslc %[last_rule_file]:%[last_rule_line] %ID"
        error-log-format "%ci:%cp [%tr] %ft %ac/%fc %[fc_err]/%[ssl_fc_err,hex]/%[ssl_c_err]/%[ssl_c_ca_err]/%[ssl_fc_is_resumed] %[ssl_fc_sni]/%sslv/%sslc %{+Q}[fc_err_str]"
        option dontlognull
        option redispatch
        timeout connect 5s
        timeout client 50s
        timeout server 50s
        unique-id-format %{+X}o\ %ci:%cp_%fi:%fp_%Ts_%rt:%pid
        unique-id-header X-Unique-ID
      
      frontend ft_public
        bind IP:PORT
        mode http
        default_backend bk_xo
      
      backend bk_xo
        mode http
        option forwarded
        option forwardfor
        option http-server-close
        http-request add-header HAProxy yes
        server xo 127.0.0.1:8080 check
      
      1. Start HAProxy: systemctl start haproxy

      2. Test it http://IP:PORT 😉

      3. Check on the network the headers sent by HAProxy to XO tcpdump -ni lo port 8080

      posted in Xen Orchestra
      delafD
      delaf
    • RE: Reverse Proxy Configuration - HAProxy

      Gheppy ok, here even with the Forwarded header (set by an HAProxy 2.8), XO does not log in right IP (in the audit log).

      posted in Xen Orchestra
      delafD
      delaf
    • RE: Reverse Proxy Configuration - HAProxy

      Gheppy The RFC says that Forwarded must (when active) be set on HTTP requets but not HTTP response. It use HTTP for HTTP and HTTPS in this context, that is confirmed by :

         Information passed in this header field can be, for example, the source IP
         address of the request, the IP address of the incoming interface on
         the proxy, or whether HTTP or HTTPS was used
      

      The syntax describe by mozilla (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded) :

      Syntax: Forwarded: by=<identifier>;for=<identifier>;host=<host>;proto=<http|https>
      
      posted in Xen Orchestra
      delafD
      delaf
    • RE: Reverse Proxy Configuration - HAProxy

      @john-c said in Reverse Proxy Configuration - HAProxy:

      Well in the case of forward-for it is fully implementing RFC7239 ("forwarded") which maintains information which may be altered or lost when proxies are involved. This implementation is for both processing and generation. See this article for more information on the involvement of forwarded headers (https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Forwarded), then this article is for how it affects the HAProxy specifically (https://www.haproxy.com/blog/announcing-haproxy-2-8).

      Oh now, I remember I saw that when the 2.8 was released. Thank you for the link!

      BTW, even with option forwarded (and not option forward) I did not get the real IP in XO logs. Do you have any idea what I should do to debug this on the XO side?
      PS: in a tcpdump trace, I see the rights headers sent to XO: forwarded: proto=https;for=X.X.X.X and my old x-forwarded-for: X.X.X.X.

      posted in Xen Orchestra
      delafD
      delaf
    • RE: Reverse Proxy Configuration - HAProxy

      @john-c Could you explain what has changed in 2.8 that fix a forward-for problem? I did not see any related changes in 2.8 changelog.
      There is nothing parlicular to do to in HAProxy to support websocket.

      posted in Xen Orchestra
      delafD
      delaf
    • RE: Reverse Proxy Configuration - HAProxy

      @ tbigs2012 I think you should add option forwardfor to add the X-Forwarded-For` header.

      olivierlambert AFAIK There is no specific configuration for websockets to make HAProxy compatible with XO.

      olivierlambert I have a similar config, but it seems that XO does not take the content of the X-Forwarded-For header. I know that some software have a IP whitelist to know when they are allowed to look into x-forwarded-for for the client IP. Is there some config like this in XO?

      PS: I do not have "login/logout" actions with the related IP in the audit log. I think it could be a nice addon 😉

      posted in Xen Orchestra
      delafD
      delaf