MajorP93

MajorP93

Hello XCP-ng community!

Since Vates released the new OpenMetrics plugin for Xen Orchestra we now have an official, built-in exporter for Prometheus metrics!

I was using xen-exporter before in order to make hypervisor internal RRD database available in the form of Prometheus metrics.
I migrated to the new plugin which works just fine.

I updated the Grafana dashboard that I was using in order to be compatible with the official OpenMetrics plugin and thought "why not share it with other users"?

In case you are interested you can find my dashboard JSON here: https://gist.github.com/MajorP93/3a933a6f03b4c4e673282fb54a68474b

It is based on the xen-exporter dashboard made by MikeDombo: https://grafana.com/grafana/dashboards/16588-xen/

In case you also use Prometheus for scraping Xen Orchestra OpenMetrics plugin in combination with Grafana you can copy the JSON from my gist, import it and you are ready to go!

Hope it helps!

Might even be a good idea to include the dashboard as an example in the Xen Orchestra documentation.

Best regards

MajorP93

@ph7 As a suggestion: it would improve readibility if you paste your logs on Vates official pastebin tool: https://paste.vates.tech/ and share the links here instead of posting the logs in multiple messages.

MajorP93

@rzr Thanks! How long does it take for these patches go be available?

[17:00 xcpng01 ~]# yum update
Geladene Plugins: fastestmirror
Loading mirror speeds from cached hostfile
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-base: mirrors.xcp-ng.org
Excluding mirror: updates.xcp-ng.org
 * xcp-ng-updates: mirrors.xcp-ng.org
No packages marked for update

//EDIT: Nevermind. After running yum clean metadata they are being picked up.

MajorP93

I updated my test environment and performed a few tests:

migrating VMs back and forth between VHD based NFS SR and QCOW2 based iSCSI SR --> VMs got converted between VHD and QCOW2 just fine, live migration worked
creation of rather big QCOW2 based VMs (2.5+ TB)
NBD-enabled delta backups of a mixed set of VMs (small, big, QCOW2, VHD)

All tests worked fine so far. Only thing that I noticed: When converting VHD-based VMs to QCOW2 format I was not able to storage migrate more than 2 VMs at a time. XO said something about "not enough memory". That might be related to my dom0 in test environment only having 4GB of RAM. Maybe not related to VHD to QCOW2 migration path. I never saw this error in my live environment where all node's dom0 have 8GB RAM.

Update candidate looks good so far from my point of view.

MajorP93

@MathieuRA I disabled Traefik and reverted to my old XO config (port 443, ssl encryption, http to https redirection), rebuild the docker container using your branch and tested:

it is working fine on my end now

Thank you very much!

I did not expect this to get fixed so fast!

MajorP93

@acebmxer Hello, thanks for sharing this here!

Me and my team have been waiting for this release!

I signed up for the Veeam newsletter hoping I would get notified once Veeam 13.1 (an XCP-ng plugin) gets released but they seem to be a bit slow.

So I found out about this release due to your post.

Will start testing!

MajorP93

I updated to branch "mra-fix-rest-memory-leak".
I will look at backup job results tomorrow and report back.

MajorP93

@Pilow said in backup mail report says INTERRUPTED but it's not ?:

@MajorP93 you say to have 8GB Ram on XO, but it OOMkills at 5Gb Used RAM.

did you do those additionnal steps in your XO Config ?

You can increase the memory allocated to the XOA VM (from 2GB to 4GB or 8GB).
Note that simply increasing the RAM for the VM is not enough.
You must also edit the service file (/etc/systemd/system/xo-server.service) 
to increase the memory allocated to the xo-server process itself.

You should leave ~512MB for the debian OS itself. Meaning if your VM has 4096MB total RAM, you should use 3584 for the memory value below.

- ExecStart=/usr/local/bin/xo-server
+ ExecStart=/usr/local/bin/node --max-old-space-size=3584 /usr/local/bin/xo-server
The last step is to refresh and restart the service:

$ systemctl daemon-reload
$ systemctl restart xo-server

Interesting!
I did not know that it is recommended to set "--max-old-space-size=" as a startup parameter for Node JS with the result of (total system ram - 512MB).
I added that, restarted XO and my backup job.

I will test if that gives my backup jobs more stability.
Thank you very much for taking the time and recommending the parameter.

MajorP93

@Mang0Musztarda said in Xen Orchestra OpenMetrics Plugin - Grafana Dashboard:

@MajorP93 hi, how can i scrape openmetrics endpoint?
i set up openmetrics plugin prometheus secret, enabled it, and ten tried to use curl like that: curl -H "Authorization: Bearer abc123" http://localhost:9004
but response i got was
{"error":"Query authentication does not match server setting"}
what am i doing wrong?

Hey!
I scrape it like so:

root@prometheus01:~# cat /etc/prometheus/scrape_configs/xen-orchestra-openmetrics.yml 
scrape_configs:
  - job_name: xen-orchestra
    honor_labels: true
    scrape_interval: 30s
    scrape_timeout: 20s
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/bearer.token
    metrics_path: /openmetrics/metrics
    static_configs:
    - targets:
      - xen-orchestra.domain.local

/etc/prometheus/bearer.token file contains the bearer token as configured in openmetrics xen orchestra plugin.

best regards

MajorP93

@anthoineb @gduperrey @bleader
Hi!

Thank you very much for your responses.

I actually found the issue and it was not caused by the XCP-ng patches.
Appearently one of my switches had a malfunction and lost it's jumbo frames config on the ports involved...
It seems like this happened in the time frame between XCP-ng updates .
Last time I used this setup the jumbo frames / storage setup was working fine so I thought it might be related to these patches.

Anyways sorry for taking your time in this regard-

Best regards

MajorP93

@anthoineb @gduperrey @bleader
Hi!

Thank you very much for your responses.

I actually found the issue and it was not caused by the XCP-ng patches.
Appearently one of my switches had a malfunction and lost it's jumbo frames config on the ports involved...
It seems like this happened in the time frame between XCP-ng updates .
Last time I used this setup the jumbo frames / storage setup was working fine so I thought it might be related to these patches.

Anyways sorry for taking your time in this regard-

Best regards

MajorP93

Hello, so I got access to the test environment back and was able to install this set of patches.

I actually installed them on the XCP-ng host before you guys released them to the stable repository.

I ran:

yum clean metadata --enablerepo=xcp-ng-testing,xcp-ng-candidates
yum update --enablerepo=xcp-ng-testing,xcp-ng-candidates

and rebooted the system.

Unfortunately I have to say that this is the first time that patches broke my system.

While the XCP-ng host is still able to boot, it is not longer able to mount my SRs.

I have 2 SR in this test environment:

1x Linstor vSAN iSCSI configured as QCOW2
1x TrueNAS Core NFSv3 configured as VHD

I spent some hours troubleshooting this and appearently it is caused by jumbo frames no longer working after applying these XCP-ng patches.

The hypervisor / storage network in this testing environment is using jumbo frames everywhere (all switches involved, all storage systems).
Prior to installing the updates everything was working fine.

Now I can not ping the storage systems anymore using jumbo frames.
(ping -M do -s 8972 ...)

Hence the tasks that are meant to mount the SRs are stuck forever:

[14:38 xcpng-test01 ~]# xe task-list
uuid ( RO)                : a6ba1324-bfe1-8aca-c5d1-d7cc57d37cca
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000


uuid ( RO)                : f9c76606-e516-553f-706c-ff52bc303e2d
          name-label ( RO): PBD.plug
    name-description ( RO): 
              status ( RO): pending
            progress ( RO): 0.000

The interesting thing is that "ip a" is still showing MTU 9000:

[14:44 xcpng-test01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: eth0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
4: eth4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq master ovs-system state UP group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
5: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
6: eth3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq master ovs-system state DOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
7: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
    link/ether 4a:1c:22:40:4a:d2 brd ff:ff:ff:ff:ff:ff
8: xenbr2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fc brd ff:ff:ff:ff:ff:ff
9: xenbr3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ec:0d:9a:8c:00:fd brd ff:ff:ff:ff:ff:ff
10: xenbr4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
11: xenbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b3 brd ff:ff:ff:ff:ff:ff
12: xenbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether ac:1f:6b:ad:2c:b2 brd ff:ff:ff:ff:ff:ff
13: xapi1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 80:61:5f:10:85:87 brd ff:ff:ff:ff:ff:ff
    inet 10.10.160.24/24 brd 10.10.160.255 scope global xapi1
       valid_lft forever preferred_lft forever

Can't start XO VM right now as it lives on one of the SR but XO Lite is also still showing jumbo frames being enabled:

Was something changed in this set of patches that could cause this issue?

Thanks and best regards

MajorP93

Hmmm I have been reading quite often in the last months that people have issues with RPU.

I tried RPU during every single update within the last year and for me it only failed when one (or multiple) VMs have not been live migrateable.

That is not really the "fault" of the RPU process but rather related to the VMs.

Finding out the cause has been a bit finicky though. (reading through log file)

Maybe the biggest improvement to RPU would be some form of better error reporting?

That would make it more clear to people what actually caused the issues rather than thinking "RPU is broken".

MajorP93

@gleh Wow! This sounds like a big release especially in the QCOW2 / storage department.
A huge thanks to the whole XCP-ng team!
The platform keeps improving and improving which is awesome to see.

I will test this batch of packages next week when my test environment is available again.

Best regards

MajorP93

@JB Yes makes sense. You are not on the branch that already has the fix. Your commit is latest master branch.
Either switch to the branch that got mentioned or wait for the fix to land in master.

MajorP93

@msupport @acebmxer
Hello guys.
Thanks for sharing your experience with veeam and reporting these issues.
My team is also planning on evaluating veeam and I was wondering: did you get a response from veeam?
Did they give you a time line on when they plan to release a fix?

MajorP93

@christopher-petzel Thank you for testing this! I am also interested in this fix since I encounter this issue from time to time. Unfortunately my testing environment / lab is currently unavailable which is why I wasn't able to compile & deploy this branch myself.

MajorP93

@pierrebrunet Oh one more thing that I just remembered.
If I recall correctly the first time I saw this issue was right after I patched my XCP-ng hosts without updating Xen Orchestra.
So maybe this is caused by XAPI code changes?

MajorP93

@pierrebrunet Hello, thank you for working on this issue.

Are you completely sure that this isn't something that got introduced within Xen Orchestra code?

Since I started using XCP-ng / Xen Orchestra I have been using these backup repositories via SMB.
All backup jobs (including this one) have been working fine for almost a year.

This repeated body timeout error is a rather new issue which is why I was wondering if this could have been introduced by XO code changes?

I did not switch to a different linux kernel major version or similar. So the SMB client kernel module is at the same version number.

Best regards

MajorP93

@olivierlambert said:

Hi! Can you try to replicate on the latest commit?

Hello!
Yes I was able to replicate this issue after updating my Xen Orchestra.

The thing is that the "XO-config-and-pool-metadata" does not always fail.
It fails from time to time then it looks exactly like shown in the screenshot.
Attached logs please find above in my earlier post.

This exact job has been working for a long time before.
After some Xen Orchestra commit this issue got introduced.

I use the same backup remotes for multiple backup jobs.
Only the "XO-config-and-pool-metadata" job is failing with this body timeout error.

Best regards