Hello XCP-ng community!

Since Vates released the new OpenMetrics plugin for Xen Orchestra we now have an official, built-in exporter for Prometheus metrics!

I was using xen-exporter before in order to make hypervisor internal RRD database available in the form of Prometheus metrics.
I migrated to the new plugin which works just fine.

I updated the Grafana dashboard that I was using in order to be compatible with the official OpenMetrics plugin and thought "why not share it with other users"?

In case you are interested you can find my dashboard JSON here: https://gist.github.com/MajorP93/3a933a6f03b4c4e673282fb54a68474b

It is based on the xen-exporter dashboard made by MikeDombo: https://grafana.com/grafana/dashboards/16588-xen/

In case you also use Prometheus for scraping Xen Orchestra OpenMetrics plugin in combination with Grafana you can copy the JSON from my gist, import it and you are ready to go!

Hope it helps!

Might even be a good idea to include the dashboard as an example in the Xen Orchestra documentation.

Best regards

I updated my test environment and performed a few tests:

• migrating VMs back and forth between VHD based NFS SR and QCOW2 based iSCSI SR --> VMs got converted between VHD and QCOW2 just fine, live migration worked
• creation of rather big QCOW2 based VMs (2.5+ TB)
• NBD-enabled delta backups of a mixed set of VMs (small, big, QCOW2, VHD)

All tests worked fine so far. Only thing that I noticed: When converting VHD-based VMs to QCOW2 format I was not able to storage migrate more than 2 VMs at a time. XO said something about "not enough memory". That might be related to my dom0 in test environment only having 4GB of RAM. Maybe not related to VHD to QCOW2 migration path. I never saw this error in my live environment where all node's dom0 have 8GB RAM.

Update candidate looks good so far from my point of view.

@MathieuRA I disabled Traefik and reverted to my old XO config (port 443, ssl encryption, http to https redirection), rebuild the docker container using your branch and tested:

it is working fine on my end now

Thank you very much!

I did not expect this to get fixed so fast!

I updated to branch "mra-fix-rest-memory-leak".
I will look at backup job results tomorrow and report back.

@Pilow said in backup mail report says INTERRUPTED but it's not ?:

@MajorP93 you say to have 8GB Ram on XO, but it OOMkills at 5Gb Used RAM.

did you do those additionnal steps in your XO Config ?

You can increase the memory allocated to the XOA VM (from 2GB to 4GB or 8GB).
Note that simply increasing the RAM for the VM is not enough.
You must also edit the service file (/etc/systemd/system/xo-server.service) 
to increase the memory allocated to the xo-server process itself.

You should leave ~512MB for the debian OS itself. Meaning if your VM has 4096MB total RAM, you should use 3584 for the memory value below.

- ExecStart=/usr/local/bin/xo-server
+ ExecStart=/usr/local/bin/node --max-old-space-size=3584 /usr/local/bin/xo-server
The last step is to refresh and restart the service:

$ systemctl daemon-reload
$ systemctl restart xo-server

Interesting!
I did not know that it is recommended to set "--max-old-space-size=" as a startup parameter for Node JS with the result of (total system ram - 512MB).
I added that, restarted XO and my backup job.

I will test if that gives my backup jobs more stability.
Thank you very much for taking the time and recommending the parameter.

@Mang0Musztarda said in Xen Orchestra OpenMetrics Plugin - Grafana Dashboard:

@MajorP93 hi, how can i scrape openmetrics endpoint?
i set up openmetrics plugin prometheus secret, enabled it, and ten tried to use curl like that: curl -H "Authorization: Bearer abc123" http://localhost:9004
but response i got was
{"error":"Query authentication does not match server setting"}
what am i doing wrong?

Hey!
I scrape it like so:

root@prometheus01:~# cat /etc/prometheus/scrape_configs/xen-orchestra-openmetrics.yml 
scrape_configs:
  - job_name: xen-orchestra
    honor_labels: true
    scrape_interval: 30s
    scrape_timeout: 20s
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/bearer.token
    metrics_path: /openmetrics/metrics
    static_configs:
    - targets:
      - xen-orchestra.domain.local

/etc/prometheus/bearer.token file contains the bearer token as configured in openmetrics xen orchestra plugin.

best regards

@rzr Installed those security update candidates in my XCP-ng test environment.
So far no issues whatsoever. Basic operations all work. VMs running etc.

I updated both my test and production XCP-ng environments.

No issues during updates on all 6 hosts.

Can also confirm that I was able to apply this round of patches using rolling update method without any issues or slowdowns on a pool of 5 hosts.

@rzr Thank you very much!

@michmoor0725 Absolutely! The community is another aspect of why working with XCP-ng is a lot more fun compared to working with VMWare!

@marcoi I found out what to enter for "vendor" by running "dmidecode -s system-product-name" on one of my XCP-ng hosts:

[15:45 xcpng01 ~]# dmidecode -s system-product-name
Super Server

//EDIT: in your case it looks like you should enter "To Be Filled By O.E.M." for vendor. Not "To Be Filled By O.E.M. (To Be Filled By O.E.M.)"

For my fellow Supermicro enjoyers: I was able to get the plugin to show some information.
Even if there are some parts missing it is better than nothing in my opinion.

Go to settings --> plugins --> ipmi-sensors --> "+" --> Add --> Fill information and add the vendor string as well as the Sensor Reg Ex.

Hope this helps.

Best regards

@All-Ki Thanks for your work on that plugin!

This is the ipmitool sdr list output on one of my Supermicro based XCP-ng hosts, how can I derive the correct plugin settings (regex) from that?

[13:36 xcpng01 ~]# ipmitool sdr list
CPU Temp         | 59 degrees C      | ok
System Temp      | 30 degrees C      | ok
Peripheral Temp  | 29 degrees C      | ok
M2NVMeSSD Temp1  | no reading        | ns
VRMCpu Temp      | 31 degrees C      | ok
VRMSoc Temp      | 43 degrees C      | ok
VRMABCD Temp     | 34 degrees C      | ok
VRMEFGH Temp     | 36 degrees C      | ok
DIMMA1 Temp      | 36 degrees C      | ok
DIMMB1 Temp      | 35 degrees C      | ok
DIMMC1 Temp      | 34 degrees C      | ok
DIMMD1 Temp      | 33 degrees C      | ok
DIMME1 Temp      | 38 degrees C      | ok
DIMMF1 Temp      | 37 degrees C      | ok
DIMMG1 Temp      | 36 degrees C      | ok
DIMMH1 Temp      | 34 degrees C      | ok
FAN1             | no reading        | ns
FAN2             | 4200 RPM          | ok
FAN3             | no reading        | ns
FAN4             | no reading        | ns
FAN5             | no reading        | ns
FANA             | 3600 RPM          | ok
FANB             | 3600 RPM          | ok
12V              | 12,05 Volts       | ok
5VCC             | 5,02 Volts        | ok
3.3VCC           | 3,33 Volts        | ok
VBAT             | 0x04              | ok
VDDCR            | 1,11 Volts        | ok
VMEMABCD         | 1,23 Volts        | ok
VMEMEFGH         | 1,22 Volts        | ok
VDD_5_DUAL       | 5,16 Volts        | ok
VDD_33_DUAL      | 3,28 Volts        | ok
SOCRUN           | 0,87 Volts        | ok
SOCDUAL          | 0,89 Volts        | ok
Chassis Intru    | 0x00              | ok
PS1 Status       | 0x01              | ok
PS2 Status       | 0x01              | ok

Thanks and best regards

@rzr Installed those security update candidates in my XCP-ng test environment.
So far no issues whatsoever. Basic operations all work. VMs running etc.

@TeddyAstie said:

param-name=platform nomigrate=true

Hi @teddyastie , thanks for working on this.

As per policy I am not allowed to test these parameters in production which is why I had to create a small test setup for being able to try your settings.

I deployed a Debian 13 VM via Cloud-Init on a XCP-ng test host using the official Debian 13 cloud image.

After deploying the VM I had the issue of slow boot.

After shutting the VM down, applying the settings that you just sent and starting it again I can say that you are on the right track!

In my case the boot time is completely normal now and on par with Debian 13 VMs that use BIOS instead of UEFI (for booting).

As this is a workaround and disables live migration this is not an option for production environments but good to have a workaround available anyways for sure!

Do you think it is possible to fix this on hypervisor level while still having live migration etc. enabled or do we have to wait for an upstream fix within Linux kernel tree?

@olivierlambert Awesome work on tracking down the issue!
Very nice, detailed, technical writeup.

@TeddyAstie These spinlock events that cause slow boot also happen on my Debian VMs. Also AMD Epyc CPU in my case.

I noticed that this only happens for UEFI enabled VMs. VMs booting in BIOS mode do not have this issue.

Hi,

I installed the update candidates within my test environment.

Updates installed fine, after reboot all looks good so far.

No apparent issues can be seen.

@pierrebrunet Thanks for fixing this critical issue!

Do you have any plans regarding internal automated tests that would ensure that critical issues like these (deletion / corruption of backups etc.) do not get shipped?

Best regards

@olivierlambert said:

Yes we use them in production since more than a year without any issue.

Only with a caveat: they only work in environments where DMC (Dynamic Memory Control) feature of XCP-ng is not being used.

Reason: Rust-based Xen guest agent currently has no ballooning driver implemented which is present in Citrix Xen guest agent as confirmed by XCP-ng developer here: https://xcp-ng.org/forum/topic/11955/memory-ballooning-dmc-broken-since-xcp-ng-8.3-january-2026-patches/13?_=1779445511424