Hello XCP-ng community!

Since Vates released the new OpenMetrics plugin for Xen Orchestra we now have an official, built-in exporter for Prometheus metrics!

I was using xen-exporter before in order to make hypervisor internal RRD database available in the form of Prometheus metrics.
I migrated to the new plugin which works just fine.

I updated the Grafana dashboard that I was using in order to be compatible with the official OpenMetrics plugin and thought "why not share it with other users"?

In case you are interested you can find my dashboard JSON here: https://gist.github.com/MajorP93/3a933a6f03b4c4e673282fb54a68474b

It is based on the xen-exporter dashboard made by MikeDombo: https://grafana.com/grafana/dashboards/16588-xen/

In case you also use Prometheus for scraping Xen Orchestra OpenMetrics plugin in combination with Grafana you can copy the JSON from my gist, import it and you are ready to go!

Hope it helps!

Might even be a good idea to include the dashboard as an example in the Xen Orchestra documentation.

Best regards

@MathieuRA I disabled Traefik and reverted to my old XO config (port 443, ssl encryption, http to https redirection), rebuild the docker container using your branch and tested:

it is working fine on my end now

Thank you very much!

I did not expect this to get fixed so fast!

@Pilow said in backup mail report says INTERRUPTED but it's not ?:

@MajorP93 you say to have 8GB Ram on XO, but it OOMkills at 5Gb Used RAM.

did you do those additionnal steps in your XO Config ?

You can increase the memory allocated to the XOA VM (from 2GB to 4GB or 8GB).
Note that simply increasing the RAM for the VM is not enough.
You must also edit the service file (/etc/systemd/system/xo-server.service) 
to increase the memory allocated to the xo-server process itself.

You should leave ~512MB for the debian OS itself. Meaning if your VM has 4096MB total RAM, you should use 3584 for the memory value below.

- ExecStart=/usr/local/bin/xo-server
+ ExecStart=/usr/local/bin/node --max-old-space-size=3584 /usr/local/bin/xo-server
The last step is to refresh and restart the service:

$ systemctl daemon-reload
$ systemctl restart xo-server

Interesting!
I did not know that it is recommended to set "--max-old-space-size=" as a startup parameter for Node JS with the result of (total system ram - 512MB).
I added that, restarted XO and my backup job.

I will test if that gives my backup jobs more stability.
Thank you very much for taking the time and recommending the parameter.

@Mang0Musztarda said in Xen Orchestra OpenMetrics Plugin - Grafana Dashboard:

@MajorP93 hi, how can i scrape openmetrics endpoint?
i set up openmetrics plugin prometheus secret, enabled it, and ten tried to use curl like that: curl -H "Authorization: Bearer abc123" http://localhost:9004
but response i got was
{"error":"Query authentication does not match server setting"}
what am i doing wrong?

Hey!
I scrape it like so:

root@prometheus01:~# cat /etc/prometheus/scrape_configs/xen-orchestra-openmetrics.yml 
scrape_configs:
  - job_name: xen-orchestra
    honor_labels: true
    scrape_interval: 30s
    scrape_timeout: 20s
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/bearer.token
    metrics_path: /openmetrics/metrics
    static_configs:
    - targets:
      - xen-orchestra.domain.local

/etc/prometheus/bearer.token file contains the bearer token as configured in openmetrics xen orchestra plugin.

best regards

Can also confirm that I was able to apply this round of patches using rolling update method without any issues or slowdowns on a pool of 5 hosts.

@rzr Thank you very much!

@michmoor0725 Absolutely! The community is another aspect of why working with XCP-ng is a lot more fun compared to working with VMWare!

@florent said in [VDDK V2V] Migration of VM that had more than 1 snapshot creates multiple VHDs:

@MajorP93 the size are different between the disks, did you modify it since the snapshots ?

would it be possible to take one new snapshot with the same disk structure ?

Sorry it was my bad indeed.
On the VMWare side there are 2 VMs that have almost the exact same name.
When I checked for disk layout to verify this was an issue I looked at the wrong VM.

I checked again and can confirm that the VM in question has 1x 60GiB and 1x 25GiB VMDK.

So this is not an issue. It is working as intended.

Thread can be closed / deleted.
Sorry again and thanks for the replies.

Best regards
MajorP

said in Xen Orchestra Node 24 compatibility:

After moving from Node 22 to Node 24 on my XO instance I started to see more "Error: ENOMEM: not enough memory, close" for my backup jobs even though my XO VM has 8GB of RAM...

I will revert back to Node 22 for now.

I did some further troubleshooting and was able to pinpoint it down to SMB encryption on Xen Orchestra backup remotes ("seal" CIFS mount flag).
"ENOMEM" errors seem to occur only when I enable previously explained option.
Seems to be related to some buffering that is controlled by Linux kernel CIFS implementation that is failing when SMB encryption is being used.
CIFS operation gets killed due to buffer exhaustion caused by encryption and Xen Orchestra shows "ENOMEM".
Somehow this issue gets more visible when using Node 24 vs Node 22 which is why I thought it was caused by the Node version + XO version combination. I switched Node version at the same time I enabled SMB encryption.
However this seems to be not directly related to Xen Orchestra and more a Node / Linux kernel CIFS implementation thing.
Apparently not a Xen Orchestra bug per se.

@dom0 As already previously mentioned XCP-ng Center / XenCenter are not officially supported and a third-party product.
It is generally advised to use Xen Orchestra for all administration / management tasks.

If it is a requirement for you to use a thick client (such as XCP-ng Center) you might want to try XenAdminQt: https://github.com/benapetr/XenAdminQt

It is also not officially supported but a very new project that gets updated frequently. Maybe that one works better for you.

Another thing that I noticed: despite enabling remote syslog (to graylog) for all XCP-ng hosts in the pool /var/log gets filled up to 100%.
Adding remote syslog seem to not change usage of /var/log at all.

Official XCP-ng documentation states otherwise here: https://docs.xcp-ng.org/installation/install-xcp-ng/#installation-on-usb-drives

The linked part of the documentation indicates that configuring remote syslog can be a possible solution for /var/log space constraints which seems to be not the case.

I feel like logging could use some investigation by Vates in general.

Hello Vates and XCP-ng community,

I have updated my Xen Orchestra from sources VM and want to report that after updating my mirror backup job does not work anymore. I updated to commit 875ceb7cab9735efc29c345603a4280cb6072fbc (feat: release XO 6.3.1).

Prior to updating XO it has been working fine for weeks. I did not change anything about it's config.

I get "Error: Cannot read properties of undefined (reading 'id')" for each any every VM that the mirror job tries to mirror.

It is a "mirror incremental" backup job.

Is this a known issue?
Did anyone else encounter this?

If a XO developer needs logs, I can provide them in a private message as I am not sure if they contain sensitive information in this case.

My XO VM is running on Debian 13 and Node 22 by the way.

Thanks and best regards

@Pilow I am using Xen Orchestra from sources. I think the commits that improved RAM / heap usage are not released for XOA. But I suspect a XOA release coming soon as they already pushed "technical release" commit which is mostly an indicator that a XOA release will follow soon.

I can confirm that recent changes in Xen Orchestra improved the situation a lot.
I am still monitoring and will keep testing before coming to a conclusion but so far the backups did not crash due to Node heap size being exhausted .

I can confirm that when using Citrix/Xenserver guest utilities version 8.4 (https://github.com/xenserver/xe-guest-utilities/releases/tag/v8.4.0) memory ballooning / DMC is working fine.

After live migration the RAM of the linux guest is expanded to dynamic_max again.

So this issue was in fact caused by Rust based xen-guest-agent.

For now I'll keep using Citrix/Xenserver guest utilities on my Linux guests until the feature is implemented in Vates rust-based guest utilities.

Best regards

@planedrop Pretty sure the official and recommended way is to use the V2V (VMWare to Vates) tool that is baked into Xen Orchestra.

I used that tool back when I migrated from our old VMWare Cluster to the new XCP-ng environment. It worked without any issues.

Best approach IMO would be to disconnect the unneeded virtual disks on the source VM on VMWare side and then use V2V tool to migrate over the VM.
I did it this way for a VM with disk bigger than 2TB aswell. After migrating the VM I created an LVM volume on the XCP-ng VM with the required size and rsync-ed the remaining data from VMDK > 2TB to target XCP-ng LVM volume.

Documentation: https://docs.xen-orchestra.com/v2v-migration-guide

@dinhngtu @teddyastie Do you have a time line / rough estimated release date regarding Rust-based xen-guest-agent version 1.0?
Currently evaluating if I need to switch my Linux guests back to Citrix guest tools in the meantime.

Looking forward to a reply.

Thank you in advance.

@olivierlambert Can you provide insights? Is nested virtualization as in running Hyper-V inside a XCP-ng VM expected to work?

@User-cxs I have one last idea:

• shutdown your VM
• disable nested virtualization for your VM via Xen Orchestra
• SSH login on your XCP-ng host (pool master)
• execute "xe vm-param-set uuid=<UUID> platform:exp-nested-hvm=true" (UUID should be the UUID of your VM)
• start your VM and try again

By following these steps you will use Xen parameter "exp-nested-hvm" instead of "nested-virt" on your VM. This may or may not make a difference.

If that does not change anything I am out of ideas.
Best regards

@User-cxs PV drivers / Xen Tools installed? Was the system rebooted cleanly after enabling nested virtualization/Hyper-V?

@User-cxs Is there an error message when clicking on / hovering over the yellow warning symbol next to "Nested virtualization" on your Xen Orchestra screenshot?

You need that feature for Docker desktop virtualization to work in your VM.

Also: do you have PV drivers / guest utilities installed on your VM?