@SylvainB said in WORM Backups with XCP-ng / Xen Orchestra - Seeking Solutions & Experience:

Hello everyone,

I'm exploring options for implementing WORM (Write Once, Read Many) capabilities for my backups within my XCP-ng environment, specifically using Xen Orchestra.

My current setup:

• XCP-ng Version: 8.3
• Xen Orchestra Version: 5.106.4 (Stable)
• Intended Backup Target: Synology NAS

My primary goal is to ensure that my backup data, once written, becomes immutable for a defined retention period, offering protection against accidental deletion or ransomware attacks.

My questions are:

1. Does Xen Orchestra offer any native WORM features or integrations that I might be overlooking for its backup jobs?
2. If not directly, has anyone successfully implemented WORM backups with a similar perimeter (XCP-ng, Xen Orchestra, and potentially a Synology NAS or other storage solution)? I'm very interested in learning about your setup, the specific technologies you used (e.g., storage features, specific configurations), and any lessons learned or best practices.

Any insights, architectural recommendations, or shared experiences would be highly valuable.

Thank you in advance for your help!

Best regards,

SylvainB

You can setup in the Synology its WriteOnce feature, then configure the appropriate configuration settings, including retention ones.

This will prepare the appropriate WORM environment you’re looking for. It will work well due it being the equivalent to Vates solution and/or the S3 based one.

https://kb.synology.com/en-in/search?tags[]=WriteOnce

@wtdrisco said in Hosts compatibility:

As I am starting to build an environment for testing to replace VMWare, I had a question related to hardware.

When setting up multiple hosts, do these need to match the same specs (like VMWare?) for HA (moving VMs from host to host)?

I have several DELL R series servers, and some do not have the exact same CPU model or one has less memory than the other.

When setting up (HOST POOLS??) if I needed to migrate VMs, will this support different host configurations?

If the the hosts don't match by close enough, especially if their capabilities (e.g. instruction sets) and specifications. Then in the case of capabilities then the non-matching ones will be suppressed by XCP-ng so that they all match. Also when migrating the specifications, of hosts really need to match so that when VMs are placed on the hosts. There's no issues when live migrating between the each of the pool member hosts.

As the VMs expect at least a certain number of cores dependent on the hosts, and the number specified per each VM. If this number isn't met then that VM can't migrate to a specific host, which don't meet or exceed it.

@ThierryC01 said in XCP-ng 8.3 betas and RCs feedback :

@bleader Update successful, no issue so far. There is just a message appearing when starting the update: "Delta RPMs disabled because /usr/bin/applydeltarpm not installed."

All VM working for now.

That's harmless. It's just notifying that the system doesn't have deltarpm package installed. This package reduces the size of updates based on what's already installed.

@still_at_work said in First SMAPIv3 driver is available in preview:

Hello @olivierlambert ,

I am joining this topic as I have a few questions about SMAPIv3:

• Will it allow provisioning of VDIs larger than 2TB?

• Will it enable thin provisioning on iSCSI SRs?

Currently, the blockers I encounter are related to my iSCSI storage. This is a major differentiating factor compared to other vendors, and resolving these blockers would significantly increase your market share.

Thanks !

@still_at_work The size limit of the VDI is due to the file format used for these, which is VHD (https://en.wikipedia.org/wiki/VHD_(file_format)). This format can't support more than 2TB, it's known about and are dealing with the issue. It will likely result in a change or addition of a new VDI format likely to be qcow2 unless necessary software for VHDX format is fully open sourced and software for Xen is created which enables create, read, write and use of this format.

It's not a limitation of iSCSI as it also emerges with both NFS and SMB based connections.

@jasonnix The Linux distribution operating system restricts certain commands, operations and/or path (location) to the root account, as well as the location owner.

The path your trying to write to is just one such location as the location is read only to non-root users. If you really wish to write to /usr/local/src then root account is required.

If you use the root account to install then the directory and the Xen Orchestra files will be owner by root. You would need to create a non-root user for the xen orchestra server process. As well as change ownership of the Xen Orchestra files and folders to that non-root account. Though at least the ones which Xen Orchestra should be able to modify as required, when needed.

As well as provide the necessary sudo permissions for the necessary commands to that account.

Using root account to run exposes any vulnerabilities present to the capacity of being exploited as root (aka Administrator) privileges.

Also how good is your internet connection and your connection to GitHub? Cause you could be experiencing a time out packet loss connection issue.

@Octopuss said in ️ XO 6: dedicated thread for all your feedback!:

@olivierlambert Well ok, it's just that last time I tried to ask about stuff I was politely told off because I used the installation script instead of paying (that's not the literal formulation but it's basically what the person responding meant).

I know nothing about any blogs, I just check the main xcp website (FFS I am not even allowed to post links, what a forum...) from time to time, so I have no idea. I just updated the admin interfce for the first time since moving over from ESXi, and was puzzled why it looked the same as the "castrated" lite version when previously (which means v5, apparently) it was full of features. I had no idea it wasn't a completed product.

TL;DR: I am just an idiot who installed this few months ago and has no idea about the details. shrug

You were told off if the script was 3rd party because officially, the scripts may have made changes to the Xen Orchestra source code prior to compilation. They have a set of steps, for building from source which are able to receive support in the forums for. Anyway getting paid support if an organisation is best as their pro support is top notch!

With the blogs which are being referred to are under the “News” links. Which detail releases and other news worthy information.

You can post links but it does take some work, also make sure you’ve read the rules, also going through the introduction training which a forum function bot runs. Doing this will eventually earn you more access and feature functionality. It’ll help you get used to the forums functions.

The XO 6 is a minimum viable product currently, but is growing from there as features from v5 transitions to v6. If there’s features missing from v5 and v6 that are needed please enter them on the Feedback portal, so they can be voted on. It’s link is in several XOA update and release news posts (especially 6.0 and 6.1).

Read both these posts along with others along the way, it will help you get caught up on all of the Xen Orchestra releases and updates.

@olivierlambert I created the issue on GithHub. https://github.com/vatesfr/xen-orchestra/issues/7884

MrGrymReaper created this issue in vatesfr/xen-orchestra

open xo-server fix: Update acme-client npm and introduce support for External Account Binding (EAB) #7884

@peb2 said in XOA letsencrpyt module not setting acmeDomain:

Hello,

I'm new to Xen and XOA (migrating from Vmware) and I was wondering if anyone could help me with a problem.

I'm trying to use the Letsencrypt module as specified in:

https://github.com/vatesfr/xen-orchestra/blob/master/%40xen-orchestra/mixins/docs/SslCertificate.md?ref=xen-orchestra.com

However, the XOA doesn't seem to be passing the variable for 'acmeDomain' so I get a certificate without a CN.

Has anyone else seen this problem or know what my mistake may be?

Here's my /etc/xo-server/config.toml:

#=====================================================================

# HTTP proxy configuration used by xo-server to fetch resources on the
# Internet.
#
# See: https://github.com/TooTallNate/node-proxy-agent#maps-proxy-protocols-to-httpagent-implementations
#httpProxy = 'http://jsmith:qwerty@proxy.lan:3128'

# Configuration of the embedded HTTP server.
[http]
# If set to true, all HTTP traffic will be redirected to the first
# HTTPs configuration.

redirectToHttps = true

# Basic HTTP.
[[http.listen]]
# Port on which the server is listening on.
port = 80


# Basic HTTPS.
[https]
#
# You can find the list of possible options there https://nodejs.org/docs/latest/api/tls.html#tls.createServer
#
# The only difference is the presence of the certificate and the
# key.

hostname = '0.0.0.0'

[[http.listen]]
port = 443

# Whether to autogenerate a self signed certificate if the `cert` and `file`
# entries could not be found.
#
# Default: true
#autoCert = false
autoCert = true

# File containing the certificate (PEM format).
#
# If a chain of certificates authorities is needed, you may bundle
# them directly in the certificate.
#
# Note: the order of certificates does matter, your certificate
# should come first followed by the certificate of the above
# certificate authority up to the root.
#cert = '/etc/ssl/cert.pem'
cert = '/etc/ssl/xoa.cert.pem'

# File containing the private key (PEM format).
#
# If the key is encrypted, the passphrase will be asked at
# server startup.
#key = '/etc/ssl/key.pem'
key = '/etc/ssl/xoa.key.pem'

# ACME (e.g. Let's Encrypt, ZeroSSL) CA directory
#
# Specifies the URL to the ACME CA's directory.
#
# A identifier `provider/directory` can be passed instead of a URL, see the
# list of supported directories here: https://www.npmjs.com/package/acme-client#directory-urls
#
# Note that the application cannot detect that this value has changed.
#
# In that case delete the certificate and the key files, and restart the
# application to generate new ones.
#
# Default is 'letsencrypt/production'
acmeCa = 'zerossl/production'

# Domain for which the certificate should be created.
#
# This entry is required.
acmeDomain = 'xoa.mydomain.com'

# Optional email address which will be used for the certificate creation.
#
# It will be notified of any issues.
acmeEmail = 'myemail@mydomain.com'

The acmeDomain needs to be the actual Fully Qualified Domain Name (FQDN) which XOA uses. Also some verification and/or validation will need to be added either as a text file to the XOA appliance (in a web accessible directory) or as a DNS text record for XOA FQDN.

Also update the acmeEmail so its the actual email to be used for receiving the notification of issues. These don't look valid to me this is critical for them to be real, active and valid as the ACME Client uses these to send the certificate requests and renewals to the certificate authority!!

@Statitica said in Install XCP-ng in old HP ProLiant DL160 G6 (gen 6):

@ilu said in Install XCP-ng in old HP ProLiant DL160 G6 (gen 6):

@nick.lloyd Thankyou...i'll try the last version, reading forums people says HP was problematic, thats why i was asking for help.

Runs fine on my HP hardware, and was pretty easy to install.

@john.c said in Install XCP-ng in old HP ProLiant DL160 G6 (gen 6):

@ilu Though note that XCP-ng 8.3.0 when it goes LTS, will be getting a fix for the 2TB VDI limit. This will allow you to handle VMs which have large, disks.

You have a source for this? IIRC, the fix the the 2TB limit will fix a few other storage limitations too so I'm keen to know more.

https://xcp-ng.org/blog/2025/03/14/the-future-of-xcp-ng-lts/

@andsmith said in 8.3 USB Passthrough - Win Server 2019 BSOD:

@dinhngtu Sorry, I failed to mention this is a Windows Server 2019 VM, it appears that Windbg is a Windows 10/11 application. I've googled installing on Server 2019, but haven't had any luck getting it installed.

Update the AppX Package Deployment software via Microsoft Store or its equivalent in Windows Server 2019. Then once winget is updated (or installed) run the following command:

winget install Microsoft.WinDbg

@olivierlambert If this works for andsmith then it may be worth documenting this, for future users of Windows Server 2019, 2022 and possibly if required the upcoming Windows Server 2025.

@acebmxer I’m sorry to say that if those Dells are at your workplace, the wrong edition of iDRAC was purchased. You see with at the very least iDRAC 9 access to the full granular IPMI sensor data, was placed behind an edition paywall by Dell Technologies. Outlet temperature is just one of the feeds, missing from the Enterprise or lower edition of iDRAC 9. You’ll get the temperature readings from the Dell iDRAC web browser based interface, but not IPMI with iDRAC 9 Enterprise.

To obtain the full IPMI sensor data you need the Datacenter edition.

@olivierlambert Worth having a word with Dell following checking, if IPMI can read under Dell the iDRAC edition, as this will affect readings reported by ipmi.

@acebmxer said:

The server i pulled info from was Dell r660 idrac 9 Enterprise.... My other servers are Dell r640s all have idrac 9 enterprise.

My other hosts are older

That’s why the outlet temperature didn’t show, the edition isn’t high enough - see my linked PDF on the above, post. Without the Datacenter edition of iDRAC you’ll lose access to the full IPMI SDR sensor records, so won’t be available to the ipmi tool, thus the ipmi xapi plugin!

@olivierlambert @acebmxer There’s readings for temperatures for outlets, connected to PSUs, especially in enterprise servers and/or workstations. It may not cover other outlets, but giving the ones going out through PSU may give some idea.

Having Datacenter edition helps, if Dell iDRAC. Even better is using Redfish especially with same edition!

The reason being as editions below Datacenter on Dell iDRAC have masking on subsets of the SDR sensor records, the highest edition unlocks the full SDR dataset.

https://www.delltechnologies.com/asset/en-us/products/servers/industry-market/openmanage-portfolio-software-licensing-guide.pdf

https://www.delltechnologies.com/asset/en-us/solutions/infrastructure-solutions/briefs-summaries/integrated_dellemc_remote_access_controller.pdf

@olivierlambert said:

I think it was fixed since, are you sure you are using an uptodate XO?

Commit 0be23

Anyway was directing attention to the GitHub issue, which was opened. The issue has another reference to it.

@pdonias @olivierlambert Ronivay has opened a bug report, GitHub issue about a problem with Xen Orchestra V5 vs V6 interface, when using an bound address (hostname) for the interface.

https://github.com/vatesfr/xen-orchestra/issues/9500

ronivay created this issue in vatesfr/xen-orchestra

open v5 interface doesn't work if hostname set #9500

@rama said:

@olivierlambert thank you. but is it possible to keep tracking all the CURD operation like we have in terraform. but currently MCP have only Read tasks.

Like if some new interns in my lab don't know about this and in this agentic framework if he/she need a VM's, delete or update it can be done very quick. it will save many hours.

I hope this will be available in future or if you wish to do tell me how far it is.

The plugin MCP Server is read only by design to keep using it safe, to have an MCP for reading and another for writing is best practice.

If you desire to have a separate MCP server for the writing actions, feel free to suggest that in the feedback portal. You can even develop your own MCP server, which makes calls to the write side of the XO REST API.

https://modelcontextprotocol.io/

@flakpyro said in backup mail report says INTERRUPTED but it's not ?:

@olivierlambert Using the prebuilt XOA appliance which reports:

[08:39 23] xoa@xoa:~$ node  --version
v20.18.3

@majorp93 @pilow Can you please capture some heap snapshots from during backup runs of XOA via NodeJS?

Then compare them to each other, they need to be in the following order:-

1. Snapshot before backup
2. Snapshot following first backup
3. Snapshot following second backup
4. Snapshot following third backup
5. Snapshot following subsequent backups to get to Node.js OOM (or as close as you’re willing to risk)

These will require that XOA (or XOCE) is started with Node.js heap snapshots enabled. Then open in a Chromium based browser the following url:-
chrome://inspect

The above URL will require using the browser’s DevTools features!

Another option is to integrate and enable use of Clinic.js (clinic heapprofiler), or configure node to use node-heapdump when it reaches a threshold amount.

Once your got those heap dumps your looking for the following:-

• Object types that grow massively between the snapshots.
• Large arrays or maps of backup-related objects (VMs, snapshots, jobs, tasks, etc.).
• Retained objects whose “retainers” point to long-lived structures (global, caches, singletons).

These will likely help to pin down what and where in the backup code, the memory leak is located.

Once have these a heap snapshot diff showing which object type (or types) growing by a stated size per backup will finally help the Vates developers fix this issue.

@florent I left the above for the original reporters of the memory leak issue, and/or yourselves.

@acebmxer Yes, definitely congratulations on migrating to Vates. You’ll be seeing improvements and innovation as a result. I’ve requested something which is about to come to fruition in Xen Orchestra, something which will possibly be of interest and use to you.

https://github.com/vatesfr/xen-orchestra/pull/9519

mpiton opened this pull request in vatesfr/xen-orchestra

closed feat(mcp): add @xen-orchestra/mcp package #9519

@Emmanuel-V said in Low end devices , share your experiences:

I wonder to which extend we could lower dom0 memory, eventually reduce the set of features for this kind of usecases.
I can definitely see a pool of machines (x86 or ARM) for industrial purpose where that could make sense, and that could be managed and update through XOA.

Actually it’s not just the above, this technology (MicroVM) will also be of interest to hyperscalers down the road. As such I think that the handling of this should be an install time choice, as changing an already existing host to a MicroVM capable one can be messy. Doing it this way will enable a lot of capabilities for use with Xen Orchestra, in the handling of such MicroVMs.

A hyperscaler already has this technology (Amazon AWS) with their Firecracker software technology.