Okay, to update on my findings:

According to the log lines

[40:0x2e27d000] 312864931 ms: Scavenge 2011.2 (2033.4) -> 2005.3 (2033.6) MB, pooled: 0 MB, 2.31 / 0.00 ms  (average mu = 0.257, current mu = 0.211) task;
[40:0x2e27d000] 312867125 ms: Mark-Compact (reduce) **2023.6** (2044.9) -> **2000.5 (2015.5)** MB, pooled: 0 MB, 83.33 / 0.62 ms  (+ 1867.4 ms in 298 steps since start of marking, biggest step 19.4 ms, walltime since start of marking 2194 ms) (average mu = 0.333,
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

The default heap size seems to be 2GB. I read some Node documentation regarding heap size and understood that configured heap size is honored on a per-process basis.
XO backup seems to spawn multiple node processes (workers) which is why I figured the value I previously set as an attempt to fix my issue was too high (max-old-space-size=6144), 6GB can cause OOM quickly when multiple Node processes are being spawned.

For now I added 512MB to the default heap which results in my heap totaling to 2.5GB.
I hope that this will suffice for my backup jobs to not fail as my log clearly indicated the cause of the Node OOM was the heap ceiling being touched.

If it was caused by Node 22+ RSS there would be other log entries.

Also I was thinking a bit more about what @pilow said and I think I observed something similar.
Due to the "interrupted" issue already occurring a few weeks back I checked "htop" once in a while on my XO VM and noticed that after backup jobs completed the RAM usage not really goes down to the value it was sitting before.
After a fresh reboot of my XO VM RAM usage sits at around 1GB.
During backups it showed around 6GB of 10GB total being used.
After backups finished XO VM was sitting at around 5GB of RAM.
So yeah maybe there is a memory leak somewhere after all.

Anyhow I will keep monitoring this and see if the increased heap makes backup jobs more robust.

Would still be interesting to hear something from XO team in this regard.

Best regards
MajorP

@john.c Considering how widely Node JS is being used out there I highly doubt that memory management in itself is broken in Node 22 and 24.
If that would be the case it would have been covered by IT bloggers and most users would switch to using something else.
Classifying memory management as unstable for the whole LTS branches 22 and 24 is something a LLM would do.
I think it is more likely a XO + Node issue.
@pilow already said that they are using XOA which (AFAIK) is still using Node 20.
Even on Node 20 there seems to be some memory leak ongoing according to them which is why it being a "XO + Node" issue rather than a Node 22/24 being borked in general becomes even more likely.

//EDIT: even if using Node 20 would improve anything here, sticking with it might not be the best idea as Node 20 will become EOL in April 2026.

@john.c Your AI generated reply is not really correct. The LLM you used is repeatedly talking about kernel OOM which did not happen here. Also "No crash logs = likely kernel OOM kill" is not correct. The Xen Orchestra / Node log indicates the Node level crash due to heap issue. I attached the log file to my first post in this thread.

Other than that investigating Node 24 related memory changes might be a good idea as XO documentation recommends to use latest Node LTS version (which is 24 as of now).

@john.c said in backup mail report says INTERRUPTED but it's not ?:

--max-old-space-size=4096

Yeah I already set "--max-old-space-size" as mentioned in my post. I set it to 6GB. I will re-test and post findings to this thread.

I already checked "dmesg" and "journalctl -k | cat". There was no OOM entry. So the kernel OOM killer is not involved in this.
Only the Node JS / Xen Orchestra log indicated the memory heap / OOM issue.

Hey,
sadly issue seems to still exist...

I had 3 backup jobs running at the same time, all of them went into status "interrupted" at the exact same timestamp without me doing anything.
Xen Orchestra did not crash, XO VM has enough RAM, no OOM errors in log or similar.

Interestingly the XO VM seems to still send data to the backup remotes even though the jobs show status interrupted.

This exact same issue happened 1 more time in the past but I thought maybe it is only a one-time-hiccup which is why I did not report it here at that point.

Issue first occurred somewhere around the release of XO6, before the XO6 I never had this kind of issue.

Best regards

//EDIT:
maybe to add some relevant info here. I am using XO from sources on Debian 13, NodeJS 24,
Commit fa110ed9c92acf03447f5ee3f309ef6861a4a0d4 ("feat: release 6.1.0").

//EDIT2: Oh I just read in the initial post of this thread that on Pilow's end XO GUI reported success for the backup tasks but the emails indicated otherwise.
In my case the tasks also have status "interrupted" in XO GUI (and in the emails aswell as shown via screenshots).

//EDIT3: I checked Xen Orchestra logs at the timestamp the backup jobs went into "interrupted" status, please find the log file attached below.
xo-log.txt
Based on the log file it looks like the Node JS process went OOM?
This is really strange as my XO VM has quite a lot of resources (8 vCPU, 10GB RAM). I checked in htop on the XO VM and it never went above 5GB during backups...

//EDIT4: I did some research and found that some people had success with limiting their Node JS heap via environment variable.
I set

export NODE_OPTIONS="--max-old-space-size=6144"

and will check if that fixes the issue for me.

@gduperrey Hey,
I tested it at can confirm that after applying latest set of patches and rebooting remote syslog is still working fine.
It appears to be fixed, good job guys

@gduperrey Thanks!
Will test tomorrow as our internal lab / test environment is currently unavailable.
I will inform you about the results of my testing here.

Best regards

@rzr Thank you very much!

@michmoor0725 Absolutely! The community is another aspect of why working with XCP-ng is a lot more fun compared to working with VMWare!

@stormi Thanks, that sounds great!
If there is something I can test / provide please let me know!

@stormi I was able to replicate this issue in our test environment.
Applying of updates --> rsyslog still working.
Rebooting after applying the updates --> rsyslog not working, same issue as previously explained.

I found out that rsyslog target IP address gets saved in /etc/rsyslog.d/xenserver.conf.
Here are the contents of the files in question:

Before applying the updates via yum:
/etc/rsyslog.conf

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
#$ModLoad imjournal # provides access to the systemd journal
$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
#$OmitLocalLogging on

# File to store the position in the journal
#$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

/etc/rsyslog.d/xenserver.conf:

# Suppress duplicate messages and report "Last line repeated n times"
$RepeatedMsgReduction on

# Don't rate-limit messages - this isn't the right way to go about 
# reducing log size!
$IMUXSockRateLimitInterval 0
$SystemLogRateLimitInterval 0

# Ensure critical and higher level errors are logged synchronously.
$ActionFileEnableSync on
$outchannel crit_log,/var/log/crit.log,104857600,/etc/cron.daily/logrotate
*.crit;mail.none;authpriv.none;cron.none		:omfile:$crit_log
$ActionFileEnableSync off

# Log crit to the console as well
*.crit                                          /dev/hvc0

# Corosync logs useful things at warning
if $programname == 'corosync' and $syslogseverity <= 4 then /dev/hvc0

# HTTP disk server backend used by XHA (LINSTOR SR).
# Redirected to a specific log file instead of daemon.log (facility 3).
$outchannel xcp_http_disk_server_log,/var/log/xcp-http-nbd-server.log,104857600,/etc/cron.daily/logrotate
if $syslogfacility == 3 and $programname == 'http-disk-server' then :omfile:$xcp_http_disk_server_log
& stop

# NBD server used by XHA (LINSTOR SR).
# Redirected to a specific log file instead of daemon.log (facility 3).
$outchannel xcp_nbd_http_server_log,/var/log/xcp-nbd-http-server.log,104857600,/etc/cron.daily/logrotate
if $syslogfacility == 3 and $programname == 'nbd-http-server' then :omfile:$xcp_nbd_http_server_log
& stop

# Log in specific file when a DRBD log is matched.
# Redirected to a specific log file instead of kern.log (facility 0).
$outchannel drbd_kern_log,/var/log/drbd-kern.log,104857600,/etc/cron.daily/logrotate
if $syslogfacility == 0 and re_match($msg, '^\\[[ ]*[0-9]+\\.[0-9]+\\] drbd([0-9]+)?:? ') then :omfile:$drbd_kern_log
& stop

# Log by facility.
$outchannel kern_log,/var/log/kern.log,104857600,/etc/cron.daily/logrotate
kern.*							:omfile:$kern_log

# dlm_controld logs to syslog local4
$outchannel daemon_log,/var/log/daemon.log,104857600,/etc/cron.daily/logrotate
daemon.*;local4.*					:omfile:$daemon_log

$outchannel user_log,/var/log/user.log,104857600,/etc/cron.daily/logrotate
user.*							:omfile:$user_log

# The authpriv file has restricted access.
$outchannel secure_log,/var/log/secure,104857600,/etc/cron.daily/logrotate
authpriv.*						:omfile:$secure_log

# Log all the mail messages in one place.
$outchannel mail_log,/var/log/maillog,104857600,/etc/cron.daily/logrotate
mail.*							:omfile:$mail_log

# Log cron stuff
$outchannel cron_log,/var/log/cron,104857600,/etc/cron.daily/logrotate
cron.*							:omfile:$cron_log

# Save boot messages also to boot.log
$outchannel boot_log,/var/log/boot.log,104857600,/etc/cron.daily/logrotate
local7.*						:omfile:$boot_log

# Xapi rbac audit log echoes to syslog local6
$outchannel audit_log,/var/log/audit.log,104857600,/etc/cron.daily/logrotate
local6.*						:omfile:$audit_log

# Xapi, xenopsd echo to syslog local5
$outchannel xensource_log,/var/log/xensource.log,104857600,/etc/cron.daily/logrotate
local5.*						:omfile:$xensource_log

# xenstore access to syslog local3
$outchannel xenstored_log,/var/log/xenstored-access.log,104857600,/etc/cron.daily/logrotate
local3.info						:omfile:$xenstored_log

# Storage Manager to syslog local2
$outchannel sm_log,/var/log/SMlog,104857600,/etc/cron.daily/logrotate
local2.info						:omfile:$sm_log

# Scheduled snapshots to syslog local1
$outchannel vmss_log,/var/log/VMSSlog,104857600,/etc/cron.daily/logrotate
local1.*						:omfile:$vmss_log

# xcp-rrdd-plugins (info and above) to local0
$outchannel xcp_rrdd_log,/var/log/xcp-rrdd-plugins.log,104857600,/etc/cron.daily/logrotate
local0.info						:omfile:$xcp_rrdd_log

# ignore default rules

*.* @10.10.160.27:5140
*.* ~

After applying the updates via yum:
/etc/rsyslog.conf:

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
#$ModLoad imjournal # provides access to the systemd journal
$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
#$OmitLocalLogging on

# File to store the position in the journal
#$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

/etc/rsyslog.d/xenserver.conf:

# Suppress duplicate messages and report "Last line repeated n times"
$RepeatedMsgReduction on

# Don't rate-limit messages - this isn't the right way to go about 
# reducing log size!
$IMUXSockRateLimitInterval 0
$SystemLogRateLimitInterval 0

# Ensure critical and higher level errors are logged synchronously.
$ActionFileEnableSync on
$outchannel crit_log,/var/log/crit.log,104857600,/etc/cron.daily/logrotate
*.crit;mail.none;authpriv.none;cron.none		:omfile:$crit_log
$ActionFileEnableSync off

# Log crit to the console as well
*.crit                                          /dev/hvc0

# Corosync logs useful things at warning
if $programname == 'corosync' and $syslogseverity <= 4 then /dev/hvc0

# HTTP disk server backend used by XHA (LINSTOR SR).
# Redirected to a specific log file instead of daemon.log (facility 3).
$outchannel xcp_http_disk_server_log,/var/log/xcp-http-nbd-server.log,104857600,/etc/cron.daily/logrotate
if $syslogfacility == 3 and $programname == 'http-disk-server' then :omfile:$xcp_http_disk_server_log
& stop

# NBD server used by XHA (LINSTOR SR).
# Redirected to a specific log file instead of daemon.log (facility 3).
$outchannel xcp_nbd_http_server_log,/var/log/xcp-nbd-http-server.log,104857600,/etc/cron.daily/logrotate
if $syslogfacility == 3 and $programname == 'nbd-http-server' then :omfile:$xcp_nbd_http_server_log
& stop

# Log in specific file when a DRBD log is matched.
# Redirected to a specific log file instead of kern.log (facility 0).
$outchannel drbd_kern_log,/var/log/drbd-kern.log,104857600,/etc/cron.daily/logrotate
if $syslogfacility == 0 and re_match($msg, '^\\[[ ]*[0-9]+\\.[0-9]+\\] drbd([0-9]+)?:? ') then :omfile:$drbd_kern_log
& stop

# Log by facility.
$outchannel kern_log,/var/log/kern.log,104857600,/etc/cron.daily/logrotate
kern.*							:omfile:$kern_log

# dlm_controld logs to syslog local4
$outchannel daemon_log,/var/log/daemon.log,104857600,/etc/cron.daily/logrotate
daemon.*;local4.*					:omfile:$daemon_log

$outchannel user_log,/var/log/user.log,104857600,/etc/cron.daily/logrotate
user.*							:omfile:$user_log

# The authpriv file has restricted access.
$outchannel secure_log,/var/log/secure,104857600,/etc/cron.daily/logrotate
authpriv.*						:omfile:$secure_log

# Log all the mail messages in one place.
$outchannel mail_log,/var/log/maillog,104857600,/etc/cron.daily/logrotate
mail.*							:omfile:$mail_log

# Log cron stuff
$outchannel cron_log,/var/log/cron,104857600,/etc/cron.daily/logrotate
cron.*							:omfile:$cron_log

# Save boot messages also to boot.log
$outchannel boot_log,/var/log/boot.log,104857600,/etc/cron.daily/logrotate
local7.*						:omfile:$boot_log

# Xapi rbac audit log echoes to syslog local6
$outchannel audit_log,/var/log/audit.log,104857600,/etc/cron.daily/logrotate
local6.*						:omfile:$audit_log

# Xapi, xenopsd echo to syslog local5
$outchannel xensource_log,/var/log/xensource.log,104857600,/etc/cron.daily/logrotate
local5.*						:omfile:$xensource_log

# xenstore access to syslog local3
$outchannel xenstored_log,/var/log/xenstored-access.log,104857600,/etc/cron.daily/logrotate
local3.info						:omfile:$xenstored_log

# Storage Manager to syslog local2
$outchannel sm_log,/var/log/SMlog,104857600,/etc/cron.daily/logrotate
local2.info						:omfile:$sm_log

# Scheduled snapshots to syslog local1
$outchannel vmss_log,/var/log/VMSSlog,104857600,/etc/cron.daily/logrotate
local1.*						:omfile:$vmss_log

# xcp-rrdd-plugins (info and above) to local0
$outchannel xcp_rrdd_log,/var/log/xcp-rrdd-plugins.log,104857600,/etc/cron.daily/logrotate
local0.info						:omfile:$xcp_rrdd_log

# ignore default rules
*.*							~

After rebooting:
/etc/rsyslog.conf:

# rsyslog configuration file

# For more information see /usr/share/doc/rsyslog-*/rsyslog_conf.html
# If you experience problems, see http://www.rsyslog.com/doc/troubleshoot.html

#### MODULES ####

# The imjournal module bellow is now used as a message source instead of imuxsock.
$ModLoad imuxsock # provides support for local system logging (e.g. via logger command)
#$ModLoad imjournal # provides access to the systemd journal
$ModLoad imklog # reads kernel messages (the same are read from journald)
#$ModLoad immark  # provides --MARK-- message capability

# Provides UDP syslog reception
#$ModLoad imudp
#$UDPServerRun 514

# Provides TCP syslog reception
#$ModLoad imtcp
#$InputTCPServerRun 514


#### GLOBAL DIRECTIVES ####

# Where to place auxiliary files
$WorkDirectory /var/lib/rsyslog

# Use default timestamp format
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# File syncing capability is disabled by default. This feature is usually not required,
# not useful and an extreme performance hit
#$ActionFileEnableSync on

# Include all config files in /etc/rsyslog.d/
$IncludeConfig /etc/rsyslog.d/*.conf

# Turn off message reception via local log socket;
# local messages are retrieved through imjournal now.
#$OmitLocalLogging on

# File to store the position in the journal
#$IMJournalStateFile imjournal.state


#### RULES ####

# Log all kernel messages to the console.
# Logging much else clutters up the screen.
#kern.*                                                 /dev/console

# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
*.info;mail.none;authpriv.none;cron.none                /var/log/messages

# The authpriv file has restricted access.
authpriv.*                                              /var/log/secure

# Log all the mail messages in one place.
mail.*                                                  -/var/log/maillog


# Log cron stuff
cron.*                                                  /var/log/cron

# Everybody gets emergency messages
*.emerg                                                 :omusrmsg:*

# Save news errors of level crit and higher in a special file.
uucp,news.crit                                          /var/log/spooler

# Save boot messages also to boot.log
local7.*                                                /var/log/boot.log


# ### begin forwarding rule ###
# The statement between the begin ... end define a SINGLE forwarding
# rule. They belong together, do NOT split them. If you create multiple
# forwarding rules, duplicate the whole block!
# Remote Logging (we use TCP for reliable delivery)
#
# An on-disk queue is created for this action. If the remote host is
# down, messages are spooled to disk and sent when it is up again.
#$ActionQueueFileName fwdRule1 # unique name prefix for spool files
#$ActionQueueMaxDiskSpace 1g   # 1gb space limit (use as much as possible)
#$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
#$ActionQueueType LinkedList   # run asynchronously
#$ActionResumeRetryCount -1    # infinite retries if host is down
# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional
#*.* @@remote-host:514
# ### end of the forwarding rule ###

/etc/rsyslog.d/xenserver.conf:

# Suppress duplicate messages and report "Last line repeated n times"
$RepeatedMsgReduction on

# Don't rate-limit messages - this isn't the right way to go about 
# reducing log size!
$IMUXSockRateLimitInterval 0
$SystemLogRateLimitInterval 0

# Ensure critical and higher level errors are logged synchronously.
$ActionFileEnableSync on
$outchannel crit_log,/var/log/crit.log,104857600,/etc/cron.daily/logrotate
*.crit;mail.none;authpriv.none;cron.none		:omfile:$crit_log
$ActionFileEnableSync off

# Log crit to the console as well
*.crit                                          /dev/hvc0

# Corosync logs useful things at warning
if $programname == 'corosync' and $syslogseverity <= 4 then /dev/hvc0

# HTTP disk server backend used by XHA (LINSTOR SR).
# Redirected to a specific log file instead of daemon.log (facility 3).
$outchannel xcp_http_disk_server_log,/var/log/xcp-http-nbd-server.log,104857600,/etc/cron.daily/logrotate
if $syslogfacility == 3 and $programname == 'http-disk-server' then :omfile:$xcp_http_disk_server_log
& stop

# NBD server used by XHA (LINSTOR SR).
# Redirected to a specific log file instead of daemon.log (facility 3).
$outchannel xcp_nbd_http_server_log,/var/log/xcp-nbd-http-server.log,104857600,/etc/cron.daily/logrotate
if $syslogfacility == 3 and $programname == 'nbd-http-server' then :omfile:$xcp_nbd_http_server_log
& stop

# Log in specific file when a DRBD log is matched.
# Redirected to a specific log file instead of kern.log (facility 0).
$outchannel drbd_kern_log,/var/log/drbd-kern.log,104857600,/etc/cron.daily/logrotate
if $syslogfacility == 0 and re_match($msg, '^\\[[ ]*[0-9]+\\.[0-9]+\\] drbd([0-9]+)?:? ') then :omfile:$drbd_kern_log
& stop

# Log by facility.
$outchannel kern_log,/var/log/kern.log,104857600,/etc/cron.daily/logrotate
kern.*							:omfile:$kern_log

# dlm_controld logs to syslog local4
$outchannel daemon_log,/var/log/daemon.log,104857600,/etc/cron.daily/logrotate
daemon.*;local4.*					:omfile:$daemon_log

$outchannel user_log,/var/log/user.log,104857600,/etc/cron.daily/logrotate
user.*							:omfile:$user_log

# The authpriv file has restricted access.
$outchannel secure_log,/var/log/secure,104857600,/etc/cron.daily/logrotate
authpriv.*						:omfile:$secure_log

# Log all the mail messages in one place.
$outchannel mail_log,/var/log/maillog,104857600,/etc/cron.daily/logrotate
mail.*							:omfile:$mail_log

# Log cron stuff
$outchannel cron_log,/var/log/cron,104857600,/etc/cron.daily/logrotate
cron.*							:omfile:$cron_log

# Save boot messages also to boot.log
$outchannel boot_log,/var/log/boot.log,104857600,/etc/cron.daily/logrotate
local7.*						:omfile:$boot_log

# Xapi rbac audit log echoes to syslog local6
$outchannel audit_log,/var/log/audit.log,104857600,/etc/cron.daily/logrotate
local6.*						:omfile:$audit_log

# Xapi, xenopsd echo to syslog local5
$outchannel xensource_log,/var/log/xensource.log,104857600,/etc/cron.daily/logrotate
local5.*						:omfile:$xensource_log

# xenstore access to syslog local3
$outchannel xenstored_log,/var/log/xenstored-access.log,104857600,/etc/cron.daily/logrotate
local3.info						:omfile:$xenstored_log

# Storage Manager to syslog local2
$outchannel sm_log,/var/log/SMlog,104857600,/etc/cron.daily/logrotate
local2.info						:omfile:$sm_log

# Scheduled snapshots to syslog local1
$outchannel vmss_log,/var/log/VMSSlog,104857600,/etc/cron.daily/logrotate
local1.*						:omfile:$vmss_log

# xcp-rrdd-plugins (info and above) to local0
$outchannel xcp_rrdd_log,/var/log/xcp-rrdd-plugins.log,104857600,/etc/cron.daily/logrotate
local0.info						:omfile:$xcp_rrdd_log

# ignore default rules
*.*							~

--> appearently after applying the updates my rsyslog target system got removed from /etc/rsyslog.d/xenserver.conf but this change got active after rebooting.

@stormi Yes, I am currently working on replicating the issue in our test environment. I will report back once done.

@rzr I configured locale via XCP-ng ISO installer. This is my locale output:

[14:38 xcpng01 ~]# locale
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC="de_DE.UTF-8"
LC_TIME="de_DE.UTF-8"
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY="de_DE.UTF-8"
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER="de_DE.UTF-8"
LC_NAME="de_DE.UTF-8"
LC_ADDRESS="de_DE.UTF-8"
LC_TELEPHONE="de_DE.UTF-8"
LC_MEASUREMENT="de_DE.UTF-8"
LC_IDENTIFICATION="de_DE.UTF-8"
LC_ALL=

For everyone hitting the "ENOMEM" error on a Debian 13 system when using SMB/CIFS encryption for transferring backups:
you might want to try a newer kernel.

I was able to solve the issue on my end by moving from kernel 6.12 (the default in Debian 13) to kernel 6.17 from debian backports by executing

apt install -t trixie-backports linux-image-amd64

You can find some changelog for the SMB/CIFS kernel module here: https://wiki.samba.org/index.php/LinuxCIFSKernel

In linux kernel 6.14 they fixed the issue regarding SMB encryption caching that was causing "ENOMEM" in Xen Orchestra on my end.
Linux kernel 6.17 from debian backports includes that fix.

Best regards

@rzr No I did not customize anything related to rsyslog on XO / XCP-ng side. No tweaking of rsyslog config file(s) or similar.
I just setup a graylog server, enabled the syslog udp input there and configured rsyslog via Xen Orchestra like so:

I consider this the most basic setup.
Also: AFAIK there are no UI options for rsyslog custom rules. Enabling it and setting IP address/port is basically all that can be done. I read that Vates recommends to not fiddle with config files on dom0 and view XCP-ng as an appliance.
So maybe custom rsyslog config does not need to get considered as long as no new GUI features for that are planned.

@stormi I have another XCP-ng pool running in our test environment / lab which does not (yet) have that round of patches applied.
I will try to reproduce the issue in that environment once I have the time to do so and let you know.

And sure, if it happens again I will save the contents of the rsyslog.conf file.

yum.log is empty.
yum.log.1 contains:

[19:01 xcpng01 log]# cat yum.log.1
Dec 19 17:02:14 Updated: xen-libs-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:14 Updated: xcp-ng-release-presets-8.3.0-35.x86_64
Dec 19 17:02:16 Updated: xcp-ng-release-config-8.3.0-35.x86_64
Dec 19 17:02:17 Updated: xen-hypervisor-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:17 Updated: xen-dom0-libs-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:17 Updated: vhd-tool-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:22 Updated: 2:qemu-4.2.1-5.2.15.1.xcpng8.3.x86_64
Dec 19 17:02:23 Updated: xen-tools-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:23 Updated: xen-dom0-tools-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: forkexecd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: qcow-stream-tool-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: varstored-guard-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:25 Updated: sm-fairlock-3.2.12-16.1.xcpng8.3.x86_64
Dec 19 17:02:26 Updated: sm-3.2.12-16.1.xcpng8.3.x86_64
Dec 19 17:02:26 Updated: message-switch-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:27 Updated: xenopsd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:27 Updated: xapi-rrd2csv-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:28 Updated: rrdd-plugins-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:28 Updated: xenopsd-cli-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:29 Updated: xenopsd-xc-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:33 Updated: xapi-core-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:34 Updated: varstored-1.2.0-3.4.xcpng8.3.x86_64
Dec 19 17:02:34 Updated: xapi-tests-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: squeezed-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: xcp-rrdd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: xcp-ng-release-8.3.0-35.x86_64
Dec 19 17:02:36 Updated: xapi-storage-script-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:36 Updated: gpumon-24.1.0-71.1.xcpng8.3.x86_64
Dec 19 17:02:36 Updated: xsconsole-11.0.9.1-1.1.xcpng8.3.x86_64
Dec 19 17:02:40 Updated: xcp-ng-pv-tools-8.3-15.xcpng8.3.noarch
Dec 19 17:02:40 Updated: amd-microcode-20251203-1.1.xcpng8.3.noarch
Dec 19 17:02:40 Updated: sm-cli-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xcp-networkd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: varstored-tools-1.2.0-3.4.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xapi-xe-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: wsproxy-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xo-lite-0.17.0-1.xcpng8.3.noarch
Dec 19 17:02:43 Updated: xcp-featured-1.1.8-3.xcpng8.3.x86_64
Dec 19 17:02:43 Updated: xha-25.2.0-1.1.xcpng8.3.x86_64
Dec 19 17:02:43 Updated: xapi-nbd-25.33.1-2.1.xcpng8.3.x86_64

@stormi Well I only reboot our XCP-ng hosts after updates have been applied. I configured remote syslog at the beginning of december as an attempt to fix /var/log partition reaching 100& usage (as described in this thread).

Remote syslog was working fine at that point.

When you guys released the december round of patches I applied them and as a result rebooted all hosts of the pool.
After checking our graylog server I can confirm that the XCP-ng pool stopped sending remote syslog data after the hosts had been rebooted.

I then searched the forum, found this thread, was able to get remote syslog working again by re-applying the remote syslog IP addresses via XO as described by other users some time ago.

Due to the fact that the behavior of the systems looked exactly as what had been described in this thread earlier I assumed that the issue may not have been investigated / fixed yet.

//EDIT: regarding the question of package versions: I applied everything that you guys released in the december round of pachtes and had a patched system prior you releasing them.
I can not say if the "yum upgrade" or reboot is the exact moment where the remote syslog stopped working.

@florent I think you meant it was locking the xoa via .tar.gz but not via .zip which seemed to be the case

Yeah I just decompressed the restored .zip and checked number of files and size via ncdu, result: "Total disk usage: 194,5 MiB Apparent size: 193,1 MiB Items: 613"

In case I can provide more information for investigating this issue, please let me know.

Thanks and best regards

Hello XCP-ng community and Vates-Team,

I just observed a weird behavior of Xen Orchestra during backup file restore.

Background: I had to restore a directory that got deleted on a small file server Windows VM by accident.

I used Xen Orchestra's file restore menu to select the VM, restore point and path of the directory in question.
Initially I selected .tar.gz as export format and started the restore process.
A new browser tab opened and after a few minutes it showed "Error proxying request".
Then Xen Orchestra became almost fully unresponsive for like 5min but started to behave normal again after said time.

I then tried the same thing again: selected same VM, restore point, path etc. but this time opted for ".zip (slow)" option as export format.
That worked without any issues. Download started after like 5 seconds, no issues whatsoever.

Did somebody else encounter similar issues?
Maybe the .tar.gz functionality of Xen Orchestra needs investigation.
Just wanted to report this issue and ask if maybe somebody else encountered it.

Thanks and best regards

//EDIT: oh forgot to mention: I am running a fully patched XCP-ng 8.3 pool and latest XO CE on a Debian 13 VM. NodeJS version is 24 LTS.

@Mang0Musztarda said in Xen Orchestra OpenMetrics Plugin - Grafana Dashboard:

@MajorP93 hi, how can i scrape openmetrics endpoint?
i set up openmetrics plugin prometheus secret, enabled it, and ten tried to use curl like that: curl -H "Authorization: Bearer abc123" http://localhost:9004
but response i got was
{"error":"Query authentication does not match server setting"}
what am i doing wrong?

Hey!
I scrape it like so:

root@prometheus01:~# cat /etc/prometheus/scrape_configs/xen-orchestra-openmetrics.yml 
scrape_configs:
  - job_name: xen-orchestra
    honor_labels: true
    scrape_interval: 30s
    scrape_timeout: 20s
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/bearer.token
    metrics_path: /openmetrics/metrics
    static_configs:
    - targets:
      - xen-orchestra.domain.local

/etc/prometheus/bearer.token file contains the bearer token as configured in openmetrics xen orchestra plugin.

best regards

Hi,
unfortunately the problem still exists (fully patched XCP-ng 8.3 pool).
After rebooting the hosts remote syslog did not work anymore (no incoming data on graylog server).
After setting remote syslog to a different IP address in Xen Orchestra and switching it back to the correct IP address again solves the issue but remote syslog should survive a host reboot...
Best regards