Remote syslog broken after update/reboot? - Changing it away, then back fixes.

michmoor0725

@Tackyone
You solution fixed it for me. Lost a few days of logs as well and i am on the latest commit of the XO community edition.

Theres some bug with syslog clear but your fix works great.

MajorP93

Hi,
unfortunately the problem still exists (fully patched XCP-ng 8.3 pool).
After rebooting the hosts remote syslog did not work anymore (no incoming data on graylog server).
After setting remote syslog to a different IP address in Xen Orchestra and switching it back to the correct IP address again solves the issue but remote syslog should survive a host reboot...
Best regards

olivierlambert

That's weird. Maybe @Team-XAPI-amp-Network could tell you the commands to check what's recorded in XAPI to see if it's a problem at XAPI or XO doing things weirdly.

olivierlambert

Bad ping: it's @Team-XAPI-Network

psafont

@olivierlambert This was fixed by xenserver last year: https://github.com/xapi-project/xen-api/pull/6328

For XCP-ng, the team decided they didn't like the solution, and there's work on a better solution, but it's not ready yet

GeraldEV opened this pull request in xapi-project/xen-api

closed CA-407370: Use remote.conf for customer rsyslog forwarding rules #6328

olivierlambert

Good to know, maybe it's time to finish that at some point, let me ping @gregoire

stormi

We paused that update because it was likely to cause new problems the way the transition is handled. We at least needed time to ponder it. @rzr is on it among other things, and it's less important than, say, upgrading to openssl 3.

There's no need for PM intervention here IMO.

Moreover, I just checked again, none of the 8.3 updates is supposed to be overwriting /etc/rsyslog.conf, where our XAPI currently writes the remote configuration. But we need to be ready the next time we want to make changes to that file.

So I don't understand what led @majorp93 to losing their configuration. Could you describe what you updated, from which version, and what you identify as the moment when the configuration was removed?

MajorP93

@stormi Well I only reboot our XCP-ng hosts after updates have been applied. I configured remote syslog at the beginning of december as an attempt to fix /var/log partition reaching 100& usage (as described in this thread).

Remote syslog was working fine at that point.

When you guys released the december round of patches I applied them and as a result rebooted all hosts of the pool.
After checking our graylog server I can confirm that the XCP-ng pool stopped sending remote syslog data after the hosts had been rebooted.

I then searched the forum, found this thread, was able to get remote syslog working again by re-applying the remote syslog IP addresses via XO as described by other users some time ago.

Due to the fact that the behavior of the systems looked exactly as what had been described in this thread earlier I assumed that the issue may not have been investigated / fixed yet.

//EDIT: regarding the question of package versions: I applied everything that you guys released in the december round of pachtes and had a patched system prior you releasing them.
I can not say if the "yum upgrade" or reboot is the exact moment where the remote syslog stopped working.

stormi

@MajorP93 If it happens again for you (or anyone else reading this thread), please save the contents of /etc/rsyslog.conf just after the lost remote syslog so that we may check whether it was overwritten or something else happened.

Also, could you upload somewhere the file that contains your yum logs? /var/log/yum.log or any rotated version of that file /var/log/yum.log.1, etc.

MajorP93

@stormi I have another XCP-ng pool running in our test environment / lab which does not (yet) have that round of patches applied.
I will try to reproduce the issue in that environment once I have the time to do so and let you know.

And sure, if it happens again I will save the contents of the rsyslog.conf file.

yum.log is empty.
yum.log.1 contains:

[19:01 xcpng01 log]# cat yum.log.1
Dec 19 17:02:14 Updated: xen-libs-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:14 Updated: xcp-ng-release-presets-8.3.0-35.x86_64
Dec 19 17:02:16 Updated: xcp-ng-release-config-8.3.0-35.x86_64
Dec 19 17:02:17 Updated: xen-hypervisor-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:17 Updated: xen-dom0-libs-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:17 Updated: vhd-tool-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:22 Updated: 2:qemu-4.2.1-5.2.15.1.xcpng8.3.x86_64
Dec 19 17:02:23 Updated: xen-tools-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:23 Updated: xen-dom0-tools-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: forkexecd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: qcow-stream-tool-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: varstored-guard-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:25 Updated: sm-fairlock-3.2.12-16.1.xcpng8.3.x86_64
Dec 19 17:02:26 Updated: sm-3.2.12-16.1.xcpng8.3.x86_64
Dec 19 17:02:26 Updated: message-switch-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:27 Updated: xenopsd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:27 Updated: xapi-rrd2csv-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:28 Updated: rrdd-plugins-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:28 Updated: xenopsd-cli-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:29 Updated: xenopsd-xc-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:33 Updated: xapi-core-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:34 Updated: varstored-1.2.0-3.4.xcpng8.3.x86_64
Dec 19 17:02:34 Updated: xapi-tests-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: squeezed-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: xcp-rrdd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: xcp-ng-release-8.3.0-35.x86_64
Dec 19 17:02:36 Updated: xapi-storage-script-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:36 Updated: gpumon-24.1.0-71.1.xcpng8.3.x86_64
Dec 19 17:02:36 Updated: xsconsole-11.0.9.1-1.1.xcpng8.3.x86_64
Dec 19 17:02:40 Updated: xcp-ng-pv-tools-8.3-15.xcpng8.3.noarch
Dec 19 17:02:40 Updated: amd-microcode-20251203-1.1.xcpng8.3.noarch
Dec 19 17:02:40 Updated: sm-cli-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xcp-networkd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: varstored-tools-1.2.0-3.4.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xapi-xe-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: wsproxy-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xo-lite-0.17.0-1.xcpng8.3.noarch
Dec 19 17:02:43 Updated: xcp-featured-1.1.8-3.xcpng8.3.x86_64
Dec 19 17:02:43 Updated: xha-25.2.0-1.1.xcpng8.3.x86_64
Dec 19 17:02:43 Updated: xapi-nbd-25.33.1-2.1.xcpng8.3.x86_64

rzr

hi, yes I tried to play and tweak rsyslog recently (there are some changes in xcp-ng-rpm github), but I was unsure about customizations options.

Are you or anyone using rsyslog with customs rules ? if yes let me know how, it can help for our testing.

MajorP93

@rzr No I did not customize anything related to rsyslog on XO / XCP-ng side. No tweaking of rsyslog config file(s) or similar.
I just setup a graylog server, enabled the syslog udp input there and configured rsyslog via Xen Orchestra like so:

I consider this the most basic setup.
Also: AFAIK there are no UI options for rsyslog custom rules. Enabling it and setting IP address/port is basically all that can be done. I read that Vates recommends to not fiddle with config files on dom0 and view XCP-ng as an appliance.
Therefore I don't think that custom rules need to be considered at this point.