XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Remote syslog broken after update/reboot? - Changing it away, then back fixes.

    Scheduled Pinned Locked Moved Compute
    15 Posts 8 Posters 773 Views 8 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • psafontP Online
      psafont Vates 🪐 XAPI & Network Team @olivierlambert
      last edited by psafont

      @olivierlambert This was fixed by xenserver last year: https://github.com/xapi-project/xen-api/pull/6328

      For XCP-ng, the team decided they didn't like the solution, and there's work on a better solution, but it's not ready yet

      GeraldEV opened this pull request in xapi-project/xen-api

      closed CA-407370: Use remote.conf for customer rsyslog forwarding rules #6328

      1 Reply Last reply Reply Quote 2
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Good to know, maybe it's time to finish that at some point, let me ping @gregoire

        1 Reply Last reply Reply Quote 0
        • stormiS Online
          stormi Vates 🪐 XCP-ng Team
          last edited by

          We paused that update because it was likely to cause new problems the way the transition is handled. We at least needed time to ponder it. @rzr is on it among other things, and it's less important than, say, upgrading to openssl 3.

          There's no need for PM intervention here IMO.

          Moreover, I just checked again, none of the 8.3 updates is supposed to be overwriting /etc/rsyslog.conf, where our XAPI currently writes the remote configuration. But we need to be ready the next time we want to make changes to that file.

          So I don't understand what led @majorp93 to losing their configuration. Could you describe what you updated, from which version, and what you identify as the moment when the configuration was removed?

          M 1 Reply Last reply Reply Quote 1
          • M Online
            MajorP93 @stormi
            last edited by MajorP93

            @stormi Well I only reboot our XCP-ng hosts after updates have been applied. I configured remote syslog at the beginning of december as an attempt to fix /var/log partition reaching 100& usage (as described in this thread).

            Remote syslog was working fine at that point.

            When you guys released the december round of patches I applied them and as a result rebooted all hosts of the pool.
            After checking our graylog server I can confirm that the XCP-ng pool stopped sending remote syslog data after the hosts had been rebooted.

            I then searched the forum, found this thread, was able to get remote syslog working again by re-applying the remote syslog IP addresses via XO as described by other users some time ago.

            Due to the fact that the behavior of the systems looked exactly as what had been described in this thread earlier I assumed that the issue may not have been investigated / fixed yet.

            //EDIT: regarding the question of package versions: I applied everything that you guys released in the december round of pachtes and had a patched system prior you releasing them.
            I can not say if the "yum upgrade" or reboot is the exact moment where the remote syslog stopped working.

            stormiS 1 Reply Last reply Reply Quote 0
            • stormiS Online
              stormi Vates 🪐 XCP-ng Team @MajorP93
              last edited by stormi

              @MajorP93 If it happens again for you (or anyone else reading this thread), please save the contents of /etc/rsyslog.conf just after the lost remote syslog so that we may check whether it was overwritten or something else happened.

              Also, could you upload somewhere the file that contains your yum logs? /var/log/yum.log or any rotated version of that file /var/log/yum.log.1, etc.

              M 1 Reply Last reply Reply Quote 0
              • M Online
                MajorP93 @stormi
                last edited by MajorP93

                @stormi I have another XCP-ng pool running in our test environment / lab which does not (yet) have that round of patches applied.
                I will try to reproduce the issue in that environment once I have the time to do so and let you know.

                And sure, if it happens again I will save the contents of the rsyslog.conf file.

                yum.log is empty.
                yum.log.1 contains:

                [19:01 xcpng01 log]# cat yum.log.1
Dec 19 17:02:14 Updated: xen-libs-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:14 Updated: xcp-ng-release-presets-8.3.0-35.x86_64
Dec 19 17:02:16 Updated: xcp-ng-release-config-8.3.0-35.x86_64
Dec 19 17:02:17 Updated: xen-hypervisor-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:17 Updated: xen-dom0-libs-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:17 Updated: vhd-tool-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:22 Updated: 2:qemu-4.2.1-5.2.15.1.xcpng8.3.x86_64
Dec 19 17:02:23 Updated: xen-tools-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:23 Updated: xen-dom0-tools-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: forkexecd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: qcow-stream-tool-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: varstored-guard-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:25 Updated: sm-fairlock-3.2.12-16.1.xcpng8.3.x86_64
Dec 19 17:02:26 Updated: sm-3.2.12-16.1.xcpng8.3.x86_64
Dec 19 17:02:26 Updated: message-switch-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:27 Updated: xenopsd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:27 Updated: xapi-rrd2csv-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:28 Updated: rrdd-plugins-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:28 Updated: xenopsd-cli-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:29 Updated: xenopsd-xc-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:33 Updated: xapi-core-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:34 Updated: varstored-1.2.0-3.4.xcpng8.3.x86_64
Dec 19 17:02:34 Updated: xapi-tests-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: squeezed-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: xcp-rrdd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: xcp-ng-release-8.3.0-35.x86_64
Dec 19 17:02:36 Updated: xapi-storage-script-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:36 Updated: gpumon-24.1.0-71.1.xcpng8.3.x86_64
Dec 19 17:02:36 Updated: xsconsole-11.0.9.1-1.1.xcpng8.3.x86_64
Dec 19 17:02:40 Updated: xcp-ng-pv-tools-8.3-15.xcpng8.3.noarch
Dec 19 17:02:40 Updated: amd-microcode-20251203-1.1.xcpng8.3.noarch
Dec 19 17:02:40 Updated: sm-cli-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xcp-networkd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: varstored-tools-1.2.0-3.4.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xapi-xe-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: wsproxy-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xo-lite-0.17.0-1.xcpng8.3.noarch
Dec 19 17:02:43 Updated: xcp-featured-1.1.8-3.xcpng8.3.x86_64
Dec 19 17:02:43 Updated: xha-25.2.0-1.1.xcpng8.3.x86_64
Dec 19 17:02:43 Updated: xapi-nbd-25.33.1-2.1.xcpng8.3.x86_64
                stormiS 1 Reply Last reply Reply Quote 0
                • rzrR Online
                  rzr
                  last edited by

                  hi, yes I tried to play and tweak rsyslog recently (there are some changes in xcp-ng-rpm github), but I was unsure about customizations options.

                  Are you or anyone using rsyslog with customs rules ? if yes let me know how, it can help for our testing.

                  M 1 Reply Last reply Reply Quote 0
                  • M Online
                    MajorP93 @rzr
                    last edited by MajorP93

                    @rzr No I did not customize anything related to rsyslog on XO / XCP-ng side. No tweaking of rsyslog config file(s) or similar.
                    I just setup a graylog server, enabled the syslog udp input there and configured rsyslog via Xen Orchestra like so:

                    1b446ebe-7bee-492b-aacf-43531e7d050c-grafik.png

                    I consider this the most basic setup.
                    Also: AFAIK there are no UI options for rsyslog custom rules. Enabling it and setting IP address/port is basically all that can be done. I read that Vates recommends to not fiddle with config files on dom0 and view XCP-ng as an appliance.
                    So maybe custom rsyslog config does not need to get considered as long as no new GUI features for that are planned.

                    1 Reply Last reply Reply Quote 0
                    • ForzaF Offline
                      Forza
                      last edited by

                      I've been considering remote syslog too. Does enabling remote syslog remove local logging?

                      1 Reply Last reply Reply Quote 0
                      • stormiS Online
                        stormi Vates 🪐 XCP-ng Team @MajorP93
                        last edited by

                        @MajorP93 Good to know that you have another pool to update. Then please also save the file prior to updating.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post