XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Remote syslog broken after update/reboot? - Changing it away, then back fixes.

    Scheduled Pinned Locked Moved Compute
    13 Posts 7 Posters 744 Views 7 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      That's weird. Maybe @Team-XAPI-amp-Network could tell you the commands to check what's recorded in XAPI to see if it's a problem at XAPI or XO doing things weirdly.

      1 Reply Last reply Reply Quote 1
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        Bad ping: it's @Team-XAPI-Network

        psafontP 1 Reply Last reply Reply Quote 1
        • psafontP Offline
          psafont Vates 🪐 XAPI & Network Team @olivierlambert
          last edited by psafont

          @olivierlambert This was fixed by xenserver last year: https://github.com/xapi-project/xen-api/pull/6328

          For XCP-ng, the team decided they didn't like the solution, and there's work on a better solution, but it's not ready yet

          GeraldEV opened this pull request in xapi-project/xen-api

          closed CA-407370: Use remote.conf for customer rsyslog forwarding rules #6328

          1 Reply Last reply Reply Quote 2
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Good to know, maybe it's time to finish that at some point, let me ping @gregoire

            1 Reply Last reply Reply Quote 0
            • stormiS Offline
              stormi Vates 🪐 XCP-ng Team
              last edited by

              We paused that update because it was likely to cause new problems the way the transition is handled. We at least needed time to ponder it. @rzr is on it among other things, and it's less important than, say, upgrading to openssl 3.

              There's no need for PM intervention here IMO.

              Moreover, I just checked again, none of the 8.3 updates is supposed to be overwriting /etc/rsyslog.conf, where our XAPI currently writes the remote configuration. But we need to be ready the next time we want to make changes to that file.

              So I don't understand what led @majorp93 to losing their configuration. Could you describe what you updated, from which version, and what you identify as the moment when the configuration was removed?

              M 1 Reply Last reply Reply Quote 1
              • M Offline
                MajorP93 @stormi
                last edited by MajorP93

                @stormi Well I only reboot our XCP-ng hosts after updates have been applied. I configured remote syslog at the beginning of december as an attempt to fix /var/log partition reaching 100& usage (as described in this thread).

                Remote syslog was working fine at that point.

                When you guys released the december round of patches I applied them and as a result rebooted all hosts of the pool.
                After checking our graylog server I can confirm that the XCP-ng pool stopped sending remote syslog data after the hosts had been rebooted.

                I then searched the forum, found this thread, was able to get remote syslog working again by re-applying the remote syslog IP addresses via XO as described by other users some time ago.

                Due to the fact that the behavior of the systems looked exactly as what had been described in this thread earlier I assumed that the issue may not have been investigated / fixed yet.

                //EDIT: regarding the question of package versions: I applied everything that you guys released in the december round of pachtes and had a patched system prior you releasing them.
                I can not say if the "yum upgrade" or reboot is the exact moment where the remote syslog stopped working.

                stormiS 1 Reply Last reply Reply Quote 0
                • stormiS Offline
                  stormi Vates 🪐 XCP-ng Team @MajorP93
                  last edited by stormi

                  @MajorP93 If it happens again for you (or anyone else reading this thread), please save the contents of /etc/rsyslog.conf just after the lost remote syslog so that we may check whether it was overwritten or something else happened.

                  Also, could you upload somewhere the file that contains your yum logs? /var/log/yum.log or any rotated version of that file /var/log/yum.log.1, etc.

                  M 1 Reply Last reply Reply Quote 0
                  • M Offline
                    MajorP93 @stormi
                    last edited by MajorP93

                    @stormi I have another XCP-ng pool running in our test environment / lab which does not (yet) have that round of patches applied.
                    I will try to reproduce the issue in that environment once I have the time to do so and let you know.

                    And sure, if it happens again I will save the contents of the rsyslog.conf file.

                    yum.log is empty.
                    yum.log.1 contains:

                    [19:01 xcpng01 log]# cat yum.log.1
Dec 19 17:02:14 Updated: xen-libs-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:14 Updated: xcp-ng-release-presets-8.3.0-35.x86_64
Dec 19 17:02:16 Updated: xcp-ng-release-config-8.3.0-35.x86_64
Dec 19 17:02:17 Updated: xen-hypervisor-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:17 Updated: xen-dom0-libs-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:17 Updated: vhd-tool-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:22 Updated: 2:qemu-4.2.1-5.2.15.1.xcpng8.3.x86_64
Dec 19 17:02:23 Updated: xen-tools-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:23 Updated: xen-dom0-tools-4.17.5-23.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: forkexecd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: qcow-stream-tool-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:24 Updated: varstored-guard-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:25 Updated: sm-fairlock-3.2.12-16.1.xcpng8.3.x86_64
Dec 19 17:02:26 Updated: sm-3.2.12-16.1.xcpng8.3.x86_64
Dec 19 17:02:26 Updated: message-switch-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:27 Updated: xenopsd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:27 Updated: xapi-rrd2csv-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:28 Updated: rrdd-plugins-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:28 Updated: xenopsd-cli-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:29 Updated: xenopsd-xc-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:33 Updated: xapi-core-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:34 Updated: varstored-1.2.0-3.4.xcpng8.3.x86_64
Dec 19 17:02:34 Updated: xapi-tests-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: squeezed-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: xcp-rrdd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:35 Updated: xcp-ng-release-8.3.0-35.x86_64
Dec 19 17:02:36 Updated: xapi-storage-script-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:36 Updated: gpumon-24.1.0-71.1.xcpng8.3.x86_64
Dec 19 17:02:36 Updated: xsconsole-11.0.9.1-1.1.xcpng8.3.x86_64
Dec 19 17:02:40 Updated: xcp-ng-pv-tools-8.3-15.xcpng8.3.noarch
Dec 19 17:02:40 Updated: amd-microcode-20251203-1.1.xcpng8.3.noarch
Dec 19 17:02:40 Updated: sm-cli-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xcp-networkd-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: varstored-tools-1.2.0-3.4.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xapi-xe-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: wsproxy-25.33.1-2.1.xcpng8.3.x86_64
Dec 19 17:02:42 Updated: xo-lite-0.17.0-1.xcpng8.3.noarch
Dec 19 17:02:43 Updated: xcp-featured-1.1.8-3.xcpng8.3.x86_64
Dec 19 17:02:43 Updated: xha-25.2.0-1.1.xcpng8.3.x86_64
Dec 19 17:02:43 Updated: xapi-nbd-25.33.1-2.1.xcpng8.3.x86_64
                    1 Reply Last reply Reply Quote 0
                    • rzrR Offline
                      rzr
                      last edited by

                      hi, yes I tried to play and tweak rsyslog recently (there are some changes in xcp-ng-rpm github), but I was unsure about customizations options.

                      Are you or anyone using rsyslog with customs rules ? if yes let me know how, it can help for our testing.

                      M 1 Reply Last reply Reply Quote 0
                      • M Offline
                        MajorP93 @rzr
                        last edited by MajorP93

                        @rzr No I did not customize anything related to rsyslog on XO / XCP-ng side. No tweaking of rsyslog config file(s) or similar.
                        I just setup a graylog server, enabled the syslog udp input there and configured rsyslog via Xen Orchestra like so:

                        1b446ebe-7bee-492b-aacf-43531e7d050c-grafik.png

                        I consider this the most basic setup.
                        Also: AFAIK there are no UI options for rsyslog custom rules. Enabling it and setting IP address/port is basically all that can be done. I read that Vates recommends to not fiddle with config files on dom0 and view XCP-ng as an appliance.
                        So maybe custom rsyslog config does not need to get considered as long as no new GUI features for that are planned.

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post