XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    [HELP] XCP-ng 4.17.5 dom0 kernel panic — page fault in TCP stack, crashdump attached

    Scheduled Pinned Locked Moved XCP-ng
    25 Posts 6 Posters 636 Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      Yeah I would avoid those shitty cards as possible. Is there any more recent driver anyway?

      A 1 Reply Last reply Reply Quote 0
      • A Offline
        Andrew Top contributor @olivierlambert
        last edited by

        @olivierlambert There are new versions of the r812x drivers. They don't compile cleanly for XCP. The r8127 driver was withdrawn and the r8125/r8126 was split into two different drivers. Realtek never publishes release notes.

        I'll have to test the new driver and see if it's worth trying. I don't know if and update would solve this panic issue as there are lots of undocumented code changes.

        The forum has been quiet about new r8125 issues, so in general the current driver has been working well enough. Just two issues I remember, including this one.

        Realtek has also released new hardware revisions of the r812x chips that need new driver support and are only recently supported by their vendor driver and in upstream Linux 6.15 (not even 6.12 LTS yet).

        As for the r8127, it looks like it could be a desktop game changer as it's a small cheap low power 10G chip. But like the others, its release is delayed and it does not have driver support yet (or test samples).

        1 Reply Last reply Reply Quote 1
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          That would be interesting if a we have a test driver solving this very issue here, but I wouldn't expect too much either 😕 Those drivers are as bad as the chips 😢

          D 1 Reply Last reply Reply Quote 0
          • A Offline
            Andrew Top contributor @dnikola
            last edited by

            @dnikola Please make sure your motherboard firmware is up to date (BIOS F30e). There are a LOT of stability issues with Intel CPUs for that board and old BIOS.

            If you still have r8125 crashes, then try a newer r8125 alt version (9.016.00) from my download page and see if it works better. I gave it a quick test and it installs and works, but YMMV... You can always uninstall it.

            1 Reply Last reply Reply Quote 0
            • A Offline
              Andrew Top contributor @dnikola
              last edited by

              @dnikola As for the other card you listed, no, it's still a 8125 card. The single port 10G card (from the same site) is a AQC113 chipset, you'll need to install the atlantic-module-alt to support it. If you must have 2.5G then the Intel i225/i226 card is the other choice (not from that site).

              D 1 Reply Last reply Reply Quote 0
              • D Offline
                dnikola @Andrew
                last edited by dnikola

                Hi, thanks for your kind replay.

                Let me share what i have noticed
                3 servers with same hardware, same XCP-ng last version, same MBO bios..
                1 server different Hardware, same XCP-ng last version

                Server C
                not so frequent crashes but yes it happens from time to time around 10 days, and toolstack restarts few times in that 10 days

                5 VM: server 2022 x3, server 2019, win 10 pro
                delta backup disabled but connected to XO

                Server D
                2 VM: server 2019, server 2022
                delta backup enabled, everything running fine from first day, not any single problem restart or toolstack crash

                Server K
                5 VM: server 2019, server 2022, win 10 pro, win 7, linux
                make the most problems, it works for 10 days than restarts 10 times in 2 days... it was triggered after delta backups, so I have disabled delta backups and disabled sending metrics from server

                Server P
                5 VM: server 2019, server 2022 x 2, win 10 pro, win 7
                not so frequent crashes but yes it happens from time to time around 10 days, and toolstack restarts few times in that 10 days
                delta backup enabled, works fine, but few time restarts occurred in that time

                From reviewing dom0.log from server K as most affected one we have noticed:

                Multiple segfaults in xcp-rrdd throughout runtime:

                INFO: xcp-rrdd[xxx]: segfault at ...

                The RRD polling is active and seems unstable on this host.

                Frequent link down/up events from the r8125 driver:

                INFO: r8125: eth0: link down
INFO: r8125: eth0: link up

                (known issue on Xen hypervisors with Realtek drivers)

                And eventually, identical kernel panics as before:

                CRIT: kernel BUG at drivers/xen/events/events_base.c:1601!
Kernel panic - not syncing: Fatal exception in interrupt
Always same stack trace, same event channel handling failure.

                📌 Actions planned:
                BIOS update on-site (currently on v1663 / Aug 2024 — latest is 1854)
                Evaluate replacing the Realtek NIC with an Intel one
                Problem is that the server is at a remote location, and we’re organizing an on-site intervention ASAP.

                📌 In the meantime:
                Can I safely disable xcp-rrdd service to reduce polling activity?
                I know it powers the RRD stats in XO and XenCenter, but we can live without the graphs for now.

                Is there anything else advisable to disable / adjust until we get on-site?
                (delta backups are already paused on this)

                The VM involved during the latest crash was a FreePBX virtual machine running management agent version 8.4.
                Is there a newer agent package available for CentOS/AlmaLinux 8/9 guests I should apply?

                📌 Question:

                • Would disabling xcp-rrdd mitigate dom0 instability short-term?
                • Is there any way to tune RRD polling frequency instead of disabling entirely?
                • Anything else to collect before the next crash (besides xen-bugtool -y) you’d recommend?

                I also noticed that my FreePBX VM (UUID: 6c725208-c266-a106-da10-50e9ec66b41e) repeatedly triggers an event processing loop via xenopsd-xc and xapi, visible both in dom0.log and xapi.log.

                Example from logs:

                Received an event on managed VM 6c725208-c266-a106-da10-50e9ec66b41e
Queue.push ["VM_check_state","6c725208-c266-a106-da10-50e9ec66b41e"]
Queue.pop returned ["VM_check_state","6c725208-c266-a106-da10-50e9ec66b41e"]
VM 6c725208-c266-a106-da10-50e9ec66b41e is not requesting any attention

                This repeats every minute, without an actual task being created (confirmed via xe task-list showing no pending tasks).

                Notably:

                • This behavior persists even after disabling RRD polling and delta backups
                • The VM shows an orange activity indicator in XCP-ng Admin Center, as if a task is ongoing
                • Previously this has caused a dom0 crash and reboot
                • Given the log pattern and event storm, it seems likely that either:
                • A stale or looping event is being triggered by the guest agent or hypervisor integration
                • Or xenopsd/xapi state machine isn't properly clearing or marking the VM state after these checks

                I'd appreciate advice on:

                • How to safely clear/reset the VM state without restarting dom0
                • Whether updating the management agent inside the FreePBX guest (currently xcp-ng-agent 8.4) to a newer version might resolve this
                  (If a newer one is available for RHEL7/FreePBX)

                Part of log in time of this happening

                Thanks in advance — we’re pushing for the hardware fixes but would appreciate advice for short-term stability in the meantime.

                1 Reply Last reply Reply Quote 0
                • D Offline
                  dnikola @olivierlambert
                  last edited by

                  @Andrew said in [HELP] XCP-ng 4.17.5 dom0 kernel panic — page fault in TCP stack, crashdump attached:

                  @dnikola Please make sure your motherboard firmware is up to date (BIOS F30e). There are a LOT of stability issues with Intel CPUs for that board and old BIOS.

                  If you still have r8125 crashes, then try a newer r8125 alt version (9.016.00) from my download page and see if it works better. I gave it a quick test and it installs and works, but YMMV... You can always uninstall it.

                  Ok, that will be done and I will report!
                  is it possible to have some quick user guide what has to be done, in which order to process with correct install - uninstall process

                  @Andrew said in [HELP] XCP-ng 4.17.5 dom0 kernel panic — page fault in TCP stack, crashdump attached:

                  @dnikola As for the other card you listed, no, it's still a 8125 card. The single port 10G card (from the same site) is a AQC113 chipset, you'll need to install the atlantic-module-alt to support it. If you must have 2.5G then the Intel i225/i226 card is the other choice (not from that site).

                  I appreciate all your help so far — thank you. I noticed that the only 2.5G NIC currently available locally is the one with 8125, so it was "first aid", but didn't work . Since I’ll likely need to order a replacement online (not possible to find it in our country without purchase), could you kindly recommend a reliable source or a specific NIC model (our ISP is 2,5gbps so i prefer 2,5+ card) you’d personally suggest for this purpose (eBay or what every)?

                  Of course, this would be just an informal recommendation — I fully respect your experience and advice, and I completely understand it wouldn’t imply any obligation or responsibility on your part for any potential purchase issues or problems later.

                  my second option is MBO replacement with intel NIC (local wholesale have few models on stock) and it will be maybe fastest option

                  • ASUS PRIME Z790-A WIFI
                  • MSI PRO Z790-P WIFI
                  • GIGABYTE Z790 AERO G rev. 1.x
                  • MSI Z790 GAMING PLUS WIFI

                  Thanks again in advance — any tip would be much appreciated.

                  A 1 Reply Last reply Reply Quote 0
                  • A Offline
                    Andrew Top contributor @dnikola
                    last edited by

                    @dnikola The AQC113 10G card (from your vendor) also support 2.5G with the driver loaded.

                    D 1 Reply Last reply Reply Quote 0
                    • D Offline
                      dnikola @Andrew
                      last edited by

                      Hi everyone,

                      Just wanted to share a follow-up on the situation discussed earlier. The issues related to Realtek NIC and stability seem resolved for now.

                      Changes Made:

                      • Replaced onboard Realtek NIC with Intel X520 dual-port SFP+ card
                      • Installed and verified Ubiquiti UACC-DAC-SFP10-3M cable (working fine)
                      • Updated BIOS to the latest version: 1820 (May 2025)

                      The system seems stable now, but if you'd like to review the full dmesg output, I've uploaded it here: https://pastebin.com/6Mgt7Nir

                      Now that we have a working Intel NIC, what do you recommend regarding the onboard Realtek NIC?

                      Would it be best to:

                      • Disable it in BIOS?
                      • Leave it enabled but unused (no cable)?
                      • Leave it connected to a disabled port on the managed switch?

                      We’re currently leaning toward just removing it from all bridges and leaving it connected on disabled managed switch port — but we’d appreciate your thoughts on the cleanest/safest long-term solution.

                      Thanks again to @olivier and @Andrew for all the help so far.
                      If there’s any other suggestion you might have for additional tuning or validation now that we're stable, please let us know.

                      1 Reply Last reply Reply Quote 0
                      • bleaderB Offline
                        bleader Vates 🪐 XCP-ng Team
                        last edited by

                        I think whatever solution suits you will work.

                        Personally, if I know there are issues with it, I would tend to disable it in the bios, to be sure nobody tries to use it later and waste their time, in a enterprise settings, that can be important.

                        One thing to keep in mind if keeping it, is that if you want to add other hosts to the pool, they will need to have similar network topology, so if you endup having eth0 and eth1 with your current management network on eth1, any new host should be able to have its management on eth1 as well. You may work around it with interface renaming, but that tends to get messy over time.

                        That being said, I'm unsure even removing the realtek nic from the bios will change the interface number now that eth1 exists already and is configured.

                        If you don't plan to add hosts to the pool, and don't have a team with people that may act on these machines in the future without being aware of this setup history, leaving it connected and disabling the port on switch should not be an issue.

                        1 Reply Last reply Reply Quote 1
                        • First post
                          Last post