XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    backup mail report says INTERRUPTED but it's not ?

    Scheduled Pinned Locked Moved Backup
    69 Posts 7 Posters 2.5k Views 9 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • olivierlambertO Offline
      olivierlambert Vates 🪐 Co-Founder CEO
      last edited by

      Okay weird, so having a very different RAM usage with Node 24 should be checked when we'll update XOA Node version.

      1 Reply Last reply Reply Quote 0
      • P Online
        Pilow @MajorP93
        last edited by

        @MajorP93 this was found in the troubleshooting section of the documentation when i tried to optimise my xoa/xoproxies deployments

        https://docs.xen-orchestra.com/troubleshooting#memory

        P 1 Reply Last reply Reply Quote 1
        • P Online
          Pilow @Pilow
          last edited by

          not looking better today. still not crashed.
          62622e51-a19c-43e3-b499-a6fde4124eb1-{093F1189-D681-4A45-B0CA-FC72B6E08E9B}.png
          but before the patches at 48h max, i topped the 8Gb and got the OOM killed process.

          P 1 Reply Last reply Reply Quote 0
          • P Online
            Pilow @Pilow
            last edited by

            before disruption, I prefer to patch/reboot my XOA
            We got to the limit
            75ebf8f1-42a3-41a4-8179-bcad56b507fb-{BD0DCE60-47E3-4386-AB7E-1BC85D08DE1F}.png
            505f9643-9c4b-4c22-982a-f48ed42798c4-{075C1E52-0AF8-495A-B8E5-372F0C4953E4}.png

            still some memory leak somewhere guys !

            florentF 1 Reply Last reply Reply Quote 0
            • florentF Offline
              florent Vates 🪐 XO Team @Pilow
              last edited by

              @Pilow we are still working on it, but for now we didn't find a solution

              M 1 Reply Last reply Reply Quote 1
              • M Offline
                MajorP93 @florent
                last edited by MajorP93

                After implementing the --max-old-space-size Node parameter as recommended by @pilow it took longer time for the VM to hit the issue.
                Still: backups went into interrupted status.
                Memory leak seems to be still there.
                With each subsequent backup run the memory usage rises and rises. After backup run the memory usage does not fully go back to "normal".

                6ad321a1-2e39-4bca-9285-062e502a17b2-grafik.png

                After adding the node parameter there was no heap size error on Node anymore since the heap size got increased. The system went into various OOM errors in kernel log (dmesg) despite not all RAM (8GB) being used.

                This is what htop looks like with 3 backup jobs running:
                68db77eb-26f2-4dbb-b1db-2273984eabb3-grafik.png

                1 Reply Last reply Reply Quote 0
                • A Offline
                  acebmxer
                  last edited by acebmxer

                  While working last night i noticed one of my backups/pools did this. Got the email that it was interupted but when i looked the tasks were still running and moving data it untill it porcess all vms in that backup job.

                  Edit - note my backup job was run via proxy on the specific pool/job.

                  2026-02-19T03_00_00.028Z - backup NG.txt

                  Edit 2 - homelab same last backup was interupted.

                  2026-02-19T05_00_00.011Z - backup NG.txt

                  M 1 Reply Last reply Reply Quote 0
                  • M Offline
                    MajorP93 @acebmxer
                    last edited by MajorP93

                    I wonder if this PR https://github.com/vatesfr/xen-orchestra/pull/9506 aims to solve the issue that was discussed in this thread.
                    To me it looks like it's the case as the issue seems to be related to RAM used by backup jobs not being freed correctly and the PR seems to add some garbage collection to backup jobs.
                    I hope that it will fix the issue and if needed I can test a branch.

                    b-Nollet opened this pull request in vatesfr/xen-orchestra

                    open Backup tasks gc #9506

                    Bastien NolletB 1 Reply Last reply Reply Quote 0
                    • Bastien NolletB Offline
                      Bastien Nollet Vates 🪐 XO Team @MajorP93
                      last edited by

                      Hi @MajorP93,

                      This PR is only about changing the way we delete old logs (linked to a bigger work of making backups use XO tasks instead of their own task system), it won't fix the issue discussed in this topic.

                      M 1 Reply Last reply Reply Quote 0
                      • M Offline
                        MajorP93 @Bastien Nollet
                        last edited by

                        Hi @Bastien-Nollet,

                        oh okay, thanks for clarifying!

                        1 Reply Last reply Reply Quote 0
                        • First post
                          Last post