XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Memory Ballooning (DMC) broken since XCP-ng 8.3 January 2026 patches

    Scheduled Pinned Locked Moved Compute
    13 Posts 3 Posters 380 Views 4 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D Offline
      dinhngtu Vates đŸȘ XCP-ng Team @MajorP93
      last edited by

      @MajorP93 Thanks for the info, I'll need to discuss with the team first

      1 Reply Last reply Reply Quote 1
      • A Offline
        andriy.sultanov Vates đŸȘ XAPI & Network Team @MajorP93
        last edited by andriy.sultanov

        @MajorP93 Could you attach /var/log/xensource.log from the time of these issues on the destination host? We want to understand what squeezed is thinking and why it doesn't rebalance the memory.

        M 1 Reply Last reply Reply Quote 0
        • M Offline
          MajorP93 @andriy.sultanov
          last edited by

          @andriy.sultanov Hello! Had to use another VM as the one used previously is a production VM and had to be rebooted in order to get dynamic max again.

          This time I used a temporary test VM that I can use for these kind of tests.

          After live migrating this test VM, issue is there again.
          It is really easy to re-produce.

          9087099d-59e0-4cb9-9378-50d1d587c5ec-image.jpeg

          root@tmptest02:~# free -m
              gesamt       benutzt     frei      gemns.  Puffer/Cache verfĂŒgbar
Speicher:       3794         344        3511           3         109        3449
Swap:

          journalctl -k | cat

MĂ€r 11 14:26:19 tmptest02 kernel: Freezing user space processes
MĂ€r 11 14:26:23 tmptest02 kernel: Freezing user space processes completed (elapsed 0.018 seconds)
MĂ€r 11 14:26:23 tmptest02 kernel: OOM killer disabled.
MĂ€r 11 14:26:23 tmptest02 kernel: Freezing remaining freezable tasks
MĂ€r 11 14:26:23 tmptest02 kernel: Freezing remaining freezable tasks completed (elapsed 0.003 seconds)
MĂ€r 11 14:26:23 tmptest02 kernel: suspending xenstore...
MĂ€r 11 14:26:23 tmptest02 kernel: xen:grant_table: Grant tables using version 1 layout
MĂ€r 11 14:26:23 tmptest02 kernel: xen: --> irq=9, pirq=16
MĂ€r 11 14:26:23 tmptest02 kernel: xen: --> irq=8, pirq=17
MĂ€r 11 14:26:23 tmptest02 kernel: xen: --> irq=12, pirq=18
MĂ€r 11 14:26:23 tmptest02 kernel: xen: --> irq=1, pirq=19
MĂ€r 11 14:26:23 tmptest02 kernel: xen: --> irq=6, pirq=20
MĂ€r 11 14:26:23 tmptest02 kernel: xen: --> irq=4, pirq=21
MĂ€r 11 14:26:23 tmptest02 kernel: xen: --> irq=7, pirq=22
MĂ€r 11 14:26:23 tmptest02 kernel: xen: --> irq=23, pirq=23
MĂ€r 11 14:26:23 tmptest02 kernel: xen: --> irq=28, pirq=24
MĂ€r 11 14:26:23 tmptest02 kernel: usb usb1: root hub lost power or was reset
MĂ€r 11 14:26:23 tmptest02 kernel: ata2: found unknown device (class 0)
MĂ€r 11 14:26:23 tmptest02 kernel: usb 1-2: reset full-speed USB device number 2 using uhci_hcd
MĂ€r 11 14:26:23 tmptest02 kernel: OOM killer enabled.
MĂ€r 11 14:26:23 tmptest02 kernel: Restarting tasks: Starting
MĂ€r 11 14:26:23 tmptest02 kernel: Restarting tasks: Done
MĂ€r 11 14:26:23 tmptest02 kernel: Setting capacity to 125829120

          xensource.log excerpt on the target XCP-ng host that the VM got live migrated to:

          [14:35 xcpng02 log]# cat xensource.log | grep "Mar 11 14:26" | grep squeeze
Mar 11 14:26:06 xcpng02 squeezed: [debug||253 ||squeeze] total_range = 20971520 gamma = 1.000000 gamma' = 18.007186
Mar 11 14:26:06 xcpng02 squeezed: [debug||253 ||squeeze] Total additional memory over dynamic_min = 377638052 KiB; will set gamma = 1.00 (leaving unallocated 356666532 KiB)
Mar 11 14:26:06 xcpng02 squeezed: [debug||253 ||squeeze] free_memory_range ideal target = 4296680
Mar 11 14:26:06 xcpng02 squeezed: [debug||253 ||squeeze] change_host_free_memory required_mem = 4305896 KiB target_mem = 9216 KiB free_mem = 371880116 KiB
Mar 11 14:26:06 xcpng02 squeezed: [debug||253 ||squeeze] change_host_free_memory all VM target meet true
Mar 11 14:26:06 xcpng02 squeezed: [debug||253 ||memory] reserved 4296680 kib for reservation 6f0f8c43-7ffa-ffbf-7723-a1be3c1a61d1
Mar 11 14:26:06 xcpng02 squeezed: [debug||254 ||squeeze_xen] Xenctrl.domain_setmaxmem domid=53 max=4297704 (was=0)
Mar 11 14:26:13 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /data/updated <- 1
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] Adding watches for domid: 53
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] Removing watches for domid: 52
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /memory/initial-reservation <- 4296680
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /memory/target <- 4196352
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /control/feature-balloon <- None
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /data/updated <- None
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /memory/memory-offset <- None
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /memory/uncooperative <- None
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /memory/dynamic-min <- 4194304
Mar 11 14:26:19 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /memory/dynamic-max <- 10485760
Mar 11 14:26:21 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /data/updated <- 1
Mar 11 14:26:22 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /data/updated <- 1
Mar 11 14:26:25 xcpng02 squeezed: [debug||3 ||squeeze_xen] domid 53 just started a guest agent (but has no balloon driver); calibrating memory-offset = 2024 KiB
Mar 11 14:26:25 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /memory/memory-offset <- 2024
Mar 11 14:26:25 xcpng02 squeezed: [debug||3 ||squeeze_xen] Xenctrl.domain_setmaxmem domid=53 max=4199400 (was=4297704)
Mar 11 14:26:28 xcpng02 squeezed: [debug||4 ||squeeze_xen] watch /data/updated <- 1

          I hope the filtered xensource.log (filtered for "squeezed") is enough or do you need other events aswell @andriy.sultanov ?

          D 1 Reply Last reply Reply Quote 0
          • D Offline
            dinhngtu Vates đŸȘ XCP-ng Team @MajorP93
            last edited by

            @MajorP93 Hi, are you using the Rust guest agent? We suspect that it's failing to rearm the ballooning feature after resuming from suspend.

            M 2 Replies Last reply Reply Quote 0
            • M Offline
              MajorP93 @dinhngtu
              last edited by MajorP93

              @dinhngtu Yes, I am using the rust based guest agent (https://gitlab.com/xen-project/xen-guest-agent) on all of my Linux guests.

              ade3a347-2452-4d9d-977d-bad46c490ee0-image.jpeg

              Good job tracking the issue down to that!

              Is a suspend/resume performed automatically during live migration?

              //EDIT: I switched from Citrix/Xenserver guest agent to the Vates rust one somewhere in January so maybe my assumption of this issue being related to January round of patches was wrong.
              Or maybe I did not see this issue since the ballooning down to dynamic min was broken before said round of patches according to changelog.

              1 Reply Last reply Reply Quote 0
              • M Offline
                MajorP93 @dinhngtu
                last edited by MajorP93

                @dinhngtu Also if it is related to the Rust-based guest agent: do you guys plan to fix the issue and release a new version soon or is it advised to revert to Xenserver guest agent for now?

                D 1 Reply Last reply Reply Quote 0
                • D Offline
                  dinhngtu Vates đŸȘ XCP-ng Team @MajorP93
                  last edited by

                  @MajorP93 AFAIK @teddyastie is working on a 1.0 release for Linux guests with this bug fix.

                  M 2 Replies Last reply Reply Quote 2
                  • M Offline
                    MajorP93 @dinhngtu
                    last edited by

                    @dinhngtu Looking forward to that! I'll stick with the Rust guest utilities for the time being—hope to see that new release soon!

                    1 Reply Last reply Reply Quote 0
                    • M Offline
                      MajorP93 @dinhngtu
                      last edited by

                      @dinhngtu @teddyastie Do you have a time line / rough estimated release date regarding Rust-based xen-guest-agent version 1.0?
                      Currently evaluating if I need to switch my Linux guests back to Citrix guest tools in the meantime.

                      Looking forward to a reply.

                      Thank you in advance.

                      M 1 Reply Last reply Reply Quote 0
                      • M Offline
                        MajorP93 @MajorP93
                        last edited by MajorP93

                        I can confirm that when using Citrix/Xenserver guest utilities version 8.4 (https://github.com/xenserver/xe-guest-utilities/releases/tag/v8.4.0) memory ballooning / DMC is working fine.

                        After live migration the RAM of the linux guest is expanded to dynamic_max again.

                        So this issue was in fact caused by Rust based xen-guest-agent.

                        For now I'll keep using Citrix/Xenserver guest utilities on my Linux guests until the feature is implemented in Vates rust-based guest utilities.

                        Best regards

                        1 Reply Last reply Reply Quote 0

                        Hello! It looks like you're interested in this conversation, but you don't have an account yet.

                        Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.

                        With your input, this post could be even better 💗

                        Register Login
                        • First post
                          Last post