XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Endless Xapi#getResource /rrd_updates in tasks list

    Scheduled Pinned Locked Moved XCP-ng
    85 Posts 10 Posters 29.7k Views 10 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • U Offline
      uwood
      last edited by

      It is gone with a548a after I've restarted XO and toolstack.

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates 🪐 Co-Founder CEO
        last edited by

        That's confusing. @Mathieu do you confirm?

        MathieuM U 2 Replies Last reply Reply Quote 0
        • MathieuM Online
          Mathieu @olivierlambert
          last edited by

          @olivierlambert
          Yes, confusing, indeed 😄
          I just now restarted XO and the toolstack one more time to be sure. Yesterday, it was OK at the beginning but the issue reappeared after a few hours.
          I'll let you know ASAP.

          1 Reply Last reply Reply Quote 1
          • olivierlambertO Offline
            olivierlambert Vates 🪐 Co-Founder CEO
            last edited by

            Thanks!

            1 Reply Last reply Reply Quote 0
            • U Offline
              uwood @olivierlambert
              last edited by

              @olivierlambert now, about 10 hours after restart of XO and Toolstack I have two task too.

              49a02320-a8b3-4953-b3f1-c0b1d7ce27d0-image.png

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates 🪐 Co-Founder CEO
                last edited by

                Thanks, might be useful for @julien-f

                MathieuM 1 Reply Last reply Reply Quote 0
                • MathieuM Online
                  Mathieu @olivierlambert
                  last edited by

                  @olivierlambert
                  Same on my host, the first stuck task appeared 5 hours after toolstack and XO reboot.

                  1 Reply Last reply Reply Quote 0
                  • U Offline
                    uwood
                    last edited by

                    8b7e1 still got the tasks

                    U 2 Replies Last reply Reply Quote 0
                    • U Offline
                      uwood @uwood
                      last edited by

                      c5767591-21cb-49cb-bb92-f5d14d9a090f-image.png

                      1 Reply Last reply Reply Quote 0
                      • U Offline
                        uwood @uwood
                        last edited by

                        @uwood 0794a still some tasks

                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO Offline
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          That's tricky, @julien-f couldn't reproduce it in our lab. So we'll need more details on your setup guys.

                          1 1 Reply Last reply Reply Quote 0
                          • 1 Offline
                            14wkinnersley @olivierlambert
                            last edited by

                            @olivierlambert
                            I have one pool, and three Nodes in my pool.
                            Node 1 – Dell R420, 2x 10gb NICs (Master) – 192.168.1.4
                            Node 2 – Dell R620, 2x 10gb NICs – 192.168.1.7
                            Node 3 – Dell R420, 2x 10gb NICs – 192.168.1.3
                            XO VM – Ubuntu 22.04.4 LTS – 192.168.1.15

                            Each Node has a Bond of NIC 2 + 3. (Node 2 with the R620 has the mac addresses re-assigned to work correctly)

                            Above each node, network wise, is 2x switches (2x Unifi USW-Agg) and I use the Unifi Dream Machine Pro as my router. I am able to ping the other Nodes from each Node.

                            Local DNS utilizes Technitium DNS (Primary & Secondary), as a recursive DNS.

                            My SR’s are two iSCSI datastores that run on a separate server running TrueNAS Scale on a Dell R320.

                            Within my pools, I run about 25x virtual machines. I run nightly backups for ~5 VM’s, and weekly backup’s for all VM’s. Backups have a remote NFS storage location hosted on a separate server. I have 3 VM’s that run on separate network vlans than the rest, and those networks are setup under the pool, and upstream on the router.

                            Plugins, I have the following enabled
                            • Backup-reports
                            • Load-balancer (performance mode)
                            • Perf-alert
                            • Transport-email
                            • Usage-report

                            From my testing, this was introduced with commit 6c16055 - Mar 15. I have since rolled back to c6451cf and have stayed on this commit for the past several days.

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              Thanks for the details @14wkinnersley !

                              MathieuM 1 Reply Last reply Reply Quote 1
                              • olivierlambertO Offline
                                olivierlambert Vates 🪐 Co-Founder CEO
                                last edited by

                                @14wkinnersley can you disable perf alert and load balancer plugin and see if it still happening?

                                1 2 Replies Last reply Reply Quote 0
                                • MathieuM Online
                                  Mathieu @olivierlambert
                                  last edited by

                                  @olivierlambert
                                  Simpler setup on my side:

                                  Pool 1 - 1 x ASRock Rack 1U4LW-X570/2L2T RPSU with about 10 VMs
                                  Pool 2 - 1 x HP DL360 Gen9 with only the XO VM (Debian 11).

                                  Each host has 2 x 10 Gb NIC in use (one for VMs/MGMT, the other one for NFS storage).

                                  The storage is a NFS share on a QNAP NAS (except for the XO VM which is on the local storage of the DL360 host).

                                  The 2 hosts and the NFS storage are connected on the same10Gbit Ubiquiti EdgeSwitch.

                                  Same plugins as @14wkinnersley + audit and sdn-controller.

                                  The issue with stuck task is only appearing on the pool 1, not on the pool 2 with only the XO VM.

                                  1 Reply Last reply Reply Quote 0
                                  • 1 Offline
                                    14wkinnersley @olivierlambert
                                    last edited by

                                    @olivierlambert Will do. Plugins are disabled and I'm updating back to master right now. Will report back.

                                    MathieuM 1 Reply Last reply Reply Quote 1
                                    • olivierlambertO Offline
                                      olivierlambert Vates 🪐 Co-Founder CEO
                                      last edited by olivierlambert

                                      Thank you very much, both of you 🙂

                                      1 Reply Last reply Reply Quote 0
                                      • MathieuM Online
                                        Mathieu @14wkinnersley
                                        last edited by

                                        I'm gonna try the same and will let you know.

                                        1 Reply Last reply Reply Quote 1
                                        • 1 Offline
                                          14wkinnersley @olivierlambert
                                          last edited by

                                          @olivierlambert Updated to master (Commit: 79c9ef0) with perf alert and load balancer plugin's disabled. Cleared the toolstack for good measure, and I got no hanging tasks.

                                          For some further testing, I flipped the load balancer plugin back on and waited about 5 hours and received no hanging tasks. I turned load balancer plugin back off (Just in case), cleared the toolstack, and turned the perf alert plugin on and received the first hanging task within about 10 minutes, and subsequently had more tasks hang every ~10-12 minutes.

                                          1 Reply Last reply Reply Quote 0
                                          • olivierlambertO Offline
                                            olivierlambert Vates 🪐 Co-Founder CEO
                                            last edited by

                                            That would have been my next question 😄 So it's the perf alert thing likely. Thank you, this will help @julien-f a lot

                                            MathieuM 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post