XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Micro XCP-ng cluster Thoughts and Suggestions.

    Scheduled Pinned Locked Moved Share your setup!
    16 Posts 3 Posters 1.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • planedropP
      planedrop Top contributor @brodiecyber
      last edited by

      @brodiecyber I don't really see an issue with this setup, just keep in mind Celeron's are pretty low end so don't expect amazing performance. Keep in mind this chip doesn't support VT-d so you can't do any PCIe passthrough, but it does have VT-x so should be fine to run things on.

      brodiecyberB 1 Reply Last reply Reply Quote 0
      • brodiecyberB
        brodiecyber @planedrop
        last edited by

        @planedrop Hi thanks for the input. From what i can tell the j in the j1900 denotes that these are low end atom cores so im not expecting great performance.

        pcie pass through is not that big a deal since its pcie 2.0 x1 and the case does not support additional pcie cards

        My biggest problem is that dom0 normally uses 1.7 -2 GB of ram and these boards have a max of 8gb of ram

        gskgerG 1 Reply Last reply Reply Quote 0
        • gskgerG
          gskger Top contributor @brodiecyber
          last edited by

          @brodiecyber Personally, I think that the setup is very limited in terms of performance with RAM being the greatest challenge from the beginning.

          While I understand your initial idea of a low hardware and cost micro XCP-ng cluster to check out HA and learning runX/XOSTOR, you might be paying some good money for a low performance setup with little head room.

          Probably not an option, but if it is about learning and testing, you could go for a used mini pc (i10500t, 64GB RAM, 1TB boot and ISO drive, 2TB VM drive) and setup everything as a nested virtualization XCP-ng cluster? That would allow for snapshots and backups to make testing and rollback a breeze. My HP ProDesk 600 G6 is at an 20W average with 16 VMs running. Going for a physical XCP-ng cluster is much more fun though 😇 .

          brodiecyberB 1 Reply Last reply Reply Quote 0
          • brodiecyberB
            brodiecyber @gskger
            last edited by

            @gskger I agree that the ram is the greatest challenge especially since dom0 takes 1.5 -2gb of ram in this setup.

            I wanted to do ServeTheHomes Tiny,Mini,Micro nodes but Ive tried looking for them secondhand and the cost of a single dell micro i5-8th gen were I live. I was able to get 3-j1900 motherboards off eBay including shipping.

            That's why when runX containers are more mature it would be a great platform for that. But apart form XCP-ng it works well as k3s nodes for kubernetes

            gskgerG 1 Reply Last reply Reply Quote 0
            • gskgerG
              gskger Top contributor @brodiecyber
              last edited by

              @brodiecyber Ok, that makes sense.

              I once run XCP-ng on a HP T620 PLUS Thin Client with some lightweight Debian VM for a test. The HPs AMD GX-420CA has a bit higher CPU Mark compared to the j1900 (1787 vs. 1148), but the single thread performance is more or less the same.

              Nothing to run major workloads on, but should be ok for testing HA and general XCP-ng operations at low speed. No idea how XOSTOR will be performing on that platform, so please update us on your findings.

              brodiecyberB 3 Replies Last reply Reply Quote 0
              • brodiecyberB
                brodiecyber @gskger
                last edited by

                @gskger

                Sure thing Ill update the post.

                Just waiting on a the cases and boot SSDs should be here Thursday. Then need to figure out if i want to use some cheap HDDs or cheap SSDs for XOSTOR

                1 Reply Last reply Reply Quote 0
                • brodiecyberB
                  brodiecyber @gskger
                  last edited by

                  @gskger But in the absence of runX containers

                  I used some Alpine Linux VMs with an overkill amount of ram at 512mb yes mb and 10gb of storage and because it supports docker and compose you can run any docker app you just need to make sure their is enough ram for the running container.

                  only thing is no VM icon

                  1 Reply Last reply Reply Quote 0
                  • brodiecyberB
                    brodiecyber @gskger
                    last edited by

                    @gskger Thought id update you the cases are coming today

                    Apparently yesterday was a public holiday✌

                    Great for relaxation.
                    Bad for when your waiting on components.

                    But on another not i had an idea
                    I have an available PCIE 2.0 x1 slot that maxes out at the speed of SATA
                    So i had the idea for using an m.2 to PCIE adapter and some cheap nvme drives
                    so i can have a local SR on each node and an HA SR distributed on all nodes. not sure yet but this could be interesting

                    1 Reply Last reply Reply Quote 0
                    • brodiecyberB
                      brodiecyber
                      last edited by

                      1682689286731.jpg New Hosts.PNG

                      Hosts are online and updating before i cluster them

                      brodiecyberB 1 Reply Last reply Reply Quote 1
                      • brodiecyberB
                        brodiecyber @brodiecyber
                        last edited by

                        @brodiecyber After 2 hours of waiting its fianlly updates and cluster configured.

                        I will note out of 8GB of RAM. Dom0 is using 1.9GB leaving me with 6.1GB of RAM

                        1 Reply Last reply Reply Quote 0
                        • brodiecyberB
                          brodiecyber
                          last edited by

                          Update the cluster is doing good i just need to get some SSDs for XOSTOR.

                          I have a 1cpu 1gb ram 25gb storage VM running over gigabit networking for ha storage. i don't recommend it its just for testing because disk io and bandwidth is limited.

                          VM runs and live migration takes about 17 Mississippi😁

                          brodiecyberB 1 Reply Last reply Reply Quote 0
                          • brodiecyberB
                            brodiecyber @brodiecyber
                            last edited by

                            @brodiecyber

                            Update Setting up cloud-init templates to provision VMs with Terraform and Ansible for application and patch management.

                            Maybe ill even setup a XO-ce dedicated to this cluster 🤠

                            MicroXen.PNG

                            brodiecyberB 1 Reply Last reply Reply Quote 0
                            • brodiecyberB
                              brodiecyber @brodiecyber
                              last edited by

                              @brodiecyber ok

                              new update

                              Redoing my cloud-init templates and preparing for the next version of Debian in June release 12.
                              Am using terraform to provision virtual machines and cloud-init and ansible to install software

                              I'm also using the boot disk as VM storage i don't recommend it as its a 120GB disk and i had to redo my templates as the VM they provision are to big for the little storage.

                              but Im running

                              • HA pihole with a keepalived

                              • WIreguard

                              • Terraform

                              • Dashboards. RSS, Archivebox

                              Will be adding

                              • dedicated XO-ce VM

                              • Unifi Controller

                              Infrastructure.PNG

                              1 Reply Last reply Reply Quote 0
                              • brodiecyberB
                                brodiecyber
                                last edited by

                                Ive still got 4GB of ram to spare on Xen- 2 and 3 so abit more room for 1 CPU/RAM vms to deploy

                                I still think that this a the perfect environment for RUNx containers just the efficiency especially with memory will help alot

                                1 Reply Last reply Reply Quote 0
                                • brodiecyberB
                                  brodiecyber
                                  last edited by

                                  New update

                                  I got a asus n3150i-c motherboard and found a spare ssd
                                  P_setting_fff_1_90_end_500.jpg

                                  Im reusing a node for parts and am installing truenas scale as a shared storage target for NFS

                                  Its only 100gb of SSD storage but before i redo my workloads im waiting on debian 12 to be released

                                  so im down a compute node but am up on shared storage with dedicated node

                                  1 Reply Last reply Reply Quote 0
                                  • olivierlambertO olivierlambert moved this topic from Community on
                                  • First post
                                    Last post