XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login
    1. Home
    2. johnnezero
    3. Best
    Offline
    • Profile
    • Following 8
    • Followers 0
    • Topics 2
    • Posts 18
    • Groups 0

    Posts

    Recent Best Controversial
    • Server Admin Guide: A Tale of Two Servers: BIOS, GPU, and NUMA Tuning for XCP-ng: Preserving the valuable work done by Tobias Kreidl (@tjkreidl)

      WHAT: This post is dedicated to preserving the valuable server administration guide content produced by @tjkreidl, originally titled "A Tale of Two Servers." These articles were first published in 2019 on the Citrix Blogs, which are unfortunately no longer available.

      KUDOS: Special thanks to @john.c for performing some incredible web sleuthing to recover this content, and to @tjkreidl for creating these essential guides in the first place — as well as for "Breathing Life" back into them by highlighting their importance before they disappeared forever into the ferocious "Bit-Bucket."

      Series Overview & Quick Reference

      If you're looking for a high-level summary of the concepts covered in this series — including specific XCP-ng/XO commands for BIOS power management, GPU scheduling, and NUMA inspection — start here:
      Quick Reference & Summary Guide

      The Complete Series

      =========================================

      Part 1: How BIOS Settings Can Affect Your Apps and GPU Performance:

      A deep dive into CPU Power Management (OS DBPM vs. System DBPM) and Turbo mode.
      Read Article

      Part 2: GPU Settings and Advanced BIOS Tuning

      Exploring Uncore frequency, C1E states, and NVIDIA GPU Scheduler modes (Best Effort vs. Equal Share).
      Read Article

      Part 3: NUMA, CPUs, Sockets/Cores, and VM Performance

      Understanding vNUMA, vCPU oversubscription, and the importance of VM startup order for optimal memory placement.
      Read Article

      Archived for the XCP-ng community. Questions or additional insights? Please discuss below!

      posted in Hardware
      johnnezeroJ
      johnnezero
    • RE: Tag-Based Automation: Manage VM CPU Priority via assigned tag.

      @tjkreidl Found it. I re-posted the topic back to the hardware category (and linked the pdf's to the #tobiaskreidl github) here:
      Server Admin Guide: A Tale of Two Servers: BIOS, GPU, and NUMA Tuning for XCP-ng: Preserving the valuable work done by Tobias Kreidl (@tjkreidl)

      posted in Management
      johnnezeroJ
      johnnezero
    • RE: Tag-Based Automation: Manage VM CPU Priority via assigned tag.

      @john.c Will do - and kudos to you on some phenomenal sleuthing work (and to @tjkreidl for sparking ithis contents recall from an early "bit-bucket" grave)! Happy Day All!

      posted in Management
      johnnezeroJ
      johnnezero
    • RE: Tag-Based Automation: Manage VM CPU Priority via assigned tag.

      @john.c FYI: Bundling/done (Plugin/To come...)

      posted in Management
      johnnezeroJ
      johnnezero
    • Tag-Based Automation: Manage VM CPU Priority via assigned tag.

      UPDATE: Tag-Based-Automation-Bundle v1.0

      WHAT'S NEW: :This update introduces the "Tag-Based-Automation" bundle -- a modular, install.sh-based package that replaces the original standalone set-performance.sh script.

      This bundle is the foundation for the upcoming "Tag-Based-Automation" plugin for Xen Orchestra (if/when all that gets worked out with Vates).
      For now it is a production-ready shell-based bundle that installs cleanly on all XCP-ng pool hosts, and will only run on the current Pool Master.

      The goal is simple: Metadata (Tags) should drive Infrastructure State.

      STANDARD DISCLAIMER HERE

      This software is provided "AS-IS" without any express or implied warranty. While these scripts are being used in a production environment managing VMs, you should always review the code and test it in a non-production environment before full deployment.
      Note: The scripts are designed to ONLY take action on VMs with specific tags assigned -- untagged VMs are never touched.
      That said, as always - your mileage may vary...

      WHAT IS IN THE BUNDLE?

      [ACTIVE]      set-performance.sh
                    Automated enforcement of CPU weights and I/O priorities
                    based on assigned VM tags. Runs via cron. Prevents
                    configuration drift across your entire pool.
      
      [COMING SOON] Manage Tags via CSV
                    Bulk VM metadata management using a CSV as Source of
                    Truth workflow. Export your VM list, edit tags in Excel,
                    sync back to the pool in one command.
      
      [COMING SOON] set-permissions.sh
                    Automated management of Xen Orchestra Resource Sets and
                    user permissions -- driven entirely by your existing VM
                    tags. No manual Resource Set updates ever again.
      

      CHANGE-LOG

      v1.0 - 2026-05-16 (Bundle Release)
        - Modular Architecture: Repackaged as a modular bundle for easier
          updates and future plugin development
        - One-Shot Installer: install.sh handles directory creation,
          permissions, and legacy standalone cleanup detection
        - Orchestrator: main.sh manages module execution and cron
          scheduling via symlink -- no manual crontab editing required
        - Dual Config System: default.conf (never edit) + custom.conf
          (your pool settings) -- your settings are preserved on re-install
        - NFS Integration: Optional log aggregation for centralized
          reporting across multiple pools
        - Master Host Check: Robust UUID comparison ensures script only
          runs on the Pool Master -- safe to deploy on all hosts
        - Foundation for upcoming Xen Orchestra plugin
      
      v3.2 - 2026-05-14 (Final Standalone Release)
        - Fixed Master Host Check: Replaced invalid xe host-is-master
          with correct UUID comparison method
      
      v3.1 - 2026-05-14
        - Config Separation: Moved to conf.d/ override pattern
        - Split into default.conf and custom.conf
      
      v3.0 - 2026-05-14
        - TAG_SUFFIX support for multi-pool deployments
        - Auto-scheduling via initialize -- no manual crontab needed
      
      v2.0 - 2026-05-14
        - External configuration file
        - Master Host Check (first attempt)
        - Main log + summary log
        - Per-tier summary counters
      
      v1.0 - Initial Release
        - Tag-based CPU weight and I/O priority enforcement
        - Four tiers: 0-core, 1-high, 2-normal, 3-low
        - Network QoS cap (100Mbps) for 3-low tagged VMs
      

      CLEANUP: REMOVING THE OLD STANDALONE VERSION (If installed)

      If you installed the original standalone script, run these first:
      
        rm -f /usr/local/bin/set-performance.sh
        rm -rf /usr/local/etc/set-performance.conf.d
        rm -f /etc/cron.hourly/set-performance
        rm -f /etc/cron.daily/set-performance
        rm -f /etc/cron.weekly/set-performance
        rm -f /etc/cron.monthly/set-performance
      

      DOWNLOAD (Recommended)

      Download the v1.0 bundle directly from GitHub:
      https://github.com/johnezero/xo-tag-automation/releases/download/v1.0/tag-automation-v1.0.tar.gz

      INSTALLING THE BUNDLE

      STEP 1: Download and extract on your Pool Master

      wget https://github.com/johnezero/xo-tag-automation/releases/download/v1.0/tag-automation-v1.0.tar.gz
      tar -xzvf tag-automation-v1.0.tar.gz
      cd tag-automation-bundle/

      -- OR --

      Download to your workstation and upload via SCP or WinSCP:
      scp tag-automation-v1.0.tar.gz root@your-pool-master:/root/
      Then on your Pool Master:
      tar -xzvf tag-automation-v1.0.tar.gz
      cd tag-automation-bundle/

      STEP 2: Run the installer
      chmod +x install.sh
      ./install.sh

      The installer will:
      - Detect and warn about any legacy standalone components
      - Create all required directories
      - Deploy all scripts and config files
      - Preserve any existing custom.conf on re-install
      - Verify your NFS code path if applicable

      STEP 3: Edit your pool-specific config:
      *** ACTION REQUIRED -- DO NOT SKIP ***
      vi /usr/local/etc/tag-automation/conf.d/custom.conf

      Set your TAG_SUFFIX to match your pool:
      POOL-1 --> TAG_SUFFIX="-1"
      POOL-2 --> TAG_SUFFIX="-2"
      Single pool --> TAG_SUFFIX=""

      STEP 4: Initialize the bundle (REQUIRED):
      *** ACTION REQUIRED -- DO NOT SKIP ***
      /usr/local/bin/tag-automation/main.sh initialize

      Expected output:
      [OK] Symlink created : /etc/cron.hourly/tag-automation
      [OK] Schedule : hourly
      [OK] Tag suffix : (none)
      [OK] Active tags : 0-core, 1-high, 2-normal, 3-low
      [OK] Performance : true
      [--] Permissions : Coming Soon

      *** Bundle will NOT run until initialize is completed ***

      STEP 5: Verify it is working
      /usr/local/bin/tag-automation/main.sh
      tail /var/log/tag-automation.log
      tail /var/log/tag-automation-summary.log

      THE FILES (i.e. The hard way)

      1-of-5: install.sh
      Note: The install bundle can e located anywhere (i.e. /root/tag-automation-bundle)

      --------------------------------------------
      
      #!/bin/bash
      
      # ============================================
      # Tag-Based-Automation -- install.sh (v1.1)
      # One-shot installer for the full bundle
      # ============================================
      
      BUNDLE_DIR="$(dirname "$0")"
      INSTALL_BIN="/usr/local/bin"
      INSTALL_CONF="/usr/local/etc"
      
      echo ""
      echo "Tag-Based-Automation -- Installer v1.1"
      echo "============================================"
      echo ""
      
      if [ "$(id -u)" != "0" ]; then
          echo "Error: This installer must be run as root."
          exit 1
      fi
      
      if ! command -v xe &>/dev/null; then
          echo "Error: xe command not found."
          echo "This installer must be run on an XCP-ng Pool Master."
          exit 1
      fi
      
      echo "Checking for legacy standalone components..."
      LEGACY_FOUND=false
      
      [ -f "/usr/local/bin/set-performance.sh" ] && \
          echo "  [!] Legacy script found: /usr/local/bin/set-performance.sh" && \
          LEGACY_FOUND=true
      
      [ -d "/usr/local/etc/set-performance.conf.d" ] && \
          echo "  [!] Legacy config dir found: /usr/local/etc/set-performance.conf.d" && \
          LEGACY_FOUND=true
      
      crontab -l 2>/dev/null | grep -q "set-performance.sh" && \
          echo "  [!] Legacy crontab entry detected." && \
          LEGACY_FOUND=true
      
      LEGACY_CRONS=$(find /etc/cron.hourly /etc/cron.daily /etc/cron.weekly \
          /etc/cron.monthly -name "set-performance" 2>/dev/null)
      if [ -n "$LEGACY_CRONS" ]; then
          echo "  [!] Legacy cron symlinks detected:"
          echo "$LEGACY_CRONS" | sed 's/^/      - /'
          LEGACY_FOUND=true
      fi
      
      if [ "$LEGACY_FOUND" = true ]; then
          echo ""
          echo "-----------------------------------------------------------------------"
          echo "CAUTION: Legacy standalone components detected."
          echo "Please remove them before proceeding:"
          echo ""
          echo "  rm -f /usr/local/bin/set-performance.sh"
          echo "  rm -rf /usr/local/etc/set-performance.conf.d"
          echo "  rm -f /etc/cron.hourly/set-performance"
          echo "  rm -f /etc/cron.daily/set-performance"
          echo "  rm -f /etc/cron.weekly/set-performance"
          echo "  rm -f /etc/cron.monthly/set-performance"
          echo "-----------------------------------------------------------------------"
          echo ""
          read -p "Abort to clean up manually? (Y/n): " choice
          case "$choice" in
            n|N ) echo "Proceeding with caution -- watch for conflicts!"; echo "" ;;
            * )   echo "Installation aborted. Clean up and re-run install.sh."; exit 1 ;;
          esac
      else
          echo "[OK] No legacy components found -- clean install!"
          echo ""
      fi
      
      echo "Creating directory structure..."
      mkdir -p "$INSTALL_BIN/tag-automation/modules"
      mkdir -p "$INSTALL_CONF/tag-automation/conf.d"
      echo "[OK] Directories created"
      
      echo "Installing main.sh..."
      cp "$BUNDLE_DIR/main.sh" "$INSTALL_BIN/tag-automation/main.sh"
      chmod +x "$INSTALL_BIN/tag-automation/main.sh"
      echo "[OK] main.sh installed"
      
      echo "Installing modules..."
      cp "$BUNDLE_DIR/modules/set-performance.sh" \
         "$INSTALL_BIN/tag-automation/modules/set-performance.sh"
      chmod +x "$INSTALL_BIN/tag-automation/modules/set-performance.sh"
      echo "[OK] set-performance.sh installed"
      echo "[--] set-permissions.sh -- Coming Soon (not installed)"
      
      echo "Installing configuration files..."
      cp "$BUNDLE_DIR/conf.d/default.conf" \
         "$INSTALL_CONF/tag-automation/conf.d/default.conf"
      echo "[OK] default.conf installed"
      
      if [ ! -f "$INSTALL_CONF/tag-automation/conf.d/custom.conf" ]; then
          cp "$BUNDLE_DIR/conf.d/custom.conf" \
             "$INSTALL_CONF/tag-automation/conf.d/custom.conf"
          echo "[OK] custom.conf installed (first time)"
      else
          echo "[OK] custom.conf already exists -- skipping (your settings preserved)"
      fi
      
      NFS_CODE_PATH="/mnt/v0/code/tag-automation"
      if [ -d "$NFS_CODE_PATH" ]; then
          mkdir -p "$NFS_CODE_PATH/logs"
          echo "[OK] NFS code path verified : $NFS_CODE_PATH"
      else
          echo "[--] NFS code path not found: $NFS_CODE_PATH"
          echo "     mkdir -p $NFS_CODE_PATH/logs"
      fi
      
      echo ""
      echo "============================================"
      echo "  *** ACTION REQUIRED -- DO NOT SKIP ***"
      echo "============================================"
      echo ""
      echo "  STEP 1: Edit your pool-specific config:"
      echo "    vi $INSTALL_CONF/tag-automation/conf.d/custom.conf"
      echo ""
      echo "    Set your TAG_SUFFIX to match your pool:"
      echo "      POOL-1 --> TAG_SUFFIX=\"-1\""
      echo "      POOL-2 --> TAG_SUFFIX=\"-2\""
      echo "      Single pool --> TAG_SUFFIX=\"\""
      echo ""
      echo "  STEP 2: Initialize the bundle (REQUIRED):"
      echo "    /usr/local/bin/tag-automation/main.sh initialize"
      echo ""
      echo "  STEP 3: Verify it is working:"
      echo "    /usr/local/bin/tag-automation/main.sh"
      echo "    tail /var/log/tag-automation.log"
      echo ""
      echo "  *** Bundle will NOT run until initialize is completed ***"
      echo "============================================"
      

      2-of-5: main.sh

      #!/bin/bash
      
      # ============================================
      # Tag-Based-Automation -- main.sh (v1.0)
      # Orchestrator: loads config, runs modules
      # ============================================
      
      CONF_DIR="/usr/local/etc/tag-automation/conf.d"
      DEFAULT_CONF="$CONF_DIR/default.conf"
      CUSTOM_CONF="$CONF_DIR/custom.conf"
      SCRIPT_PATH="/usr/local/bin/tag-automation/main.sh"
      
      if [ -f "$DEFAULT_CONF" ]; then
          source "$DEFAULT_CONF"
      else
          echo "Error: default.conf not found at $DEFAULT_CONF"
          exit 1
      fi
      [ -f "$CUSTOM_CONF" ] && source "$CUSTOM_CONF"
      
      CORE_TAG="${CORE_BASE}${TAG_SUFFIX}"
      HIGH_TAG="${HIGH_BASE}${TAG_SUFFIX}"
      NORMAL_TAG="${NORMAL_BASE}${TAG_SUFFIX}"
      LOW_TAG="${LOW_BASE}${TAG_SUFFIX}"
      
      initialize_plugin() {
          echo ""
          echo "Tag-Based-Automation -- Initialization"
          echo "============================================"
      
          if [ ! -d "$CONF_DIR" ]; then
              echo "Error: Config directory not found: $CONF_DIR"
              exit 1
          fi
      
          if [ ! -f "$DEFAULT_CONF" ]; then
              echo "Error: default.conf not found."
              exit 1
          fi
      
          [ ! -f "$CUSTOM_CONF" ] && \
              echo "Warning: custom.conf not found. Using defaults only."
      
          TARGET_DIR="/etc/cron.${SCHEDULE}"
          SYMLINK_PATH="${TARGET_DIR}/tag-automation"
      
          if [ ! -d "$TARGET_DIR" ]; then
              echo "Error: Cron directory not found: $TARGET_DIR"
              echo "Valid SCHEDULE options: hourly, daily, weekly, monthly"
              exit 1
          fi
      
          for interval in hourly daily weekly monthly; do
              rm -f "/etc/cron.${interval}/tag-automation"
          done
      
          ln -sf "$SCRIPT_PATH" "$SYMLINK_PATH"
      
          if [ -L "$SYMLINK_PATH" ]; then
              echo "[OK] Symlink created : $SYMLINK_PATH"
              echo "[OK] Schedule        : $SCHEDULE"
              echo "[OK] Tag suffix      : ${TAG_SUFFIX:-(none)}"
              echo "[OK] Active tags     : $CORE_TAG, $HIGH_TAG, $NORMAL_TAG, $LOW_TAG"
              echo "[OK] Performance     : ${ENABLE_PERFORMANCE:-true}"
              echo "[--] Permissions     : Coming Soon"
              echo ""
              echo "Initialization complete. Bundle will run $SCHEDULE via cron."
          else
              echo "Error: Failed to create symlink."
              exit 1
          fi
      }
      
      [ "$1" == "initialize" ] && initialize_plugin && exit 0
      
      # --- MASTER HOST CHECK ---
      POOL_MASTER=$(xe pool-list params=master --minimal)
      LOCAL_HOST=$(xe host-list name-label=$(hostname) --minimal)
      [ "$POOL_MASTER" != "$LOCAL_HOST" ] && exit 0
      
      exec >> "$MAIN_LOG" 2>&1
      echo "--- Tag-Automation Starting: $(date) ---"
      echo "    Tags: $CORE_TAG | $HIGH_TAG | $NORMAL_TAG | $LOW_TAG"
      
      if [ "${ENABLE_PERFORMANCE:-true}" == "true" ]; then
          echo "--- Running set-performance module ---"
          bash /usr/local/bin/tag-automation/modules/set-performance.sh
      fi
      
      # --- set-permissions MODULE (COMING SOON) ---
      # if [ "${ENABLE_PERMISSIONS:-false}" == "true" ]; then
      #     echo "--- Running set-permissions module ---"
      #     bash /usr/local/bin/tag-automation/modules/set-permissions.sh
      # fi
      
      # --- NFS LOG PUSH (graceful -- no auto-remount) ---
      NFS_CODE_PATH="/mnt/v0/code/tag-automation"
      if mountpoint -q "/mnt/v0" && [ -d "$NFS_CODE_PATH" ]; then
          cp "$SUMMARY_LOG" "$NFS_CODE_PATH/logs/" 2>/dev/null
          echo "[OK] Logs pushed to NFS"
      else
          echo "[--] NFS not available -- skipping log push"
      fi
      
      echo "--- Tag-Automation Complete: $(date) ---"
      echo "$(date '+%Y-%m-%d %H:%M:%S') $(hostname) Suffix:${TAG_SUFFIX:-(none)} Performance:${ENABLE_PERFORMANCE:-true}" >> "$SUMMARY_LOG"
      

      3-of-5: modules/set-performance.sh

      #!/bin/bash
      
      # ============================================
      # Tag-Based-Automation Module: set-performance.sh (v3.2)
      # Called by main.sh -- inherits all variables
      # ============================================
      
      count_0=0
      count_1=0
      count_2=0
      count_3=0
      
      echo "--- Starting Performance Sync: $(date) ---"
      echo "    Tags: $CORE_TAG | $HIGH_TAG | $NORMAL_TAG | $LOW_TAG"
      
      echo "=== Applying $CORE_TAG (Weight: $CORE_WEIGHT, Pri: $CORE_IO_PRI) ==="
      for uuid in $(xe vm-list tags:contains="$CORE_TAG" --minimal | tr ',' '\n'); do
          [ -z "$uuid" ] && continue
          xe vm-param-set uuid=$uuid VCPUs-params:weight=$CORE_WEIGHT \
              other-config:sched-pri=$CORE_IO_PRI
          echo "  [OK] CORE applied: $uuid"
          ((count_0++))
      done
      
      echo "=== Applying $HIGH_TAG (Weight: $HIGH_WEIGHT, Pri: $HIGH_IO_PRI) ==="
      for uuid in $(xe vm-list tags:contains="$HIGH_TAG" --minimal | tr ',' '\n'); do
          [ -z "$uuid" ] && continue
          xe vm-param-set uuid=$uuid VCPUs-params:weight=$HIGH_WEIGHT \
              other-config:sched-pri=$HIGH_IO_PRI
          echo "  [OK] HIGH applied: $uuid"
          ((count_1++))
      done
      
      echo "=== Applying $NORMAL_TAG (Weight: $NORMAL_WEIGHT, Pri: $NORMAL_IO_PRI) ==="
      for uuid in $(xe vm-list tags:contains="$NORMAL_TAG" --minimal | tr ',' '\n'); do
          [ -z "$uuid" ] && continue
          xe vm-param-set uuid=$uuid VCPUs-params:weight=$NORMAL_WEIGHT \
              other-config:sched-pri=$NORMAL_IO_PRI
          echo "  [OK] NORMAL applied: $uuid"
          ((count_2++))
      done
      
      echo "=== Applying $LOW_TAG (Weight: $LOW_WEIGHT, Pri: $LOW_IO_PRI) ==="
      for uuid in $(xe vm-list tags:contains="$LOW_TAG" --minimal | tr ',' '\n'); do
          [ -z "$uuid" ] && continue
          xe vm-param-set uuid=$uuid VCPUs-params:weight=$LOW_WEIGHT \
              other-config:sched-pri=$LOW_IO_PRI
          echo "  [OK] LOW applied: $uuid"
          ((count_3++))
      done
      
      echo "--- Performance Sync Complete: $(date) ---"
      echo "$(date '+%Y-%m-%d %H:%M:%S') $(hostname) \
          $CORE_TAG:$count_0 $HIGH_TAG:$count_1 \
          $NORMAL_TAG:$count_2 $LOW_TAG:$count_3" >> "$SUMMARY_LOG"
      

      4-of-5: conf.d/default.conf

      # ============================================
      # default.conf
      # Global defaults for Tag-Based-Automation
      # DO NOT edit -- use custom.conf for overrides
      # ============================================
      
      SCHEDULE="hourly"
      
      ENABLE_PERFORMANCE=true
      # ENABLE_PERMISSIONS=false  # Coming Soon
      
      MAIN_LOG="/var/log/tag-automation.log"
      SUMMARY_LOG="/var/log/tag-automation-summary.log"
      
      CORE_BASE="0-core"
      HIGH_BASE="1-high"
      NORMAL_BASE="2-normal"
      LOW_BASE="3-low"
      
      CORE_WEIGHT="2048"
      CORE_IO_PRI="7"
      
      HIGH_WEIGHT="1024"
      HIGH_IO_PRI="7"
      
      NORMAL_WEIGHT="256"
      NORMAL_IO_PRI="4"
      
      LOW_WEIGHT="128"
      LOW_IO_PRI="1"
      

      5-of-5: conf.d/custom.conf

      # ============================================
      # custom.conf
      # Pool-specific overrides
      # Edit this file to customize your environment
      # ============================================
      
      # --- TAG SUFFIX ---
      # POOL-1 --> TAG_SUFFIX="-1"
      # POOL-2 --> TAG_SUFFIX="-2"
      # Generic --> TAG_SUFFIX=""
      TAG_SUFFIX=""
      
      # --- OPTIONAL OVERRIDES ---
      # Uncomment to override default.conf values:
      # SCHEDULE="daily"
      # ENABLE_PERFORMANCE=true
      # CORE_WEIGHT="4096"
      # HIGH_WEIGHT="2048"
      # NORMAL_WEIGHT="512"
      # LOW_WEIGHT="256"
      
      posted in Management
      johnnezeroJ
      johnnezero
    • RE: Server Admin Guide: A Tale of Two Servers: BIOS, GPU, and NUMA Tuning for XCP-ng: Preserving the valuable work done by Tobias Kreidl (@tjkreidl)

      @poddingue Thank you!
      "Anything and Everything we can to to improve XCP-ng", is the "Name of the Game" 🙂

      posted in Hardware
      johnnezeroJ
      johnnezero