acebmxer

install_xen_orchestra v0.5.0 — build your own cloud-init VM templates

As I am sure most of you know that XOA Hub is not accessible from Xen Orchestra from sources.

So v0.5.0 of my install script adds a VM Template Library — a new
--build-templates flag (and menu entry) that builds the equivalent on your own pool, from each distribution's own published cloud image.

./install-xen-orchestra.sh --build-templates

It runs from your workstation over SSH against the pool master, like --deploy does. Nothing is installed locally and your XO install isn't touched.

What you get

A sealed template that shows up next other vm templates you have, with:

• cloud-init — supply an SSH key or a full cloud-config at VM creation.
• XCP-ng guest tools, installed from guest-tools.iso on the pool. Without these XO never reports the VM's IP. On Debian 13 there's no
  xe-guest-utilities package, and a failed package install doesn't stop cloud-init, so an apt-first approach silently gives you a template that looks fine and is useless. The ISO is what the XCP-ng docs prescribe anyway.
• A scrubbed identity — machine-id, SSH host keys, cloud-init state and logs are all wiped before sealing, so clones don't collide on DHCP leases or share an SSH fingerprint.
• growroot, so the disk size you ask for at creation is actually filled.
• One template for both BIOS and UEFI — the Debian generic images carry an ESP and a BIOS boot partition, so you just flip "Boot firmware" in XO's advanced settings. The build itself isn't prompted for storage or network —
  it uses your pool's default SR (or the emptiest one, if you haven't set a default) and the management network. Both are just build-time choices; you pick whatever you want when you actually create a VM from the template.

Images are pulled from the distribution's own mirror (nothing redistributed), downloaded onto the pool master so the ~3 GB never crosses your workstation's link, and verified against the origin's published SHA512SUMS at build time
—so new upstream releases get picked up without me editing anything.

Allow roughly five to ten minutes per template; it has to boot the VM once to
install the guest agent and run the scrub.

Distros

Debian 13 (Trixie) is the only one for now. That's a starting point, not the limit — the catalogue is a table with one row per distro, and adding another is a row in that table, not a change to the build.

So: comment here if there's a specific distro you'd want to see sooner rather than later and I'll prioritise accordingly.

Repo: https://github.com/acebmxer/install_xen_orchestra
Docs for this feature: https://github.com/acebmxer/install_xen_orchestra/blob/main/docs/templates.md

@flakpyro The patch was pushed though to XO from Sources. Hasnt hit XOA yet i dont beleive.

https://github.com/vatesfr/xen-orchestra/commit/003cc6f9cc7de6bff9eaee9fa0bc14a1f7b64cc0

I am not seeing this error after patch applied to XOA...

fix(proxy): fetch failed  (#10346)
* fix(backups): fetch failed

disable compresion between xoa on proxy on xoa side

* fix(xo-proxy): disable compression on the proxy

it was disable with hrp, replaced by #10038 since it was never
sending accept-encoding header

compression can be put back later on a route by route basis after
careful checks ( for example : file restore, backup listing )

v0.4.1 and v0.4.2 are out.

v0.4.1 — mostly credential encryption (ENCRYPT_REDIS_CREDENTIALS)

• The preflight check now catches a Xen guest that's missing xenstore-read / xenstore-write. Before, that guest passed the systemd-detect-virt check and then failed later at xo-server startup, because there was nowhere to store the XenStore half of the key. It now names the missing tools and the package that provides them.
• --backup now warns that it contains neither half of the encryption key, and points at the passphrase-protected XO config export as the actual recovery artifact.
• --uninstall warns before deleting this host's on-disk key half — doing so while leaving Redis in place turns the stored records into unreadable ciphertext.
• The docs (README, sample config, generated config.toml comments) now explain why the XAPI credential in Redis is stored reversibly instead of hashed: it's replayed on every connect and auto-reconnect, so it can't be a hash. They also spell out the recovery caveats — the key is split between XenStore and /var/lib/xo-server/data, and losing either half while encryption is on makes the records permanently undecryptable.
• Also flagged in the sample config: pointing REDIS_URI at an off-host Redis sends the pool credentials over the network in cleartext.

v0.4.2 — docs and packaging only, no behaviour change:

• The README had grown past 700 lines, with the --deploy walkthrough alone taking up a third of it. The large reference sections moved out to docs/deployment.md, docs/configuration.md, docs/authentication.md and docs/troubleshooting.md; the README is back to ~330 lines with a table pointing at them. Nothing was removed.
• Expanded the badge row — release tag, last commit, open issues, unit-test count, number of distros the CI container matrix covers, and ShellCheck status.

Where the project's got to

I started this in February as a script to save myself doing the from-source install by hand every time I rebuilt my homelab. Seven months and twelve tagged releases later it's grown into something a fair bit bigger:

• --install does the from-source build on whatever distro you run it on — the Debian/Ubuntu, RHEL/Alma/Rocky/CentOS and Fedora families, eight distributions in all, each one smoke-tested in CI alongside 118 unit tests.
• --deploy (added in v0.3.0) starts a step earlier: point it at a XCP-ng / XenServer pool and it builds the VM, pulls a Debian cloud image straight onto the pool, seeds it with cloud-init and runs the install inside it. It's there for people who have a pool but no Linux VM to put XO on.
• --proxy deploys an XO Proxy VM the same way — that one came straight out of requests in this thread.
• Around that: a config file so nothing needs editing in the script itself, --update / --rebuild with a local build cache, --backup / --restore / --uninstall, non-root operation with encrypted Redis credentials, firewall handling, and a lot of --deploy security work in v0.4.0 — image checksums, pool host-key pinning, the deploy SSH key destroyed at the end of the run, a required admin password, passwordless sudo revoked once the install finishes.

Since the v0.3.0 / v0.4.0 post a couple of weeks ago the repo has had 263 clones from 49 unique cloners.

Thanks

To everyone in this thread who's installed it, tested it and come back with feedback — genuinely, thank you. The proxy support, the credential-encryption questions that turned into proper docs, the config-file suggestions, the distro edge cases on systems I don't run myself: several of the releases above have something in them that started as a post here. Keep it coming.

Repo: https://github.com/acebmxer/install_xen_orchestra
Changelog: https://github.com/acebmxer/install_xen_orchestra/blob/main/CHANGELOG.md

@pdonias said:

Hello everyone! We need you!

We're currently designing the XO 6 UI for non-admin users, and some choices are genuinely hard to make. If you'd like to give us your opinion, here's a 30-second survey with 2 questions we couldn't settle ourselves: https://survey.vates.tech/s/cms4nrqb4005wrw01021ru6dy

Thanks!

Thank you with presenting us with a choice. I have made my comments and offered a 3rd option.

To sum up... It appears the patch has worked for me but has shed some light on miss configuration with the network design at this remote location, that I will need to take a real closer look at.

@florent thank you very much for your continue assistance with my issues.

@florent

Backups ran after patch applied...

1 pool had zero issues on backup 100% complete...
Other pool while it complete did have some errors...

Started a second backup on same pool and giving same results.

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "compactMjml",
    "hideSuccessfulItems": true
  },
  "id": "1788428437610",
  "jobId": "c0e31730-7d45-4f3a-8733-8cea444b127d",
  "jobName": "Salem - Delta Backups",
  "message": "backup",
  "proxyId": "7840039e-a3cb-41ed-acee-3690f89a1426",
  "scheduleId": "ce3c801c-4384-4726-bda1-f6bae788e2f4",
  "start": 1788428437610,
  "status": "success",
  "tasks": [
    {
      "id": "0mtlc477z-gdvpj68kva",
      "start": 1788428438927,
      "status": "success",
      "tasks": [
        {
          "id": "0mtlc478f-guqsurb38qa",
          "start": 1788428438943,
          "status": "success",
          "end": 1788428440120,
          "result": {
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        },
        {
          "id": "0mtlc48cx-c1y79k93vl",
          "start": 1788428440401,
          "status": "success",
          "end": 1788428442225,
          "result": "67677eec-494b-6ab6-0447-ac4dba8932a1",
          "message": "snapshot"
        },
        {
          "id": "0mtlc49rm-ou0nslm2c7l",
          "start": 1788428442227,
          "status": "success",
          "tasks": [
            {
              "id": "0mtlc724s-62ae1bzdupa",
              "start": 1788428572300,
              "status": "success",
              "end": 1788428932605,
              "result": {
                "size": 20385362944
              },
              "message": "transfer",
              "data": {
                "progress": 99
              }
            },
            {
              "id": "0mtlceupa-f476k7iq2u4",
              "start": 1788428935918,
              "status": "success",
              "tasks": [
                {
                  "id": "0mtlceutz-yztgqk0sv0c",
                  "start": 1788428936087,
                  "status": "success",
                  "end": 1788429285986,
                  "result": {
                    "size": 20380123136,
                    "id": "5fb5f6f1-709b-b149-4aa6-31514ef2c7f2"
                  },
                  "message": "transfer"
                },
                {
                  "id": "0mtlcmctg-8z00dmmisds",
                  "start": 1788429285988,
                  "status": "success",
                  "end": 1788429291289,
                  "message": "vmstart"
                }
              ],
              "end": 1788429296722,
              "message": "health check"
            },
            {
              "id": "0mtlcml5b-ywe7qa02f7q",
              "start": 1788429296783,
              "status": "success",
              "end": 1788429300723,
              "result": {
                "merge": false,
                "size": 0
              },
              "message": "clean-vm"
            }
          ],
          "end": 1788429300727,
          "message": "export",
          "data": {
            "id": "19270323-baaa-4a7b-bf1f-e152ae44f263",
            "isFull": false,
            "type": "remote"
          }
        }
      ],
      "warnings": [
        {
          "message": "can't compute delta OpaqueRef:afcf0af9-64c9-845d-207c-789c9c5de78f from OpaqueRef:13ce290f-2194-280e-72c5-9e16271505df, fall back to a full"
        },
        {
          "message": "can't connect through NBD, fall back to stream export"
        },
        {
          "message": "Backup fell back to a full"
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:afcf0af9-64c9-845d-207c-789c9c5de78f"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1788429300728,
      "message": "backup VM",
      "data": {
        "id": "0618e9fe-854e-7f01-1f22-6fb1667806d4",
        "type": "VM",
        "name_label": "Salem XO-CE",
        "progress": 0
      }
    },
    {
      "id": "0mtlc4783-s6m3pnf32pl",
      "start": 1788428438931,
      "status": "success",
      "tasks": [
        {
          "id": "0mtlc478g-epd65nptnrn",
          "start": 1788428438944,
          "status": "success",
          "end": 1788428439927,
          "result": {
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        },
        {
          "id": "0mtlc486t-mq755lvlxi",
          "start": 1788428440181,
          "status": "success",
          "end": 1788428443614,
          "result": "d52e8853-1102-8bcd-b588-adbe5b411a6b",
          "message": "snapshot"
        },
        {
          "id": "0mtlc4au7-q4va6ntph6c",
          "start": 1788428443615,
          "status": "success",
          "tasks": [
            {
              "id": "0mtlc9pax-xyje1fo25lf",
              "start": 1788428695641,
              "status": "success",
              "end": 1788429704445,
              "result": {
                "size": 44032329216
              },
              "message": "transfer",
              "data": {
                "progress": 100
              }
            },
            {
              "id": "0mtlcvdy1-pxmgi1rx7tk",
              "start": 1788429707353,
              "status": "success",
              "tasks": [
                {
                  "id": "0mtlcvdzc-uodtj65grrn",
                  "start": 1788429707400,
                  "status": "success",
                  "end": 1788430481457,
                  "result": {
                    "size": 44021317632,
                    "id": "ed4d4ce7-2685-a35a-b1e4-c54a1385364e"
                  },
                  "message": "transfer"
                },
                {
                  "id": "0mtldbz96-dsndwbb93hv",
                  "start": 1788430481466,
                  "status": "success",
                  "end": 1788430486658,
                  "message": "vmstart"
                }
              ],
              "end": 1788430492516,
              "message": "health check"
            },
            {
              "id": "0mtldc7vt-oopf5wcg74e",
              "start": 1788430492649,
              "status": "success",
              "end": 1788430501065,
              "result": {
                "merge": false,
                "size": 0
              },
              "message": "clean-vm"
            }
          ],
          "end": 1788430501070,
          "message": "export",
          "data": {
            "id": "19270323-baaa-4a7b-bf1f-e152ae44f263",
            "isFull": false,
            "type": "remote"
          }
        }
      ],
      "warnings": [
        {
          "message": "can't compute delta OpaqueRef:82b55a23-b13e-df37-c9f7-11373a42c40b from OpaqueRef:9824e8f1-0199-885c-e69a-d9275cd62d2a, fall back to a full"
        },
        {
          "message": "can't connect through NBD, fall back to stream export"
        },
        {
          "message": "Backup fell back to a full"
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:82b55a23-b13e-df37-c9f7-11373a42c40b"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1788430501071,
      "message": "backup VM",
      "data": {
        "id": "64476a1a-20f5-32a4-6352-47916b2162d8",
        "type": "VM",
        "name_label": "Salem UnifiOS",
        "progress": 0
      }
    },
    {
      "id": "0mtlcmo92-ezud8mmu26",
      "start": 1788429300806,
      "status": "success",
      "tasks": [
        {
          "id": "0mtlcmoaf-ah3ohsdoy7r",
          "start": 1788429300855,
          "status": "success",
          "end": 1788429304315,
          "result": {
            "merge": false,
            "size": 0
          },
          "message": "clean-vm"
        },
        {
          "id": "0mtlcmrdk-xm266uwj1ni",
          "start": 1788429304856,
          "status": "success",
          "end": 1788429307794,
          "result": "dd0d0918-efdd-1a1d-ce94-85afbf0fd25c",
          "message": "snapshot"
        },
        {
          "id": "0mtlcmtn7-clidkdhy2ht",
          "start": 1788429307795,
          "status": "success",
          "tasks": [
            {
              "id": "0mtlcs7g8-3k59s9tshcf",
              "start": 1788429558968,
              "status": "success",
              "end": 1788430829213,
              "result": {
                "size": 43868711424
              },
              "message": "transfer",
              "data": {
                "progress": 100
              }
            },
            {
              "id": "0mtldjhbz-rfnh54dr3y",
              "start": 1788430831487,
              "status": "success",
              "tasks": [
                {
                  "id": "0mtldjhd9-g6rihy05ec7",
                  "start": 1788430831533,
                  "status": "success",
                  "end": 1788431285549,
                  "result": {
                    "size": 43857739776,
                    "id": "e3215b6b-9b5a-0d49-3392-491cc39c19e4"
                  },
                  "message": "transfer"
                },
                {
                  "id": "0mtldt7ox-r81abwrax4",
                  "start": 1788431285553,
                  "status": "success",
                  "end": 1788431290565,
                  "message": "vmstart"
                }
              ],
              "end": 1788431296314,
              "message": "health check"
            },
            {
              "id": "0mtldtg0h-8kvnbox5dg3",
              "start": 1788431296337,
              "status": "success",
              "end": 1788431297610,
              "result": {
                "merge": false,
                "size": 0
              },
              "message": "clean-vm"
            }
          ],
          "end": 1788431297611,
          "message": "export",
          "data": {
            "id": "19270323-baaa-4a7b-bf1f-e152ae44f263",
            "isFull": false,
            "type": "remote"
          }
        }
      ],
      "warnings": [
        {
          "message": "can't compute delta OpaqueRef:4b3d9930-d795-03f8-c887-f8b7d2a95524 from OpaqueRef:894ead1a-dae6-f156-d8c7-c7f5ba2200ef, fall back to a full"
        },
        {
          "message": "can't connect through NBD, fall back to stream export"
        },
        {
          "message": "Backup fell back to a full"
        }
      ],
      "infos": [
        {
          "message": "will delete snapshot data"
        },
        {
          "data": {
            "vdiRef": "OpaqueRef:4b3d9930-d795-03f8-c887-f8b7d2a95524"
          },
          "message": "Snapshot data has been deleted"
        }
      ],
      "end": 1788431297611,
      "message": "backup VM",
      "data": {
        "id": "ddf296c8-20a0-6aa9-0d74-ebc029d0d1b3",
        "type": "VM",
        "name_label": "Salem UnifiOS",
        "progress": 0
      }
    }
  ],
  "end": 1788431297621,
  "infos": [
    {
      "data": {
        "vms": [
          "0618e9fe-854e-7f01-1f22-6fb1667806d4",
          "64476a1a-20f5-32a4-6352-47916b2162d8",
          "ddf296c8-20a0-6aa9-0d74-ebc029d0d1b3"
        ]
      },
      "message": "vms"
    }
  ]
}

@florent I sent pm with support tunnel. Are you able to apply patch or will patch be pushed live soon?

@florent Yes you may deploy a patch thought the support tunnel.

@pierrebrunet

v6 dahsboard is now connected and working.

Both Proxies are connected and vms are visible from the remote pools...

Backup job still fails. same error.

How and why does it have to many conections? To where XOA? to the Proxy? to the Backup remote?

I did recently have two seperate issues each different at both remote sites recenlty but the inital backup error seemed similar to the OP. And all backups worked until recent updates.

Edit -

This is soemthing new....

THis is still an active tranfer when the job reported failed....
Reran backup again. It does appear to be doing the backup but it stil fails after creating snapshots or around that point. Once the job fails the backup is still happening and health checks happening.

@pierrebrunet

My post this morning after that update.

I see the same error in task view prior to the update but under the scheduled task as it was the actual backup job not me tring.

API call: schedule.runSequence

I am starting a fresh backup instead of retrying the last failed backup.

Edit - From fresh backup....

Also i have support ticket up about v6 dash board not loading and support tunnel open if you want to look. Ticket#7763637

{
  "data": {
    "mode": "delta",
    "reportWhen": "always",
    "backupReportTpl": "mjml",
    "hideSuccessfulItems": true
  },
  "id": "1788347692784",
  "jobId": "bb0adfc7-93b9-45c9-a90f-7aa327f95a05",
  "jobName": "Tilton - Delta Backups",
  "message": "backup",
  "proxyId": "9d22fb88-8bdd-44d3-a3a3-9ff5401f95df",
  "scheduleId": "249a8339-da2b-4031-b85d-dc3d9526efa5",
  "start": 1788347692784,
  "status": "failure",
  "end": 1788347752961,
  "result": {
    "message": "fetch failed",
    "name": "TypeError",
    "stack": "TypeError: fetch failed\n    at node:internal/deps/undici/undici:14976:13\n    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)\n    at cb (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:355:42)\n    at tryCatcher (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromise0 (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:649:10)\n    at Promise._settlePromises (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/promise.js:725:18)\n    at _drainQueueStep (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:93:12)\n    at _drainQueue (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues (/usr/local/lib/node_modules/xo-server/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (node:internal/timers:484:21)\n    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
  }
}