@johnnezero The full HTML versions will render much better. The PDF conversion is less than perfect. iIll try to get those uploaded, as well.

@simonp said: @acebmxer Hi, Thanks to your help we were able to identify an issue with Redis that we think is the source of the v6 dashboard loading issue. Could you try and checkout the fix_redis_encryption_issue branch, rebuild xo and restart ? This should solve the 401 issues. Will continue to test but i think that did resolve it. I am hesitant from the logs after update. I have refreshed the page 20 times and still loads correctly no glitches. [INFO] Detected package manager: apt [INFO] Checking for updates... [INFO] Installed commit: fc4d34b354e3 [INFO] Remote commit: e5210ad478b8 [INFO] New version available. Proceeding with update... [INFO] Checking for active Xen Orchestra tasks before updating... [INFO] Using authentication token from xo-config.cfg... [INFO] Querying active tasks as authentication token... [SUCCESS] No active tasks found. Proceeding with update... [INFO] Task check performed by: authentication token [INFO] Stopping xo-server service... [INFO] Preserving API token ("xo-installer") [INFO] Flushed browser session token (client_id y7iutpmvys7) [INFO] Deleted stale collection index: xo:token::indexes [INFO] Preserving API token ("no description") [INFO] Flushed 1 session token(s) — users will need to log in again [INFO] Preserved 2 API token(s) — third-party integrations unaffected [INFO] Creating backup of current installation... [SUCCESS] Backup created: /opt/xo-backups/xo-backup-20260527_094604 [INFO] Cleaning old backups (keeping 5)... [SUCCESS] Old backups cleaned [INFO] Pulling latest changes... Updated 3625 paths from the index remote: Enumerating objects: 182, done. remote: Counting objects: 100% (91/91), done. remote: Compressing objects: 100% (31/31), done. remote: Total 182 (delta 63), reused 76 (delta 58), pack-reused 91 (from 1) Receiving objects: 100% (182/182), 103.02 KiB | 6.87 MiB/s, done. Resolving deltas: 100% (104/104), completed with 31 local objects. From https://github.com/vatesfr/xen-orchestra fc4d34b35..dae64be80 master -> origin/master + 4c172c569...d4ed13417 feat/sdn-controller/improve-startup-perf -> origin/feat/sdn-controller/improve-startup-perf (forced update) * [new branch] fix_qa_test -> origin/fix_qa_test * [new branch] fix_redis_encryption_issue -> origin/fix_redis_encryption_issue * [new branch] lite/warning-before-leaving-XOA-deploy -> origin/lite/warning-before-leaving-XOA-deploy * [new branch] mel/acl/pool-RBAC-check -> origin/mel/acl/pool-RBAC-check 2ca9e58d1..24506cfa6 mel/acls/pbds-RBAC-check -> origin/mel/acls/pbds-RBAC-check * [new branch] refacto/migrate-to-register-rest-routes -> origin/refacto/migrate-to-register-rest-routes + 761fb851c...766311798 xo6/general-tab-traffic-rule-vif -> origin/xo6/general-tab-traffic-rule-vif (forced update) de5fc9cc4..a10ed619a xo6/traffic-rules-vif -> origin/xo6/traffic-rules-vif branch 'fix_redis_encryption_issue' set up to track 'origin/fix_redis_encryption_issue'. Switched to a new branch 'fix_redis_encryption_issue' [INFO] Node.js v24.15.0 satisfies configured version 24 — no runtime change needed [INFO] Installing Node.js 24... [INFO] Node.js 24 requirement satisfied (installed: v24.15.0) [INFO] npm is available: 11.12.1 [INFO] Installing Yarn... [INFO] Yarn is already installed: 1.22.22 [INFO] Building Xen Orchestra (this may take a while)... [INFO] Sufficient swap space available: 4095MB [INFO] Clearing build cache for clean rebuild... [INFO] Moderate memory detected (5399MB RAM + 4095MB swap). Limiting build concurrency to 2. yarn install v1.22.22 (node:87888) [DEP0169] DeprecationWarning: `url.parse()` behavior is not standardized and prone to errors that have security implications. Use the WHATWG URL API instead. CVEs are not issued for `url.parse()` vulnerabilities. (Use `node --trace-deprecation ...` to show where the warning was created) [1/5] Validating package.json... [2/5] Resolving packages... success Already up-to-date. $ husky install husky - Git hooks installed Done in 0.57s. yarn run v1.22.22 $ TURBO_TELEMETRY_DISABLED=1 turbo run build --filter xo-server --filter xo-server-'*' --filter xo-web --filter @xen-orchestra/web --filter @xen-orchestra/backup-archive --concurrency=2 ╭───────────────────────────────────────────────────────────────────────────╮ │ │ │ Update available v2.9.12 ≫ v2.9.14 │ │ Changelog: https://github.com/vercel/turborepo/releases/tag/v2.9.14 │ │ Run "yarn dlx @turbo/codemod@latest update" to update │ │ │ │ Follow @turborepo for updates: https://x.com/turborepo │ ╰───────────────────────────────────────────────────────────────────────────╯ • turbo 2.9.12 WARNING no caches are enabled • Packages in scope: @xen-orchestra/backup-archive, @xen-orchestra/web, xo-server, xo-server-audit, xo-server-auth-github, xo-server-auth-google, xo-server-auth-ldap, xo-server-auth-oidc, xo-server-auth-saml, xo-server-backup-reports, xo-server-ipmi-sensors, xo-server-load-balancer, xo-server-netbox, xo-server-openmetrics, xo-server-perf-alert, xo-server-sdn-controller, xo-server-test-plugin, xo-server-transport-email, xo-server-transport-icinga2, xo-server-transport-nagios, xo-server-transport-slack, xo-server-transport-xmpp, xo-server-usage-report, xo-server-web-hooks, xo-web • Running build in 25 packages • Remote caching disabled (remote cache requested — set TURBO_TOKEN and TURBO_TEAM, or run "turbo login" and "turbo link") Tasks: 35 successful, 35 total Cached: 0 cached, 35 total Time: 48.025s Done in 48.47s. [SUCCESS] Xen Orchestra built successfully [WARNING] XO 6 web UI (@xen-orchestra/web/dist) was not built — browser will use XO 5 UI at /v5. [WARNING] This may be caused by unmet peer dependencies (pinia 3.x, vue-router 5.x). [WARNING] Access your XO instance at: https://<host>/v5 until the upstream issue is resolved. [INFO] Creating systemd service... [SUCCESS] Systemd service created and enabled [INFO] Configuring sudo for xo-service (mount/umount/findmnt)... [SUCCESS] Sudo configured for xo-service (mount, umount, findmnt) [INFO] Applying security hardening... [INFO] Starting xo-server service... [INFO] Waiting for Xen Orchestra to become ready (up to 60s)... [INFO] Not ready yet (attempt 1/10), retrying in 6s... [SUCCESS] Xen Orchestra is ready (HTTPS on port 443) [SUCCESS] Update completed successfully! [INFO] New commit: e5210ad478b8 [SUCCESS] Xen Orchestra built successfully [WARNING] XO 6 web UI (@xen-orchestra/web/dist) was not built — browser will use XO 5 UI at /v5. [WARNING] This may be caused by unmet peer dependencies (pinia 3.x, vue-router 5.x). [WARNING] Access your XO instance at: https://<host>/v5 until the upstream issue is resolved. [image: 1779875491511-screenshot_20260527_055115-resized.png] [image: 1779875500570-screenshot_20260527_055051-resized.png]

Thanks for that information. I will make this message short because @stormi is busy but I want to say thanks to Vates and XCP-ng for all their work done to support Windows on the Xen platform. This includes TPM2 and secure boot support and Microsoft-signed pv drivers. Well done!

Still, something to fix

@Danp I was going to do the upgrade over the December / January Christmas break in Australia but that is when this issue became apparent so we didn't want to be making major changes until it was resolved. I note @anthoineb noted the first report was late August / September 2025 which was just before 8.2.1 reached EOL, so it would be a shame not to have a fix for it tested on 8.2.1 as it seems an upgrade to 8.3 would be unwise while we still have this issue?

Dug a little deeper. For a VM where the disks are not shown the following XO API call fails: /rest/v0/vms/a519e879-3971-9210-51b6-7df14336e7b7/vdis { "error": "no such VDI ac37700d-3157-4df7-b8e8-e1799a994591", "data": { "id": "ac37700d-3157-4df7-b8e8-e1799a994591", "type": [ "VDI" ] } } Also the VDI cannot be retrieved over the XO API: /rest/v0/vms/a519e879-3971-9210-51b6-7df14336e7b7 ... "$VBDs": [ "4ea8a3cd-0d1b-dc60-4d9c-fd70e060f06c", "9f4ca686-9fc2-35a9-c3e9-c871c9f68aba" ], ... /rest/v0/vbds/9f4ca686-9fc2-35a9-c3e9-c871c9f68aba { "type": "VBD", "attached": false, "bootable": false, "device": "xvda", "is_cd_drive": false, "position": "0", "read_only": false, "VDI": "ac37700d-3157-4df7-b8e8-e1799a994591", "VM": "a519e879-3971-9210-51b6-7df14336e7b7", "id": "9f4ca686-9fc2-35a9-c3e9-c871c9f68aba", "uuid": "9f4ca686-9fc2-35a9-c3e9-c871c9f68aba", "$pool": "93d361b7-f549-53b7-a3aa-c9695bf0abe4", "$poolId": "93d361b7-f549-53b7-a3aa-c9695bf0abe4", "_xapiRef": "OpaqueRef:1d424d94-f540-2eb4-9e52-2a9b21ec0a19" } /rest/v0/vdis/ac37700d-3157-4df7-b8e8-e1799a994591 { "error": "no such VDI ac37700d-3157-4df7-b8e8-e1799a994591", "data": { "id": "ac37700d-3157-4df7-b8e8-e1799a994591", "type": "VDI" } } However the VDI can be listed using the xe cli: $ xe vm-list uuid=a519e879-3971-9210-51b6-7df14336e7b7 uuid ( RO) : a519e879-3971-9210-51b6-7df14336e7b7 name-label ( RW): XXX power-state ( RO): halted $ xe vbd-list vm-uuid=a519e879-3971-9210-51b6-7df14336e7b7 uuid ( RO) : 4ea8a3cd-0d1b-dc60-4d9c-fd70e060f06c vm-uuid ( RO): a519e879-3971-9210-51b6-7df14336e7b7 vm-name-label ( RO): XXX vdi-uuid ( RO): <not in database> empty ( RO): true device ( RO): xvdd uuid ( RO) : 9f4ca686-9fc2-35a9-c3e9-c871c9f68aba vm-uuid ( RO): a519e879-3971-9210-51b6-7df14336e7b7 vm-name-label ( RO): XXX vdi-uuid ( RO): ac37700d-3157-4df7-b8e8-e1799a994591 empty ( RO): false device ( RO): xvda $ xe vdi-list uuid=ac37700d-3157-4df7-b8e8-e1799a994591 uuid ( RO) : ac37700d-3157-4df7-b8e8-e1799a994591 name-label ( RW): XXX Disk 0 name-description ( RW): Created by XO sr-uuid ( RO): 977b7e63-bb84-57b2-3e0d-206afea553bf virtual-size ( RO): 34359738368 sharable ( RO): false read-only ( RO): false Seems almost like something changed in the XCP-ng API which XO cannot consume.

@tjkreidl My pleasure.

@pkgw I tested it with a cluster size of 2 megabytes. I got similar results to those with the default size.

@florent said: @acebmxer back to work thank you for yor patience and help on this. I feel that it's not the same issue , with abrupt increase W will try our best to also fix this one Yes i replied to ticket also.... Yes you can do what is needed to XOA. Just looked at memory and it dropped.... [image: 1779875710969-screenshot_20260527_055458.png]

Ah excellente nouvelle Je passe le sujet en résolu !

@MathieuRA I noticed you merged https://github.com/vatesfr/xen-orchestra/pull/9787 I just tried it. And it does seem to fix my original issue! Thank you! I am always impressed by you guys. Making testing and reporting upstream (to you guys) a good experience! Elise-FZI opened this pull request in vatesfr/xen-orchestra closed fix(xo6): remove dev routes from prod #9787

@olivierlambert Thanks again for your input and recomendations! I'll verify that this is solved by having the LUN expanded to 8GB instead. Afterwards I'll mark your answer as the solution!

@vlamincktr XO PROXY from source is pretty reliable at no cost either use @acebmxer script or @ronivay here is a quick tuto on an ubuntu VM https://omnibox.huducloud.com/shared_article/QJ9y1bRSPj9VTbWp6NKaV7yn/installation-xoa-a-partir-des-sources-github-ronivay first part is XO CE, second part is XO PROXY CE beware as you delegate some jobs to XO PROXY, to ever upgrade XO PROXY when you upgrade XOA, so that they have the same backup mechanisms/code

I updated XO again - just thought I would add the API error details: Error: ENOENT: no such file or directory, open '/run/xo-server/mounts/d806298f-85cf-4565-bb7e-9fad479b941f/xo-vm-backups/645b7426-0b09-e192-40ea-4f6c36e35c23/vdis/8190942e-8b0d-4044-95b6-6b8d91a463f7/7b86306a-fffa-4528-b946-d0bc60dcb609/20260527T000011Z.vhd' From: at NfsHandler.addSyncStackTrace (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:21:26) at NfsHandler._openFile (/opt/xen-orchestra/@xen-orchestra/fs/dist/local.js:154:35) at /opt/xen-orchestra/@xen-orchestra/fs/dist/utils.js:29:26 at new Promise (<anonymous>) at NfsHandler.<anonymous> (/opt/xen-orchestra/@xen-orchestra/fs/dist/utils.js:24:12) at loopResolver (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:83:46) at new Promise (<anonymous>) at loop (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:85:22) at NfsHandler.retry (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:87:10) at NfsHandler._openFile (/opt/xen-orchestra/node_modules/promise-toolbox/retry.js:103:18) The error in XO remains: Error: ENOENT: no such file or directory, stat '/run/xo-server/mounts/d806298f-85cf-4565-bb7e-9fad479b941f/xo-vm-backups/645b7426-0b09-e192-40ea-4f6c36e35c23/vdis/8190942e-8b0d-4044-95b6-6b8d91a463f7/c1e95b3b-cca1-457a-a0d4-0a37924c8170/20260527T181636Z.alias.vhd' What's bizarre is that I absolutely nuked the path /run/xo-server/mounts/d806298f-85cf-4565-bb7e-9fad479b941f/xo-vm-backups with rm -rf /run/xo-server/mounts/d806298f-85cf-4565-bb7e-9fad479b941f/xo-vm-backups so that there was nothing left behind. After starting the job, XO rebuilds the path fine: ls -al /run/xo-server/mounts/d806298f-85cf-4565-bb7e-9fad479b941f/xo-vm-backups/645b7426-0b09-e192-40ea-4f6c36e35c23/vdis/8190942e-8b0d-4044-95b6-6b8d91a463f7/c1e95b3b-cca1-457a-a0d4-0a37924c8170/ total 13 drwxr-xr-x 3 3001 3001 3 May 27 18:16 . drwxr-xr-x 4 3001 3001 4 May 27 18:16 .. drwxr-xr-x 2 3001 3001 2 May 27 18:18 data but it is correct, the file 20260527T181636Z.alias.vhd never appears. To be, this proves the NFS is 100% fine... XO can mount, test, write, and delete from the mount just fine. But I don't know why it doesn't create the *alias.vhd file.

@florent said: where does this disk comes from From my Redhat 10 @florent said: if you have access to your SR from the outside, you can also put the qcow2 file directly I create a VM to be a NFS to access the 3 HDs, the qcow disks are on the Redhat 10 that I was trying to import from. Do you mean I put the qcow disks on one of the HDs and access them when I create a VM?

@MajorP93 that’s fine - I never use ballooning anyway so I guess I am covered good

@tonyp90 great, thanks !

the last rewrite of the stream processing ( spring 2025 ) focused on stability and memory footprint, and , on a standard cpu, it tops at around 300MB/s per backup job. Your benchmarks are very interesting, and they confirm most of it. this limit was not really an issue since, in most case the xapi was limiting around 100MB/s per disk , but it will be more a more visible limit Note that master have some fixes on the memory usage (not related to backups) That's why we have started an internal workforce focused on performance, with all the teams from the kernel to the backups, including storage, network and xapi. If I can brag a little : [image: 1779106650898-afd7b59b-a4f0-4a92-88ee-2c7ba52d18bf-image.jpeg] i9 , nvme disk , backup to a nvme disk in passthrough, xoa and vm are on the same host, so it's quite far from real world data, but it shows where the limit is