1 out of 3 pools at work failed rolling pool update.

When I try to put host 2 into maintence mode to finish updates i get this error...

Support Ticket - Ticket#7763405

host.setMaintenanceMode
{
  "id": "60701efd-089c-4822-97c0-1a1057f3f9aa",
  "maintenance": true
}
{
  "code": "VM_REQUIRES_SR",
  "params": [
    "OpaqueRef:841e7606-d545-cb71-f67c-e48b5114a1e8",
    "OpaqueRef:0f356ee4-62f7-9608-8be5-df68d9a5cbb2"
  ],
  "task": {
    "uuid": "a24f5565-b2cb-f392-8095-1beceb7a4a17",
    "name_label": "Async.host.evacuate",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260828T19:57:29Z",
    "finished": "20260828T19:57:29Z",
    "status": "failure",
    "resident_on": "OpaqueRef:43c121e7-4ba4-193b-1e41-08c0f7e15690",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "VM_REQUIRES_SR",
      "OpaqueRef:841e7606-d545-cb71-f67c-e48b5114a1e8",
      "OpaqueRef:0f356ee4-62f7-9608-8be5-df68d9a5cbb2"
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 629))((process xapi)(filename hashtbl.ml)(line 159))((process xapi)(filename hashtbl.ml)(line 165))((process xapi)(filename hashtbl.ml)(line 170))((process xapi)(filename ocaml/xapi/xapi_host.ml)(line 625))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "VM_REQUIRES_SR(OpaqueRef:841e7606-d545-cb71-f67c-e48b5114a1e8, OpaqueRef:0f356ee4-62f7-9608-8be5-df68d9a5cbb2)",
  "name": "XapiError",
  "stack": "XapiError: VM_REQUIRES_SR(OpaqueRef:841e7606-d545-cb71-f67c-e48b5114a1e8, OpaqueRef:0f356ee4-62f7-9608-8be5-df68d9a5cbb2)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1229:24)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1263:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1253:12)
    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1460:14)"
}

SR are not connected to master host after reboot from updates. When try to connect them i get error...

pbd.connect
{
  "id": "e96ec5d8-c70f-c877-4263-f2cf763e7048"
}
{
  "code": "SR_BACKEND_FAILURE_108",
  "params": [
    "",
    "Unable to detect an NFS service on this target.",
    ""
  ],
  "task": {
    "uuid": "5574bfe5-1874-35f8-f53b-c9fc1abaebb5",
    "name_label": "Async.PBD.plug",
    "name_description": "",
    "allowed_operations": [],
    "current_operations": {},
    "created": "20260828T20:03:19Z",
    "finished": "20260828T20:03:22Z",
    "status": "failure",
    "resident_on": "OpaqueRef:43c121e7-4ba4-193b-1e41-08c0f7e15690",
    "progress": 1,
    "type": "<none/>",
    "result": "",
    "error_info": [
      "SR_BACKEND_FAILURE_108",
      "",
      "Unable to detect an NFS service on this target.",
      ""
    ],
    "other_config": {},
    "subtask_of": "OpaqueRef:NULL",
    "subtasks": [],
    "backtrace": "(((process xapi)(filename ocaml/xapi-idl/storage/storage_interface.ml)(line 455))((process xapi)(filename src/lib/idl.ml)(line 558))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 148))((process xapi)(filename lib/backtrace.ml)(line 251))((process xapi)(filename ocaml/xapi/xapi_pbd.ml)(line 196))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 6061))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))"
  },
  "message": "SR_BACKEND_FAILURE_108(, Unable to detect an NFS service on this target., )",
  "name": "XapiError",
  "stack": "XapiError: SR_BACKEND_FAILURE_108(, Unable to detect an NFS service on this target., )
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at default (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_getTaskResult.mjs:13:29)
    at Xapi._addRecordToCache (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1229:24)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1263:14
    at Array.forEach (<anonymous>)
    at Xapi._processEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1253:12)
    at Xapi._watchEvents (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/index.mjs:1460:14)"
}

Update - bad port on switch.......

Update 2 - After updating synology to latest version i am no longer able to mount nfs 4.1 shares. I can not create one using 4.1. I will let me create with 4.0 not 4.1 How to remount NFS that were mounted as 4.1 now as 4.0?

ALl VMS down at this location currently.

Update 3 - Resolved... Synology update broke NFS 4.1. I had to ssh into synology and edit /etc/nfs/syno_nfs_conf and add nfs_minor_ver_enable=1 Now SR's mounted.

Update 4 - Found Patch notes about NFS version change. -
Version: 7.0.1-42218
NFSv4.1 and related advanced functions (multipathing) are no longer supported on Synology NAS models with the following package architectures: Alpine, Alpine4k, armada38x, Avoton, Braswell, Bromolow, Cedarview, Grantley, Monaco. If you had previously enabled NFSv4 and NFSv4.1, the maximum NFS protocol is set to NFSv4 by default after the update. For more information about the package architecture of your model, please refer to this article.

Our unit - RS1219+ Intel Atom C2538 Avoton

@gleh

Installed...

Updated:
  blktap.x86_64 0:3.55.5-9.4.xcpng8.3                forkexecd.x86_64 0:26.1.16-1.2.xcpng8.3                
  message-switch.x86_64 0:26.1.16-1.2.xcpng8.3       qcow-stream-tool.x86_64 0:26.1.16-1.2.xcpng8.3         
  rrdd-plugins.x86_64 0:26.1.16-1.2.xcpng8.3         sm.x86_64 0:3.2.12-23.5.xcpng8.3                       
  sm-cli.x86_64 0:26.1.16-1.2.xcpng8.3               sm-fairlock.x86_64 0:3.2.12-23.5.xcpng8.3              
  squeezed.x86_64 0:26.1.16-1.2.xcpng8.3             varstored-guard.x86_64 0:26.1.16-1.2.xcpng8.3          
  vhd-tool.x86_64 0:26.1.16-1.2.xcpng8.3             wsproxy.x86_64 0:26.1.16-1.2.xcpng8.3                  
  xapi-core.x86_64 0:26.1.16-1.2.xcpng8.3            xapi-nbd.x86_64 0:26.1.16-1.2.xcpng8.3                 
  xapi-rrd2csv.x86_64 0:26.1.16-1.2.xcpng8.3         xapi-storage-script.x86_64 0:26.1.16-1.2.xcpng8.3      
  xapi-tests.x86_64 0:26.1.16-1.2.xcpng8.3           xapi-xe.x86_64 0:26.1.16-1.2.xcpng8.3                  
  xcp-networkd.x86_64 0:26.1.16-1.2.xcpng8.3         xcp-rrdd.x86_64 0:26.1.16-1.2.xcpng8.3                 
  xenopsd.x86_64 0:26.1.16-1.2.xcpng8.3              xenopsd-cli.x86_64 0:26.1.16-1.2.xcpng8.3              
  xenopsd-xc.x86_64 0:26.1.16-1.2.xcpng8.3

So i was trying out a new Linux distro Umarchy and had issues with the screen resolution.

Looks like this can be done by creating new vm templates with this UEFI setting preset.

Is that something Vates and work on when pushing new templates?

Checked the official docs — short version: there's no supported way to change the OVMF default. It's per-VM only, done inside the guest's UEFI setup, and XCP-ng/XO expose no pool setting, template field, or xe parameter for it.

What the official docs actually say
docs.xcp-ng.org/vms/ has a "Manage screen resolution" section, but it only covers the OS layer:

BIOS VM — set it in the guest OS (Display settings).
UEFI VM — install the XCP-ng Standard VGA Display Driver (part of the Windows Guest Tools package); after that the guest OS controls resolution normally.
The OVMF Platform Configuration menu itself isn't documented there. On the forum, Vates support (Darkbeldin) answered the exact question you're asking with "to my knowledge it's not possible inside the template." That's still the state of things.

Setting it per VM
Device Manager → OVMF Platform Configuration → Change Preferred → pick resolution → F10 to save → Esc back out → Reset (a full reboot; it does not apply to the current boot).

Getting into setup is the annoying part since boot wait is 0s — either spam Esc at the Tianocore logo, or from the GRUB screen press c and type fwsetup.

Making new VMs come up at 1024x768+
The practical workaround, since the setting lives in the VM's UEFI NVRAM (varstored) and NVRAM is part of the VM record:

Build one VM, set the OVMF preferred resolution, shut it down.
Convert it to a template.
VMs cloned from that template inherit the NVRAM, including the resolution.
Caveats: this won't help VMs created from XCP-ng's built-in templates, and there are reports of guest UEFI NVRAM not persisting in some cases, so verify on a clone before rolling it out.

Related knob worth checking first
Separate from the OVMF preference, the emulated adapter caps what modes are available:

xe vm-param-set uuid=$UUID platform:vga=std
xe vm-param-set uuid=$UUID platform:videoram=16
In XO these are the VGA toggle and Video RAM field on the VM's Advanced tab. 16 MiB is the max and is what people use for 1080p and above — with the default low videoram the higher modes may not even be offered in the OVMF menu.

Sources:

Virtual Machines (VMs) | XCP-ng Documentation
Set default resolution for UEFI | XCP-ng forum
VGA on Xen Orchestra | XCP-ng forum
Guest VM UEFI NVRAM not saved / not persistent | XCP-ng forum
How to set screen size for UEFI VMs on XCP-NG | TechOverflow

v0.4.0 Release - https://github.com/acebmxer/install_xen_orchestra/releases

0.4.0 - 2026-08-23
Changed
The cloud image is staged on the pool master by default instead of being streamed. Staging is the only path that can resume a broken download, retry a transient failure, and check the downloaded size before anything reaches the VDI; streaming can do none of those, because a pipe already feeding a fixed-Content-Length PUT cannot be rewound. On a link that drops the occasional TLS record — which any multi-gigabyte transfer eventually meets — streaming failed every attempt while staging rode it out. Streaming is now what it should always have been: the fallback for a host without the few gigabytes of scratch space staging needs.
Fixed
A stalled streaming import no longer has to be interrupted by hand. When the download end died, the upload sat waiting for a response XAPI would never send. --speed-time could not help — by that point curl is waiting, not transferring, so the speed meter has stopped ticking and only --max-time 3600 would eventually fire. The two transfers now run as separate processes with the download's exit status watched, and the upload is killed the moment it fails.

A pool master short on scratch space no longer fails the whole deploy. The staged path signalled "no room" with a plain non-zero return, which under set -e took the script down before the fallback could be reached.

A failed cloud-image download no longer looks like a successful import. The streaming import piped one curl into another, and the remote shell reported only the upload side's exit status. A download that died partway — a transient SSL_read ... bad record mac on a 3 GB transfer is the usual cause — therefore produced a truncated disk that the deploy reported as imported, and a VM that booted into a corrupt filesystem. The pipeline now runs under bash -o pipefail.

A broken image download no longer hangs the deploy for an hour. When the download end died, the upload curl had already promised XAPI an exact Content-Length and sat waiting to send bytes that were never coming, with XAPI waiting alongside it until --max-time 3600 expired — the visible symptom being a XAPI task frozen at partial progress and a script that had to be interrupted. Both ends now abort after 60s below 1 KiB/s.

The staged image download resumes instead of starting over. It now uses -C - with --retry 5 --retry-delay 3 --retry-all-errors, so the transient TLS failures that a multi-gigabyte single-connection download eventually hits are ridden out rather than failing the deploy. The streaming path deliberately does not retry: curl re-issues from byte 0, and piped into a fixed-Content-Length PUT those bytes would be appended to the ones already sent, corrupting the image while appearing to succeed.

A short staged download is refused rather than imported. The file's size is now checked against the length the server advertised before anything is written into the VDI.

The staged download reports progress. It was silent for several minutes on the longest step of the deploy, which reads as a hang worth killing.

A failed deploy names the VM it left behind, with the xe vm-destroy command, instead of leaving a half-built VM to be rediscovered later in the pool's VM list. Nothing is destroyed automatically.

--update/--reconfigure/--rebuild no longer abort on a root install. Reading User= from a systemd unit that has no such line (which is what a root install looks like) failed the pipeline under set -o pipefail and took the script down before the fallback could run.

Deploy prompts validate values, not just their shape. 999.999.999.999 was accepted as an address and 70000 as a port; both were only rejected after the VM existed, by an unreachable guest or by the installer inside it.

Prompted settings are no longer lost when the base config omits the key. The generated xo-config.cfg was patched with sed, which silently does nothing for a key that is not there — so the VM installed on the default while the summary showed the value you typed. Missing keys are now appended.

An $EDITOR with arguments works. code --wait passed the availability check and then failed with "No such file", since the whole string was treated as one executable path.

Values edited into the config are validated. An unusable port or branch was silently ignored, leaving the summary showing one thing and the VM installing another.

Troubleshooting commands point at a key that still exists. The ssh -i lines printed on a failed install and on a non-200 health check named the temporary key, which the exit trap had already deleted.

A second VM with the same hostname no longer overwrites the first one's SSH key, which was the only way into that machine.

The disk-space check before staging an image on the pool master is derived from the image's actual size instead of a hard-coded 4 GiB, which rejected small images and let large ones fill /var/tmp mid-download.

tests/probe-xapi-deploy.sh acquires its XAPI session from the pool master, the way --deploy does, so a firewall that blocks port 443 from your workstation no longer skips the HTTP transport probes that matter. It also validates --host, --user, --sr, --image and --payload-mb before they reach a shell, drops the eval in the workstation-side transport, and exits non-zero when any probe failed rather than whenever one transport worked.

The menu example in the README and the layout comment above MENU_NAMES described the old fixed 5/4/centered grid; with ten items the menu draws five entries in each column.

Security
The cloud image is verified against its published checksum. The size check catches a download that was cut short; it cannot catch one that arrived complete from the wrong place, because a substituted image has a perfectly consistent Content-Length. The staged image is now checked against the SHA512SUMS its origin publishes beside it — which is what Debian ships — and a mismatch aborts before anything reaches the disk. An origin that publishes no sums warns and continues, so a custom XO_DEPLOY_IMAGE_URL keeps working. Set XO_DEPLOY_IMAGE_SHA512 to require a specific digest instead: that makes the check mandatory, aborting rather than continuing unverified, and refuses the streaming import outright because a pipe fed straight into the VDI leaves no file to hash. Fetching sums over the same connection as the image is not a detached signature — it defends against a bad mirror or a stale cache, not an attacker holding the TLS session for both requests.
A pinned pool-master fingerprint is now enforced instead of advised. deploy_verify_host_key fingerprinted the host key and then returned success on every path that could not complete the check — so with XO_DEPLOY_POOL_FINGERPRINT set, a ssh-keyscan that timed out meant the host password was sent to whatever answered on that address, which is exactly what pinning exists to prevent and the easiest outcome for an on-path attacker to arrange. A pin that cannot be checked is now a hard failure.
The verified host key is bound to the connection that carries the password. The scanned key was fingerprinted, shown, and then discarded, while dom0_exec connected with StrictHostKeyChecking=accept-new against the default known_hosts — verifying one transaction and trusting another. Nothing stopped a different key, or the host's RSA key when the ED25519 one had been displayed, being accepted at connect time. The whole scan is now pinned into a run-scoped known_hosts that dom0_exec enforces with StrictHostKeyChecking=yes, the same way deploy_wait_for_guest already treated the guest.
A hostile pool master can no longer run commands on the workstation. The free-space probe in deploy_import_vdi_staged fed the host's reply straight into (( )), which expands an array subscript before evaluating it — so an answer of PATH[$(...)] executed locally rather than being rejected. It is now checked against ^[0-9]+$ first, matching the guards already applied to size and got in the same function. Note that set -euo pipefail does not cover this: set -u blocks only the unbound-variable form of the payload. This mattered more after staging became the default import path, because the probe went from rarely reached to running on every deploy.
The pool master's root password is no longer visible in ps. Three calls predating dom0_exec still used sshpass -p "$HOST_PASSWORD", putting the password in the process list where any other user on the workstation could read it. They now use sshpass -e with $SSHPASS, as dom0_exec does.
The admin password hash is kept out of XO_DEBUG=1 output. deploy_harden_guest_sudo and deploy_build_config_drive were missing the local - / set +x guard the rest of the script uses, so the hash was printed by xtrace. Previously masked on automated runs only because --non-interactive left the hash empty; requiring a password made it reachable on every deploy.
Revoking the deployment key can no longer empty authorized_keys. A grep failure — no space for the temporary file, an unreadable source — was swallowed by || true and the empty result written back, taking the operator's own key with it. grep's "nothing matched" (a legitimate empty result) is now distinguished from a real error, which aborts and leaves the file untouched.
The streaming import's FIFO is created inside a private directory. mktemp -u returns a name without creating anything, leaving a window in dom0's world-writable /tmp. The FIFO now lives in a mktemp -d directory.
DSA public keys are rejected. ssh-dss was accepted by deploy_load_pubkey, but OpenSSH has refused DSA since 7.0 and removed it in 9.8, so it only installed a key that silently never worked.
The cloud-init cache scrub covers cloud-config.txt. The rendered config holds hashed_passwd just as the raw user-data does; only the latter was being redacted.
The deployment SSH key is destroyed at the end of a deploy. It used to be...

Update V0.3.0

Deploying to a New VM (--deploy)
If you don't already have a Linux VM to install into, --deploy builds one for you. Run it from your own workstation — it is the only operation in this script that does not run on the machine Xen Orchestra ends up on:

git clone https://github.com/acebmxer/install_xen_orchestra.git
cd install_xen_orchestra
./install-xen-orchestra.sh --deploy

It will ask for your pool master's address and root password, let you pick a storage repository and network from what the pool actually has, then ask for the VM's size, admin account (optionally with a password), where to clone this repository inside the guest, and its static address. From there it:

Creates the VM and streams a stock Debian 13 cloud image from cloud.debian.org straight into its disk. The download runs on the pool master, so the 3 GB never crosses your workstation's link and never lands on dom0's root filesystem.
Attaches a cloud-init config drive that creates your admin user, installs a freshly generated SSH key, applies the static address, and clones this repository into the guest — either /opt/install_xen_orchestra or /home/<admin>/install_xen_orchestra, whichever you pick.
SSHes in and runs --install --non-interactive, streaming the output to your terminal so you see the build as it happens.
Verifies XO answers on /signin, then detaches and destroys the cloud-init config drive — it has served its purpose, and it holds the admin password hash. If the guest refuses the hot-unplug, you get the xe commands to remove it by hand rather than a failed deploy.
Changing settings the prompts don't cover

--deploy only asks about the HTTP/HTTPS ports and the git branch. Everything else the VM is installed with — INSTALL_DIR, SERVICE_USER, NODE_VERSION, SSL and backup paths — comes from a base config, and you get to see it before anything is created on the pool:

The base is sample-xo-config.cfg from the repo. If you also keep an xo-config.cfg beside the script, you are asked which of the two the VM should start from. (Check its paths first — they were written for whatever machine it came from, not a fresh Debian VM.)
Right after the prompts, --deploy offers to open the generated config in your editor ($EDITOR/$VISUAL, else the base config's PREFERRED_EDITOR, else nano/vim/vi). Save and quit and the VM is built with exactly what you left there.
The edit happens on a throwaway copy in a temp directory, so neither the tracked sample nor your own xo-config.cfg is modified. Changing the ports in the editor is picked up too — the review screen, the post-install check and the summary all follow what the file ends up saying.

Requirements

The pool master must have outbound internet access.
A free static IP — this is required, not optional. A stock Debian cloud image has no xe-guest-utilities, so the host cannot report a DHCP lease back and the script would have no address to install over.
On your workstation: ssh, scp, ssh-keygen, and an ISO writer (genisoimage or xorriso). Only the ISO writer might need installing, and that is the sole reason --deploy would ask for sudo — nothing else about this operation touches your machine. sshpass is optional: with it you are asked for the pool master password once, without it ssh asks a second time.
Afterwards the VM contains an ordinary checkout of this repository, so updates work there exactly as anywhere else:

ssh -i xo-deploy-<hostname>.key <admin>@<ip>
cd <clone dir> && ./install-xen-orchestra.sh --update
The generated SSH private key is saved next to the script as xo-deploy-<hostname>.key (git-ignored). Keep it, or add your own key to the VM and delete it.

The admin account's password is optional and asked for during the prompts. The account always gets the generated SSH key, so a password only matters for the VM's console in XO Lite or XCP-ng Center, where no key can be offered, and for su. Leave the prompt empty for a key-only account. If you do set one, a second prompt asks whether SSH should accept it too; the default is no, keeping SSH key-only. Setting the password needs openssl (or mkpasswd) on your workstation — without either, the prompt is skipped and the account stays key-only.

The VM is created with SERVICE_USER=root (the current default) and XO's usual admin@admin.net / admin starting credentials — change that password before putting the VM to use.

To deploy a different Debian release, set XO_DEPLOY_IMAGE_VERSION and XO_DEPLOY_IMAGE_RELEASE, or point XO_DEPLOY_IMAGE_URL at any raw cloud image with cloud-init installed.

Checking a host before deploying
--deploy depends on XAPI behaviour that the test suite cannot exercise without a hypervisor. If a deploy fails, or you want to check a host first, run the probe:

./tests/probe-xapi-deploy.sh --host 192.168.1.10
It verifies each assumption in turn — SR and network enumeration, the pool master's internet access, vdi-import from a pipe (round-tripped and checksummed, not just exit-code checked), the /import_raw_vdi HTTP fallback, and VM creation with the boot and memory parameters deploy sets. Everything it creates is named xo-probe-<run id> and destroyed on exit, including on failure; it never touches objects it did not create, and never starts a VM.

Veeam scheduled a remote call with me and pulled more log files. Of coarse when we ran the backup job twice in a row both times al vms were successful. Veeam needs to baby sit our backups :). The call was cut short do to internet going down. I have uploaded the logs and waiting to hear back.

Update - Veeam took alot more logs from Veeam and from xcp-ng pool.

Their response back -

I've got someone else getting similiar results, so I'm providing both of your logs to get some insights. Basically when you see the error, it's because something happened to the bitmap we left behind on the previous run and so next run, we re-read the entire disk. I've not found anything super clear to what's going wrong with the bitmap and why its gone, even from the Xen server logs, so I'm hoping from QA's eyes might see what I might be missing. I will keep you posted if they have any details.

Update 8.26.26 -

I just wanted to provide an update, the QA team is still checking stuff, but they did advised the following.
They noticed for the disks, they show there are configured XO native backups:
ie.
xo:backup:deltaChainLength: 4; xo:backup:contentKey: 4bce47c7-04bb-44ae-a687-c35c9cfbb2b2;
xo:backup:job: e3616a64-6b83-4bd0-80b1-00523678e909; xo:backup:includeNonNbdQcow2Fix: true; xo:backup:schedule: 3488baee-fe48-4acb-a5d0-e728fe28efa1;
xo:backup:vm: 5703adef-d804-6b15-ba2f-7b3357a711bb; xo:backup:datetime: 20260819T01:00:33Z

I believe you said the native XO Backup was disabled, can you re-confirm if that is accurate, and provide a screenshot for XO's Job and backup list to verify.
QA did confirm that native XO Backups can cause issues as it treats the objects as two different chains and so that's why a CBT comparison can fail. QA says they are looking into ways to change how the comparison works in a future update, and expects it to help if in situations if there are native backups, but asked if the Native backup can be verified as disabled for now.

I have respnded stating that Veeam backup is only used for Windows based vms. Backup in Xen Orchestra is only used for linux base vms.

@poddingue

I haven't seen this issue since we last discussed it back in June. Also looking back at the older comments its seems those where having issues were on AMD systems. I have migrated off AMD in my home lab. Work was always Intel.

@msupport - Looks like veeam is still working with your on your issues. While veeam has pushed me off to vates / xen.

@poddingue - Any updates from Vates about these issues? Is it possible the least patches just pushed might help with either mine or @msupport's issue?

Update - Just got a reply back from veeam ...

As per internal testing, I’m escalation this to the next tier

Regards

@dsauce

To test your theory out on my issue i just tried to disable CBT on XOA side and one vm gave me this error...

vdi.set
{
  "id": "11286a97-b773-4ec4-a0b7-464ab87254ca",
  "cbt": false
}
{
  "code": "UUID_INVALID",
  "params": [
    "VDI",
    "a52fe4ab-edc2-4975-a118-f6db84435379"
  ],
  "call": {
    "duration": 1,
    "method": "VDI.get_by_uuid",
    "params": [
      "* session id *",
      "a52fe4ab-edc2-4975-a118-f6db84435379"
    ]
  },
  "message": "UUID_INVALID(VDI, a52fe4ab-edc2-4975-a118-f6db84435379)",
  "name": "XapiError",
  "stack": "XapiError: UUID_INVALID(VDI, a52fe4ab-edc2-4975-a118-f6db84435379)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:64:5)
    at processImmediate (node:internal/timers:452:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

Another vm..

vdi.set
{
  "id": "966fe072-4865-48a2-867a-0eb59b23dcc7",
  "cbt": false
}
{
  "code": "UUID_INVALID",
  "params": [
    "VDI",
    "d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5"
  ],
  "call": {
    "duration": 1,
    "method": "VDI.get_by_uuid",
    "params": [
      "* session id *",
      "d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5"
    ]
  },
  "message": "UUID_INVALID(VDI, d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5)",
  "name": "XapiError",
  "stack": "XapiError: UUID_INVALID(VDI, d6662a49-0d65-4c4b-a8fa-cddd42ab4cd5)
    at Function.wrap (file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/_XapiError.mjs:16:12)
    at file:///usr/local/lib/node_modules/xo-server/node_modules/xen-api/transports/json-rpc.mjs:38:21
    at runNextTicks (node:internal/process/task_queues:64:5)
    at processImmediate (node:internal/timers:452:9)
    at process.callbackTrampoline (node:internal/async_hooks:130:17)"
}

Update - after backup completed with warnings i looked back and CBT was re-enabled in XOA. Did not help with my Veeam issues.

@gleh

I notice the following error...

(1/4): xcp-ng-candidates/primary_db                                                            | 4.0 kB  00:00:00     
(2/4): xcp-ng-testing/primary_db                                                               | 193 kB  00:00:00     
xcp-ng-updates/primary_db      FAILED                                               ] 1.2 MB/s | 2.1 MB  00:00:03 ETA 
http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/repodata/e3780b9c3ab9712c33f7966c80311e07fa34e345f4242fc43b745e07a2f06826-primary.sqlite.bz2: [Errno -1] Metadata file does not match checksum
Trying other mirror.
(3/4): xcp-ng-base/primary_db                                                                  | 3.9 MB  00:00:02     
xcp-ng-updates/primary_db      FAILED                                               ]  0.0 B/s |    0 B  --:--:-- ETA 
http://mirrors.xcp-ng.org/8/8.3/updates/x86_64/repodata/e3780b9c3ab9712c33f7966c80311e07fa34e345f4242fc43b745e07a2f06826-primary.sqlite.bz2: [Errno -1] Metadata file does not match checksum
Trying other mirror.
xcp-ng-updates/primary_db                                                                      | 1.6 MB  00:00:01     
Resolving Dependencies
--> Running transaction check

Update completed...

Dependency Installed:
  xcp-efivar-utils.noarch 0:1.0.0-1.xcpng8.3                                                                          

Updated:
  amd-microcode.noarch 0:20260519-1.1.xcpng8.3                  bash.x86_64 0:4.2.46-30.1.xcpng8.3                    
  blktap.x86_64 0:3.55.5-9.3.xcpng8.3                           ca-certificates.noarch 0:2021.2.50-73.1.xcpng8.3      
  edk2.x86_64 0:20220801-1.7.11.2.xcpng8.3                      forkexecd.x86_64 0:26.1.16-1.1.xcpng8.3               
  gmp.x86_64 1:6.2.1-8.1.xcpng8.3                               gpumon.x86_64 0:24.1.0-96.1.xcpng8.3                  
  kernel.x86_64 0:4.19.19-8.0.46.10.xcpng8.3                    krb5-libs.x86_64 0:1.21.3-4.1.xcpng8.3                
  message-switch.x86_64 0:26.1.16-1.1.xcpng8.3                  openssh.x86_64 0:9.8p1-1.2.6.xcpng8.3                 
  openssh-clients.x86_64 0:9.8p1-1.2.6.xcpng8.3                 openssh-server.x86_64 0:9.8p1-1.2.6.xcpng8.3          
  p11-kit.x86_64 0:0.24.1-4.xcpng8.3                            p11-kit-trust.x86_64 0:0.24.1-4.xcpng8.3              
  qcow-stream-tool.x86_64 0:26.1.16-1.1.xcpng8.3                redhat-lsb-core.x86_64 0:4.1-28.2.1.xcpng8.3          
  redhat-lsb-submod-security.x86_64 0:4.1-28.2.1.xcpng8.3       rrdd-plugins.x86_64 0:26.1.16-1.1.xcpng8.3            
  sm.x86_64 0:3.2.12-23.4.xcpng8.3                              sm-cli.x86_64 0:26.1.16-1.1.xcpng8.3                  
  sm-fairlock.x86_64 0:3.2.12-23.4.xcpng8.3                     squeezed.x86_64 0:26.1.16-1.1.xcpng8.3                
  varstored.x86_64 0:1.3.4-2.1.xcpng8.3                         varstored-guard.x86_64 0:26.1.16-1.1.xcpng8.3         
  varstored-tools.x86_64 0:1.3.4-2.1.xcpng8.3                   vhd-tool.x86_64 0:26.1.16-1.1.xcpng8.3                
  wsproxy.x86_64 0:26.1.16-1.1.xcpng8.3                         xapi-core.x86_64 0:26.1.16-1.1.xcpng8.3               
  xapi-nbd.x86_64 0:26.1.16-1.1.xcpng8.3                        xapi-rrd2csv.x86_64 0:26.1.16-1.1.xcpng8.3            
  xapi-storage-script.x86_64 0:26.1.16-1.1.xcpng8.3             xapi-tests.x86_64 0:26.1.16-1.1.xcpng8.3              
  xapi-xe.x86_64 0:26.1.16-1.1.xcpng8.3                         xcp-emu-manager.x86_64 0:1.2.1-2.xcpng8.3             
  xcp-featured.x86_64 0:1.2.1-4.xcpng8.3                        xcp-networkd.x86_64 0:26.1.16-1.1.xcpng8.3            
  xcp-ng-generic-lib.x86_64 0:1.1.1-5.xcpng8.3                  xcp-ng-xapi-plugins.noarch 0:1.17.0-1.xcpng8.3        
  xcp-rrdd.x86_64 0:26.1.16-1.1.xcpng8.3                        xen-dom0-libs.x86_64 0:4.17.6-12.2.xcpng8.3           
  xen-dom0-tools.x86_64 0:4.17.6-12.2.xcpng8.3                  xen-hypervisor.x86_64 0:4.17.6-12.2.xcpng8.3          
  xen-libs.x86_64 0:4.17.6-12.2.xcpng8.3                        xen-tools.x86_64 0:4.17.6-12.2.xcpng8.3               
  xenopsd.x86_64 0:26.1.16-1.1.xcpng8.3                         xenopsd-cli.x86_64 0:26.1.16-1.1.xcpng8.3             
  xenopsd-xc.x86_64 0:26.1.16-1.1.xcpng8.3                      xo-lite.noarch 0:0.24.0-1.xcpng8.3                    
  xsconsole.x86_64 0:11.0.9.1-1.3.xcpng8.3                      zlib.x86_64 0:1.2.7-17.1.xcpng8.3                     

Complete!

@poddingue

Sorry I thought I replied back to this thread. Yes the changes did fix my issue.

@bogdantomasciuc

Hello, Yes my script does offer to install xo-proxy and handles some of the licensing issues. You may give my script a try and let me know if you having any further issues i can try to work them out. I advise setting up a new vm for XO as the two script are not compatible that i am aware of.

https://xcp-ng.org/forum/topic/11895/install-xo-from-sources.
https://github.com/acebmxer/install_xen_orchestra/

I have also opened a support ticket with Veeam. Well trying to... Issue with my account preventing me to and working to fix that issue.

Veeam forum did make this statment about CBT and snapshots -

• There is one known snapshot issue that I'm guessing could possibly have this side-effect (noted in the release notes BTW). We take standard Xen snapshots for CBT but then immediately rename them with our own naming convention. We've noticed that sometimes this rename operation returns success status even if it fails. In this case I can see how this might throw this sort of error but again best to confirm w/support

Veeam support ticket - 08188639

Update from Veeam -

Thanks for the update

I'm currently checking with my team and I will update as soon as possible, what I noticed in the debug logs is the following API failing

\Backup\Plugins\XEN\Backup\Backup Job 1\Delilah ArcDC02 with debug enable

2026-08-07 20:05:04.067 00006 DEBUG | [BackupService]:  <== Request localhost:19000, IsTaskFinished, body: {"taskId":"491371d9-4699-4087-b549-9862d2188177"}
2026-08-07 20:05:04.067 00006 DEBUG | [BackupService]:  ==> Response localhost:19000, IsTaskFinished, success, duration: 0.1404 msec, body: false
2026-08-07 20:05:04.843 00019 INFO  | [XenRpcClient]: Start ListChangedBlocks
2026-08-07 20:05:04.843 00019 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/Async.VDI.list_changed_blocks, body: ["OpaqueRef:b77ca409-d438-d97e-d8c3-f212647ad469","OpaqueRef:8c83f698-8836-3b4c-1506-8e97315aae1f"]
2026-08-07 20:05:04.845 00019 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/Async.VDI.list_changed_blocks code: "OK", duration: 2 msec, body:{"opaque_ref":"OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"}
2026-08-07 20:05:04.845 00019 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.get_record, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"]
2026-08-07 20:05:04.847 00019 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.get_record code: "OK", duration: 2 msec, body:{"uuid":"72b5eb58-5053-d77a-9ee9-089ffaed4541","name_label":"Async.VDI.list_changed_blocks","name_description":"","allowed_operations":["cancel"],"current_operations":{},"created":"2026-08-08T00:05:04Z","finished":"1970-01-01T00:00:00Z","status":"pending","resident_on":"OpaqueRef:05e44734-86d0-c9f6-5079-25633df7bf06","progress":0.0,"type":"<none/>","result":"","error_info":[],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"()","opaque_ref":null}
2026-08-07 20:05:04.848 00019 DEBUG | [XenRpcClient]: Current task Async.VDI.list_changed_blocks:72b5eb58-5053-d77a-9ee9-089ffaed4541. Status: "pending" Progress: 0
2026-08-07 20:05:06.848 00018 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.get_record, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"]
2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.get_record code: "OK", duration: 1 msec, body:{"uuid":"72b5eb58-5053-d77a-9ee9-089ffaed4541","name_label":"Async.VDI.list_changed_blocks","name_description":"","allowed_operations":[],"current_operations":{},"created":"2026-08-08T00:05:04Z","finished":"2026-08-08T00:05:05Z","status":"failure","resident_on":"OpaqueRef:05e44734-86d0-c9f6-5079-25633df7bf06","progress":1.0,"type":"<none/>","result":"","error_info":["SR_BACKEND_FAILURE_460","","Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]",""],"other_config":{},"subtask_of":"OpaqueRef:NULL","subtasks":[],"backtrace":"(((process xapi)(filename lib/backtrace.ml)(line 210))((process xapi)(filename ocaml/xapi/storage_utils.ml)(line 150))((process xapi)(filename ocaml/xapi/message_forwarding.ml)(line 141))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 24))((process xapi)(filename ocaml/libs/xapi-stdext/lib/xapi-stdext-pervasives/pervasiveext.ml)(line 39))((process xapi)(filename ocaml/xapi/rbac.ml)(line 228))((process xapi)(filename ocaml/xapi/rbac.ml)(line 238))((process xapi)(filename ocaml/xapi/server_helpers.ml)(line 78)))","opaque_ref":null}
2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: Current task Async.VDI.list_changed_blocks:72b5eb58-5053-d77a-9ee9-089ffaed4541. Status: "failure" Progress: 1
2026-08-07 20:05:06.850 00018 DEBUG | [XenRpcClient]: <== Request https://192.168.20.3/task.destroy, body: ["OpaqueRef:c9389aa3-552a-0aa4-a5bc-f9b4f7a40e98"]
2026-08-07 20:05:06.851 00018 DEBUG | [XenRpcClient]: ==> Response https://192.168.20.3/task.destroy code: "OK", duration: 0.735 msec, body:""
2026-08-07 20:05:06.851 00018 ERROR | [XenRpcClient]: Failed ListChangedBlocks. Error: [Task 72b5eb58-5053-d77a-9ee9-089ffaed4541 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. 
Veeam.Vbf.Common.Exceptions.ExceptionWithDetail: [Task 72b5eb58-5053-d77a-9ee9-089ffaed4541 (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]. 
 ---> Failed to calculate changed blocks for given VDIs.
   --- End of inner exception stack trace ---
   at Veeam.XenBackup.RestClient.XenRpcClient.GetTaskResult(XenRef`1 taskRef, CancellationToken cancellationToken)
   at Veeam.XenBackup.RestClient.XenRpcClient.<>c__DisplayClass94_0.<<ListChangedBlocksAsync>b__0>d.MoveNext()
--- End of stack trace from previous location ---
   at Veeam.Vbf.Common.Helper.Retry.RetryHelper.ExecuteActionAsync[T](Func`2 asyncAction, String description, ILogger logger, LogLevel logLevel, CancellationToken cancellationToken)
2026-08-07 20:05:06.851 00018 INFO  | [XenRpcClient]: Retry after 10 sec. Retry 6/10 for ListChangedBlocks 

There is a XEN forum and guide from him, but I need to check internally as first

https://docs.xenserver.com/en-us/xenserver/developer/changed-block-tracking-guide/troubleshoot.html#you-cant-list-changed-blocks-between-two-vdi-snapshots

CBT: the thread to centralize your feedback | XCP-ng and XO forum
https://xcp-ng.org/forum/topic/9268/cbt-the-thread-to-centralize-your-feedback/364
Regards

Update -

Veeam came back and suggested I power off the vms with the warrning and power back on. This cleared the error for 2 out of the 4 vms. 1 vm still showed the error the other vm i was not able to power off at that time.

@olivierlambert - From veeam...

After checking with my team, is it possible if you can engage XEN support on this, I checked some of the XEN forum including the KB article about how to troubleshoot CBT errors, seems to be a clean metadata that can be done from their side Troubleshoot Changed Block Tracking | Develop for XenServer

CBT: the thread to centralize your feedback | XCP-ng and XO forum

Regards

Looks like our issue may be different at the end even with involving CBT... Let me know if i should continue in a new post. Support ticket created - Ticket#7762393

@JB @gregbinsd @olivierlambert

Yes restarting the tool stack works for me. I currently do not have any host offline at least not intentionally. If there is a network glitch that i am not aware of that might cause the issue if that is the actual cause.

As @olivierlambert stated maybe we were having issues all along and they just changed something that is making the issue visible now. The support tunnel should still be open on my environment if you or the team want to look at logs.

@pierrebrunet

No worries,
Mine seems to be intermittent and restarting that pools tool stack resolves the issue for now.

@pierrebrunet

I use nfs for my remotes and still get the error occasionally.

Edit - Note i am only seeing this error in my work production environment. I am not seeing this issue in my home lab. Both use NFS for remote storage and vm storage.

So I ended up going with the Windows B&R as i could net setup the windows mount server when setting up the repo for the backups. It would setup the linux one but not the windows one. This was with using the veeam console from a windows client. So i ended up with the Windows one.

Now after a few backups i am see this warning / error....

8/5/2026 5:34:50 PM Warning : Failed to use CBT: [Task e75c4247-2ee1-7b51-088c-b970ace60f3d (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]

First i thought it was because i maped the new job to older backups from Beta v1. So i purged all old backups and started fresh. The full backups were successful. Now on first delta 4 out of 7 vms have that warrning.

This a Veeam issue or xcp-ng?

@msupport said:

The CBT metadata VDIs remain as snapshots, but their VHD parent references point to nothing. The tapdisk crashes, the VM loses disk access, the filesystem becomes corrupt.

For VM-A (SQL Server), both disks (60GB + 80GB) have broken CBT snapshots. virtual-size=0, physical-utilisation=0.

For VM-B (File Server), the 80GB disk has a broken CBT snapshot. Interestingly: virtual-size=85899345920 (80GB, same as the base disk) and physical-utilisation=8388608 (8MB). This is different behavior from VM-A — possibly a different code path in the CBT lifecycle.

In my testing i did see similar issue

Edit - updated with actual error message from what I am getting. The first full backup were all successful. This first delta backup have 4 vms with this error. The first time I saw similar was only 2 vms.

8/5/2026 5:34:50 PM Warning : Failed to use CBT: [Task e75c4247-2ee1-7b51-088c-b970ace60f3d (Async.VDI.list_changed_blocks) failed: . SR_BACKEND_FAILURE_460. . Failed to calculate changed blocks for given VDIs. [opterr=Source and target VDI are unrelated]

For me i took it as because i original had backups from veeam 12.x the first xcp-ng beta. Pointed new backups from 13.1 to older backups. I then purged all older backups and started with a fresh backup chain.

I ended up reverting to the windows base as i was not able to get the windows mount server setup. This was setting up from a windows client using the console to setup the repo.

@olivierlambert

Mine has not done it since last post but i have opened a support ticket along with tunnel being opened - Ticket#7762089

If i recall from my migration I belive thats all there is to it. I know the docs mention about doing a test migration but mine just migrated and it just worked as you stated after making the needed adjustments.

Let other chime in but i think you are good to go. And you could delete the snapshot from the vm.