Hello everyone. I tripped over this issue. If someone got another approach I would be interested.

Thanks to @lsouai-vates I had a look at:

https://github.com/vatesfr/xen-orchestra/blob/ab56924b1d046ccf6c09dfe7a4ab47deb5d77f4a/packages/xo-acl-resolver/index.js

and

https://github.com/vatesfr/xen-orchestra/blob/ab56924b1d046ccf6c09dfe7a4ab47deb5d77f4a/packages/xo-server/src/xo-mixins/acls.mjs#L150-L168

To what I understand it is not possible as a Non-Admin user to get information like pools, ... By creating a new admin user limiting the resources via ACLS with viewer right worked around this. However, granting admin rights still looks sort of strange.

Just in case someone struggled as well this information might help.

@peo
I am sorry, I must have missed your statement about the discs.

Okay. We can both confirm:

• CBT based delta backups fail on some VMs, causing an iterating success / failure backup behavior.
• removing "Purge snapshot data when using CBT" flag works around it, but at the costs of additional "doubling" storage space.
• In addition only I observed, that it always and only happens to me, when there is are additional discs involves. Delta backup of VMs within the same backup job but different machines work at my place.

I will observe this during this week and then file a bug report.

@peo
Hi. Sorry for bothering you. I may have expressed in-precise in my question.

For reporting a bug we need to get the most specific information possible. That helps developers to pinpoint a bug. Since I have only trouble with VMs having additional virtual discs (disc created on a local SR on the same host as the VM) attached to it, I would like to know if you have a counter example. Or in other words: Do all your VMs only have a single virtual disc?

@manilx feedback would be very interesting as well.

Thank you for your support.

@peo
Hi thank you. Your are most likely right about the backup storage overflow.

Just two more questions:

• have you every experienced this problem on a VM with only one disc attached? I am asking since at my site only delta backups fail with additional discs attached.
• Is this somehow addressed as a bug or shall I officially report a bug, since now quite a few users experience this problem.

@peo Hi, thank you for your quick reply. Since I had this storage issues after disabling it, I am a little bit careful. My knowledge is really limited about CBT based backups, can you tell me, what it means in terms of storage use. To my understanding it will keep the snapshots and thereby significantly increase space usage, or do I miss something? And have you heard about whether the bug is officially known and worked on?

Hi everyone again. I pretty much got back to square one. What I can observer is, that all my VMs where I added additional disc run into the above error code. So 1 disc per VM works fine for 6 backups, the two others VMs (on with 2 disc, one with 5 discs) fail. Does CBT based delta backups only work if there is no disc attached? I really appreciate any help.

Hi everyone, just FYI. During my delta backup testing, I ran out of space on my NAS (although it should have had enough?!?). It must have created more data then I expected. Therefore I had other priorities, I removed the backups and set up new delta backups. However, thereby I could not dive into further exploring the problems.

@manilx okay. I will wait and see. Backup runs tonight.

Hi @manilx.

thank you for the quick reply. I indeed do have it enabled.

I am just surprised since the exact same job worked previously and it does not make sense to me to keep it. Nevertheless I have to admit, that my knowledge is limited to these types of snapshots and its impact upon storage usage. Just FYI, there is this discussion I found: https://xcp-ng.org/forum/topic/10400/purge-snapshot-data-when-using-cbt-why-wouldn-t-i-enable-this.

However, your point is very good. I just disabled it in order to validate your thesis. If so, I think upon our system behavior we could then report some sort of "bug". Did you backup did not work from the start either or was it also after "some sort of modifications"?

Hello everyone,

this type of error now popped up at my delta backup as well.

Some specs:
OS: Debian 12 patched
Xen Orchestra: as of 20250616 0005
XEN-NG: 8.3 latest patches applied
Job Type: Delta Backup with 8 VMs backing up to remote Synology NAS

The job run for a quiet a few weeks without any problems. But I have to admit, that I cannot say, what exactly induced the problem, since I updated xen orchestra and modified the backup job (I removed 2 VM, moved them to another machine, added additional disks and readded them the same backup job). Manually triggering the backup via "Restart VM backup" upon the failures dialog successfully runs the backup.

I get the following error in the log:

...
Jun 18 02:41:28 xoa xo-server[258369]: 2025-06-18T00:41:28.527Z xo:backups:MixinBackupWriter INFO merge in progress {
Jun 18 02:41:28 xoa xo-server[258369]:   done: 6895,
Jun 18 02:41:28 xoa xo-server[258369]:   parent: '/xo-vm-backups/924b4cf4-c8b3-18ab-5f78-d1daa77bc3fc/vdis/8c0477b9-b6e8-45ca-bcac-b78549e05b6f/ab2c3be9-bec5-4361-9ad2-81acfc14c16e/20250611T005140Z.vhd',
Jun 18 02:41:28 xoa xo-server[258369]:   progress: 97,
Jun 18 02:41:28 xoa xo-server[258369]:   total: 7132
Jun 18 02:41:28 xoa xo-server[258369]: }
Jun 18 02:41:38 xoa xo-server[258369]: 2025-06-18T00:41:38.528Z xo:backups:MixinBackupWriter INFO merge in progress {
Jun 18 02:41:38 xoa xo-server[258369]:   done: 7073,
Jun 18 02:41:38 xoa xo-server[258369]:   parent: '/xo-vm-backups/924b4cf4-c8b3-18ab-5f78-d1daa77bc3fc/vdis/8c0477b9-b6e8-45ca-bcac-b78549e05b6f/ab2c3be9-bec5-4361-9ad2-81acfc14c16e/20250611T005140Z.vhd',
Jun 18 02:41:38 xoa xo-server[258369]:   progress: 99,
Jun 18 02:41:38 xoa xo-server[258369]:   total: 7132
Jun 18 02:41:38 xoa xo-server[258369]: }
Jun 18 02:41:46 xoa xo-server[258369]: 2025-06-18T00:41:46.228Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
Jun 18 02:41:46 xoa xo-server[258369]:   path: '/xo-vm-backups/924b4cf4-c8b3-18ab-5f78-d1daa77bc3fc/20250617T235823Z.json',
Jun 18 02:41:46 xoa xo-server[258369]:   actual: 108580044800,
Jun 18 02:41:46 xoa xo-server[258369]:   expected: 108606965248
Jun 18 02:41:46 xoa xo-server[258369]: }
Jun 18 02:46:20 xoa xo-server[258369]: 2025-06-18T00:46:20.182Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
Jun 18 02:46:20 xoa xo-server[258369]:   path: '/xo-vm-backups/9960fd34-ad5a-8854-6a90-3b1e88c1398f/20250618T004205Z.json',
Jun 18 02:46:20 xoa xo-server[258369]:   actual: 12184453120,
Jun 18 02:46:20 xoa xo-server[258369]:   expected: 12190142976
Jun 18 02:46:20 xoa xo-server[258369]: }
Jun 18 02:46:41 xoa xo-server[258369]: 2025-06-18T00:46:41.281Z @xen-orchestra/xapi/disks/Xapi WARN openNbdCBT Error: can't connect to any nbd client
Jun 18 02:46:41 xoa xo-server[258369]:     at connectNbdClientIfPossible (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/xapi/disks/utils.mjs:23:19)
Jun 18 02:46:41 xoa xo-server[258369]:     at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Jun 18 02:46:41 xoa xo-server[258369]:     at async XapiVhdCbtSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/xapi/disks/XapiVhdCbt.mjs:75:20)
Jun 18 02:46:41 xoa xo-server[258369]:     at async #openNbdCbt (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/xapi/disks/Xapi.mjs:129:7)
Jun 18 02:46:41 xoa xo-server[258369]:     at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Jun 18 02:46:41 xoa xo-server[258369]:     at async file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/backups/_incrementalVm.mjs:65:5
Jun 18 02:46:41 xoa xo-server[258369]:     at async Promise.all (index 1)
Jun 18 02:46:41 xoa xo-server[258369]:     at async cancelableMap (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/backups/_cancelableMap.mjs:11:12)
Jun 18 02:46:41 xoa xo-server[258369]:     at async exportIncrementalVm (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/backups/_incrementalVm.mjs:28:3)
Jun 18 02:46:41 xoa xo-server[258369]:     at async IncrementalXapiVmBackupRunner._copy (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:38:25) {
Jun 18 02:46:41 xoa xo-server[258369]:   code: 'NO_NBD_AVAILABLE'
Jun 18 02:46:41 xoa xo-server[258369]: }
Jun 18 02:46:43 xoa xo-server[258369]: 2025-06-18T00:46:43.098Z xo:xapi:vdi WARN invalid HTTP header in response body {
Jun 18 02:46:43 xoa xo-server[258369]:   body: 'HTTP/1.1 500 Internal Error\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]:     'content-length: 318\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]:     'content-type: text/html\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]:     'connection: close\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]:     'cache-control: no-cache, no-store\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]:     '\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]:     '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:1ed06eb9-ed6f-d8f0-25a4-647a4ff22ca6; CBT metadata ]</body></html>'
Jun 18 02:46:43 xoa xo-server[258369]: }
...

Anyone has any ideas?

@rtjdamen
Thank you for your hint! I indeed did not enable NBD on my management interface. Somehow I did not notice that in the docs.

Maybe it would be a helpful for others to mention in the docs that you need to enable NBD in: a.) the network settings b.) the vm disc settings and c.) the actual backup task in order to work.

Thank you again for your quick help!!!

@rtjdamen Thanks a lot for the quick response. The second step I already tried. But I did not set up a dedicated nbd network yet. I will have to try that. Thanks!

Hi everyone,

Using Xen-NG for couple of years but I am new to Xen Orchestra. And first of all thank you for this great software.

Considering moving partly away from my own written backups scripts I am currently playing around being particular interested in the XOA backup routines, in particular the CBT based delta backups. But maybe first the official facts about my setup:

Server: HP ProLiant DL380p Gen8
Xen-NG: 8.3 - fully patched
Xen Orchestra: self build - commit 9ed55 (as of today)
SR: 2 / both local ext / both are raid 10 disc based by the HPE Smart Array Gen8 Controllers
Remote: NFS mounted (Synology - Fully patched as of today)

Everything I tested so far works great. However I cannot get CBT based delta backups to run. I always get full backups and the error message is:

Can't do delta with this vdi, transfer will be a full
Can't do delta, will try to get a full stream

As said I am using an NFS based remote and all "normal" backups work like a charm. The Dummy VM I am testing to backup a newly set up Debian 12 (management tools installed) on the 2nd - non default- SR. CBT is enabled upon the disc as well in the the backup job ("Use NBD + CBT to transfer disk if available" enabled as well as "Purge snapshot data when using CBT "). I also enabled "Merge backups synchronously".

Unfortunately I do not really get my hands on this problem, because I do not find any particular error messages and do not really find good docs upon cbt. There are 2 observations I made:

• There are 2 *.cbtlog files in the SR directory of the VM. One showing 00000000-0000-0000-0000-000000000000 the other e011e9dd-e14b-4f0a-b143-092ea8f1b6a3. Is that normal?
• If I enable a memory including snapshot mode for each backup run there is one snapshot created in the default SR of the Host. These are not removed after the backup and remain orphaned. It looks to me these snapshots contain the memory (maybe this total BS - then please excuse this, but the observation might be helpful). However this problem is gone, if I switch to "offline snapshots".

Maybe I am just missing some stupid setting. Does anyone have any suggestion where to start troubleshoot.