Hello everyone. I tripped over this issue. If someone got another approach I would be interested.
Thanks to @lsouai-vates I had a look at:
and
To what I understand it is not possible as a Non-Admin user to get information like pools, ... By creating a new admin user limiting the resources via ACLS with viewer right worked around this. However, granting admin rights still looks sort of strange.
Just in case someone struggled as well this information might help.
Hello everyone. I tripped over this issue. If someone got another approach I would be interested.
Thanks to @lsouai-vates I had a look at:
and
To what I understand it is not possible as a Non-Admin user to get information like pools, ... By creating a new admin user limiting the resources via ACLS with viewer right worked around this. However, granting admin rights still looks sort of strange.
Just in case someone struggled as well this information might help.
@peo
I am sorry, I must have missed your statement about the discs.
Okay. We can both confirm:
I will observe this during this week and then file a bug report.
@peo
Hi. Sorry for bothering you. I may have expressed in-precise in my question.
For reporting a bug we need to get the most specific information possible. That helps developers to pinpoint a bug. Since I have only trouble with VMs having additional virtual discs (disc created on a local SR on the same host as the VM) attached to it, I would like to know if you have a counter example. Or in other words: Do all your VMs only have a single virtual disc?
@manilx feedback would be very interesting as well.
Thank you for your support.
@peo
Hi thank you. Your are most likely right about the backup storage overflow.
Just two more questions:
@peo Hi, thank you for your quick reply. Since I had this storage issues after disabling it, I am a little bit careful. My knowledge is really limited about CBT based backups, can you tell me, what it means in terms of storage use. To my understanding it will keep the snapshots and thereby significantly increase space usage, or do I miss something? And have you heard about whether the bug is officially known and worked on?
Hi everyone again. I pretty much got back to square one. What I can observer is, that all my VMs where I added additional disc run into the above error code. So 1 disc per VM works fine for 6 backups, the two others VMs (on with 2 disc, one with 5 discs) fail. Does CBT based delta backups only work if there is no disc attached? I really appreciate any help.
Hi everyone, just FYI. During my delta backup testing, I ran out of space on my NAS (although it should have had enough?!?). It must have created more data then I expected. Therefore I had other priorities, I removed the backups and set up new delta backups. However, thereby I could not dive into further exploring the problems.
@manilx okay. I will wait and see. Backup runs tonight.
Hi @manilx.
thank you for the quick reply. I indeed do have it enabled.
I am just surprised since the exact same job worked previously and it does not make sense to me to keep it. Nevertheless I have to admit, that my knowledge is limited to these types of snapshots and its impact upon storage usage. Just FYI, there is this discussion I found: https://xcp-ng.org/forum/topic/10400/purge-snapshot-data-when-using-cbt-why-wouldn-t-i-enable-this.
However, your point is very good. I just disabled it in order to validate your thesis. If so, I think upon our system behavior we could then report some sort of "bug". Did you backup did not work from the start either or was it also after "some sort of modifications"?
Hello everyone,
this type of error now popped up at my delta backup as well.
Some specs:
OS: Debian 12 patched
Xen Orchestra: as of 20250616 0005
XEN-NG: 8.3 latest patches applied
Job Type: Delta Backup with 8 VMs backing up to remote Synology NAS
The job run for a quiet a few weeks without any problems. But I have to admit, that I cannot say, what exactly induced the problem, since I updated xen orchestra and modified the backup job (I removed 2 VM, moved them to another machine, added additional disks and readded them the same backup job). Manually triggering the backup via "Restart VM backup" upon the failures dialog successfully runs the backup.
I get the following error in the log:
...
Jun 18 02:41:28 xoa xo-server[258369]: 2025-06-18T00:41:28.527Z xo:backups:MixinBackupWriter INFO merge in progress {
Jun 18 02:41:28 xoa xo-server[258369]: done: 6895,
Jun 18 02:41:28 xoa xo-server[258369]: parent: '/xo-vm-backups/924b4cf4-c8b3-18ab-5f78-d1daa77bc3fc/vdis/8c0477b9-b6e8-45ca-bcac-b78549e05b6f/ab2c3be9-bec5-4361-9ad2-81acfc14c16e/20250611T005140Z.vhd',
Jun 18 02:41:28 xoa xo-server[258369]: progress: 97,
Jun 18 02:41:28 xoa xo-server[258369]: total: 7132
Jun 18 02:41:28 xoa xo-server[258369]: }
Jun 18 02:41:38 xoa xo-server[258369]: 2025-06-18T00:41:38.528Z xo:backups:MixinBackupWriter INFO merge in progress {
Jun 18 02:41:38 xoa xo-server[258369]: done: 7073,
Jun 18 02:41:38 xoa xo-server[258369]: parent: '/xo-vm-backups/924b4cf4-c8b3-18ab-5f78-d1daa77bc3fc/vdis/8c0477b9-b6e8-45ca-bcac-b78549e05b6f/ab2c3be9-bec5-4361-9ad2-81acfc14c16e/20250611T005140Z.vhd',
Jun 18 02:41:38 xoa xo-server[258369]: progress: 99,
Jun 18 02:41:38 xoa xo-server[258369]: total: 7132
Jun 18 02:41:38 xoa xo-server[258369]: }
Jun 18 02:41:46 xoa xo-server[258369]: 2025-06-18T00:41:46.228Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
Jun 18 02:41:46 xoa xo-server[258369]: path: '/xo-vm-backups/924b4cf4-c8b3-18ab-5f78-d1daa77bc3fc/20250617T235823Z.json',
Jun 18 02:41:46 xoa xo-server[258369]: actual: 108580044800,
Jun 18 02:41:46 xoa xo-server[258369]: expected: 108606965248
Jun 18 02:41:46 xoa xo-server[258369]: }
Jun 18 02:46:20 xoa xo-server[258369]: 2025-06-18T00:46:20.182Z xo:backups:MixinBackupWriter WARN cleanVm: incorrect backup size in metadata {
Jun 18 02:46:20 xoa xo-server[258369]: path: '/xo-vm-backups/9960fd34-ad5a-8854-6a90-3b1e88c1398f/20250618T004205Z.json',
Jun 18 02:46:20 xoa xo-server[258369]: actual: 12184453120,
Jun 18 02:46:20 xoa xo-server[258369]: expected: 12190142976
Jun 18 02:46:20 xoa xo-server[258369]: }
Jun 18 02:46:41 xoa xo-server[258369]: 2025-06-18T00:46:41.281Z @xen-orchestra/xapi/disks/Xapi WARN openNbdCBT Error: can't connect to any nbd client
Jun 18 02:46:41 xoa xo-server[258369]: at connectNbdClientIfPossible (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/xapi/disks/utils.mjs:23:19)
Jun 18 02:46:41 xoa xo-server[258369]: at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Jun 18 02:46:41 xoa xo-server[258369]: at async XapiVhdCbtSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/xapi/disks/XapiVhdCbt.mjs:75:20)
Jun 18 02:46:41 xoa xo-server[258369]: at async #openNbdCbt (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/xapi/disks/Xapi.mjs:129:7)
Jun 18 02:46:41 xoa xo-server[258369]: at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
Jun 18 02:46:41 xoa xo-server[258369]: at async file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/backups/_incrementalVm.mjs:65:5
Jun 18 02:46:41 xoa xo-server[258369]: at async Promise.all (index 1)
Jun 18 02:46:41 xoa xo-server[258369]: at async cancelableMap (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/backups/_cancelableMap.mjs:11:12)
Jun 18 02:46:41 xoa xo-server[258369]: at async exportIncrementalVm (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/backups/_incrementalVm.mjs:28:3)
Jun 18 02:46:41 xoa xo-server[258369]: at async IncrementalXapiVmBackupRunner._copy (file:///opt/xo/xo-builds/xen-orchestra-202506160005/@xen-orchestra/backups/_runners/_vmRunners/IncrementalXapi.mjs:38:25) {
Jun 18 02:46:41 xoa xo-server[258369]: code: 'NO_NBD_AVAILABLE'
Jun 18 02:46:41 xoa xo-server[258369]: }
Jun 18 02:46:43 xoa xo-server[258369]: 2025-06-18T00:46:43.098Z xo:xapi:vdi WARN invalid HTTP header in response body {
Jun 18 02:46:43 xoa xo-server[258369]: body: 'HTTP/1.1 500 Internal Error\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]: 'content-length: 318\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]: 'content-type: text/html\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]: 'connection: close\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]: 'cache-control: no-cache, no-store\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]: '\r\n' +
Jun 18 02:46:43 xoa xo-server[258369]: '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>VDI_INCOMPATIBLE_TYPE: [ OpaqueRef:1ed06eb9-ed6f-d8f0-25a4-647a4ff22ca6; CBT metadata ]</body></html>'
Jun 18 02:46:43 xoa xo-server[258369]: }
...
Anyone has any ideas?