XO Community edition backups dont work as of build 6b263

RobWhalley

@marcoi Hi, the service restarted as part of the upgrade process, but I have done a restart of the VM running Orchestra as well, just to be on the safe side.

Please see attached log.txt for additional log information.

git log shows the expected commit:

/opt/xo/xo-builds/xen-orchestra-202506301106# git log
commit fcefaaea85651c3c0bb40bfba8199dd4e963211c (HEAD, origin/master, origin/HEAD)
Author: Pierre Brunet <pierre.demonthelie@gmail.com>
Date:   Mon Jun 30 10:43:41 2025 +0200

    feat: technical release (#8767)

olivierlambert

The rebuild maybe?

marcoi

@RobWhalley to see if it's behaving the same as what I saw on prior versions can you do the following steps?

open tasks in a new browser window so you can monitor it. If you can keep the backup window and task window side by side you can see if the export command comes and goes quickly.
1.a If you go between backup and tasks in the same window, ive noticed the prior task history clears out.
ssh into the VM running XO and monitor the syslog
tail -100f /var/log/syslog
#may need root then try sudo tail -100f /var/log/syslog
In primary window kick off a backup then watch the task window to see if export command comes and goes.
If the task disappears then click on previous tasks and see if you even had export command come up.
reply with the results of how task shows export step and any errors in the syslog file.

Hopefully that helps the devs figure out whats going on in your system.

olivierlambert

Also maybe xo-server -h can give the info we need

RobWhalley

Thanks both for the help, it's much appreciated.

@olivierlambert - apologies, I'm having some difficulty in finding xo-server... the closest I can find is a mention to it in /opt/xo/xo-src/xen-orchestra/packages/xo-server/xo-server.service:

ExecStart=/usr/local/bin/xo-server

# /usr/local/bin/xo-server
bash: /usr/local/bin/xo-server: No such file or directory

However, when I check /etc/systemd/system/xo-server.service, I get something quite different:

ExecStart=/opt/xo/xo-server/dist/cli.mjs

This file does exist, but running it with -h doesn't seem to provide a useful output:

# /opt/xo/xo-server/dist/cli.mjs -h
Usage: xo-server [--safe-mode]

xo-server v5.180.1

I can run the script in place of the service, but in terms of output it only gives me the same as the journalctl -u xo-server output (see previous attachment):

# systemctl stop xo-server
# /opt/xo/xo-server/dist/cli.mjs

2025-06-30T15:00:42.871Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
2025-06-30T15:00:42.892Z xo:backups:worker WARN possibly unhandled rejection {
  error: RequestAbortedError [AbortError]: Request aborted
      at BodyReadable.destroy (/opt/xo/xo-builds/xen-orchestra-202506301106/node_modules/undici/lib/api/readable.js:51:13)
      at XapiVhdStreamSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:160:22)
      at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:34:29)
      at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiStreamNbd.mjs:81:19)
      at #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:92:21)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:198:18)
      at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
      at async file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/backups/_incrementalVm.mjs:67:5
      at async Promise.all (index 0) {
    code: 'UND_ERR_ABORTED'
  }
}

@marcoi - I didn't see anything appear other than this:

Edited to correct screenshot

marcoi

@RobWhalley I see in the error it mentioning NDB, do your backups have that enabled?

can you try disabling it and testing backup if you do have it on? I think i saw someone else post issue with NDB code in the forums.

RobWhalley

Thanks @marcoi - I don't believe so, though I have no experience of the Network Block Device protocol. The VMs are backed up to a NFS share on a Synology NAS, so pretty low tech in comparison to what a lot of folks here will be using. Based on this page, I'm guessing NBD isn't natively supported on these devices?

This is a sample of the settings, let me know if you need any further information:

marcoi

@RobWhalley Okay you have similar setup to me then. Can you confirm NFS share for backup is working and enabled?
it under settings -->Remote

Make sure the enabled is green and try clicking on the test icon in far right.

RobWhalley

@marcoi Good thinking, here we go - seems OK:

I think I neglected to mention that while backups using the Delta Backup mode failed, other modes have been working:

marcoi

@RobWhalley lol i forgot to test deltas - let me see on that.

marcoi

scratch that prior comment. My backups are all setup as delta backup with schedule 2nd run that forces full backup. So i believe both are working correctly.

if you can also try deleting all the backup snapshots of the VMs in the backup and running full backup then the delta backup jobs.

RobWhalley

Thanks again @marcoi - I wasn't able to get a full backup to work using the Delta Backup with Force full backup option selected. However, using the Backup method was successful, so I have backups for now. I'll try to take another look at it today and do some further testing.

bufanda

Hi,

I tested build 48bf9c9 and backups whether they using NBD or not seem not to start transfers at all.

This is the log output without NBD:

2025-07-01T05:09:04.436Z xo:backups:worker INFO starting backup
2025-07-01T05:09:06.481Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
}
2025-07-01T05:09:06.490Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
}
2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/20250701T050331Z.vhd'
}
2025-07-01T05:09:08.028Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250701T050331Z.vhd'
}
2025-07-01T05:09:08.029Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250701T050331Z.vhd'
}
2025-07-01T05:09:52.631Z xo:backups:worker WARN possibly unhandled rejection {
  error: TypeError: this.source.readBlock is not a function
      at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
      at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
      at buildDiskBlockGenerator.next (<anonymous>)
      at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Promise.all (index 0)
}
2025-07-01T05:09:52.633Z xo:backups:worker WARN possibly unhandled rejection {
  error: TypeError: this.source.readBlock is not a function
      at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
      at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
      at buildDiskBlockGenerator.next (<anonymous>)
      at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Promise.all (index 0)
}
2025-07-01T05:09:52.634Z xo:backups:worker WARN possibly unhandled rejection {
  error: TypeError: this.source.readBlock is not a function
      at XapiDiskSource.readBlock (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
      at XapiDiskSource.buildDiskBlockGenerator (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
      at buildDiskBlockGenerator.next (<anonymous>)
      at XapiDiskSource.diskBlocks (file:///home/node/xen-orchestra/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async Promise.all (index 0)
}

And smae VM when trying to use NBD

2025-07-01T05:02:49.119Z xo:backups:worker INFO starting backup
2025-07-01T05:02:51.256Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.268Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.270Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.627Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/01c64faa-ed71-4190-a11b-bce454d2a5d4/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/38289b78-efb3-439e-90e9-41bbe5f365c2/.20250629T003525Z.vhd'
}
2025-07-01T05:02:51.628Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/e707a6aa-c26b-766c-a2da-051ebdfec000/vdis/a9df501c-2126-4db2-8e30-77ecc7010e35/91528678-9ae0-4d85-864f-69c327f0bcba/.20250629T003525Z.vhd'
}
2025-07-01T05:06:04.924Z xo:backups:worker INFO starting backup
2025-07-01T05:06:09.185Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
  path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
}
2025-07-01T05:06:09.852Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
  path: '/xo-vm-backups/3df8fe91-2c2c-c264-4f21-a07e831d3bb7/vdis/49431dbf-9b1a-4a01-a378-d096edda8ff3/8c1a888f-b3a1-4a8d-bddf-c899c7c9b181/.20250629T010511Z.vhd'
}
2025-07-01T05:06:18.711Z xo:xapi:vdi WARN invalid HTTP header in response body {
  body: 'HTTP/1.1 500 Internal Error\r\n' +
    'content-length: 346\r\n' +
    'content-type: text/html\r\n' +
    'connection: close\r\n' +
    'cache-control: no-cache, no-store\r\n' +
    '\r\n' +
    '<html><body><h1>HTTP 500 internal server error</h1>An unexpected error occurred; please wait a while and try again. If the problem persists, please contact your support representative.<h1> Additional information </h1>Db_exn.Read_missing_uuid(&quot;VDI&quot;, &quot;&quot;, &quot;OpaqueRef:9805714f-7a36-6c2c-c194-9f08baf519df&quot;)</body></html>'
}

RobWhalley

@marcoi @olivierlambert OK, I think I've found the problem, and a fix... at least for the error I was seeing...

If you create a new backup with the method Delta Backup, it will fail with the error posted previously.
Activate the "Use NBD to transfer disk if available" switch
Save the backup job
Deactivate the "Use NBD to transfer disk if available" switch
Save the backup job
Run the backup job again

This time, journalctl doesn't log any errors, and the export task can be seen:

My guess is, that even though by default the "Use NBD to transfer disk if available" switch is shown as inactive when creating a new backup job...

...in reality NBD is switched on by default. Activating and deactivating the switch presumably creates an entry in the job that specifies NBD shouldn't be used.

It would be interesting to know if anyone else can replicate this behaviour, or if it's just something odd in my setup?

florent

@RobWhalley nice work, I am testing it right now

siemsenit

@RobWhalley Thanks! It looks like that fixed it for me.

RobWhalley

@florent Thanks for taking a look, much appreciated!
@siemsenit Glad to help! I can't claim credit though, @marcoi pointed out the mention of NBD in the logs

torso70

@RobWhalley
It works for me too.

florent

you're right, preferNbd default value is not the same on the front and the back, we'll fix it

Most of the backup issue are solved by either removing the snapshot, or having it fa enough so thaht the next backup will be a base one, with all the data

I am still not reproducing the nackup stuck issue after this commit https://github.com/vatesfr/xen-orchestra/commit/3c8efb4dc3e1cb3955521063047227849cfb6ed3

0

fbeauchamp committed to vatesfr/xen-orchestra

fix(backups): non NBD backups stalls (#8766)

marcoi

@RobWhalley I think in the beginning i had that option enabled in my backup jobs, but found it was a hassle to move VMs between hosts with it on so i turned it off for all backups. Might be why my jobs worked because i had already done that switch in the past.