XO Community edition backups dont work as of build 6b263

DustinB

@Danp said in XO Community edition backups dont work as of build 6b263:

@wf said in XO Community edition backups dont work as of build 6b263:

How do I roll back to a previous commit?

git checkout <target commit>

Then rebuild with yarn; yarn build

@marcoi use the above to roll back to a specific commit.

florent

@marcoi
is there any error message in XO logs ? (like journalctl )

marcoi

@florent
are you asking about logs available in the gui or are there any logs stored locally under XO install for example?

marcoi

Here are the tasks that show up when i try running a full backup.

Seems it does not do anything after the snap shot.
Im current running Xen Orchestra, commit 1cb9c
Will update to the latest commit and see what it does.

here part of the log of backup while it sitting in Started. you see tasks are sitting in pending.

       {
          "data": {
            "id": "f992fff1-e245-48f7-8eb3-25987ecbfbd4",
            "isFull": true,
            "type": "remote"
          },
          "id": "1751028165392:0",
          "message": "export",
          "start": 1751028165392,
          "status": "pending",
          "tasks": [
            {
              "id": "1751028165994",
              "message": "transfer",
              "start": 1751028165994,
              "status": "pending"
            }

Update 2 ( i updated to latest Xen Orchestra, commit 54141)
I deleted the snaps of prior backup attempts from each VM and ran a full backup job again.
You can see it did the snap tasks (create/cleanup) but it still pending on export and transfer.

     "id": "1751028772562",
      "message": "backup VM",
      "start": 1751028772562,
      "status": "pending",
      "tasks": [
        {
          "id": "1751028772569",
          "message": "clean-vm",
          "start": 1751028772569,
          "status": "success",
          "end": 1751028772628,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1751028772999",
          "message": "snapshot",
          "start": 1751028772999,
          "status": "success",
          "end": 1751028778291,
          "result": "db1a49fb-10af-026d-7d27-4ff0f5da97c6"
        },
        {
          "data": {
            "id": "f992fff1-e245-48f7-8eb3-25987ecbfbd4",
            "isFull": true,
            "type": "remote"
          },
          "id": "1751028778291:0",
          "message": "export",
          "start": 1751028778291,
          "status": "pending",
          "tasks": [
            {
              "id": "1751028778863",
              "message": "transfer",
              "start": 1751028778863,
              "status": "pending"

marcoi

I got this from the syslog file. some error are showing up when i ran backup job.

2025-06-27T09:40:56.178745-04:00 xen xo-server[2283]: (node:2283) [DEP0044] DeprecationWarning: The `util.isArray` API is deprecated. Please use `Array.isArray()` instead.
2025-06-27T09:40:56.178843-04:00 xen xo-server[2283]: (Use `node --trace-deprecation ...` to show where the warning was created)
2025-06-27T09:40:56.182409-04:00 xen xo-server[2283]: 2025-06-27T13:40:56.178Z xo:xo-server WARN Node warning {
2025-06-27T09:40:56.182468-04:00 xen xo-server[2283]:   error: DeprecationWarning: The `util.isArray` API is deprecated. Please use `Array.isArray()` instead.
2025-06-27T09:40:56.182496-04:00 xen xo-server[2283]:       at IncomingMessage.flash (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/connect-flash/lib/flash.js:67:16)
2025-06-27T09:40:56.182526-04:00 xen xo-server[2283]:       at file:///opt/xo/xo-builds/xen-orchestra-202506270848/packages/xo-server/src/index.mjs:322:11
2025-06-27T09:40:56.182552-04:00 xen xo-server[2283]:       at Layer.handle [as handle_request] (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/layer.js:95:5)
2025-06-27T09:40:56.182580-04:00 xen xo-server[2283]:       at trim_prefix (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:328:13)
2025-06-27T09:40:56.182603-04:00 xen xo-server[2283]:       at /opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:286:9
2025-06-27T09:40:56.182633-04:00 xen xo-server[2283]:       at Function.process_params (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:346:12)
2025-06-27T09:40:56.182656-04:00 xen xo-server[2283]:       at next (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:280:10)
2025-06-27T09:40:56.182679-04:00 xen xo-server[2283]:       at Xo._handleHttpRequest (file:///opt/xo/xo-builds/xen-orchestra-202506270848/packages/xo-server/src/xo.mjs:173:7)
2025-06-27T09:40:56.182704-04:00 xen xo-server[2283]:       at Layer.handle [as handle_request] (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/layer.js:95:5)
2025-06-27T09:40:56.182744-04:00 xen xo-server[2283]:       at trim_prefix (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:328:13)
2025-06-27T09:40:56.182767-04:00 xen xo-server[2283]:       at /opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:286:9
2025-06-27T09:40:56.182822-04:00 xen xo-server[2283]:       at Function.process_params (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:346:12)
2025-06-27T09:40:56.182845-04:00 xen xo-server[2283]:       at next (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:280:10)
2025-06-27T09:40:56.182883-04:00 xen xo-server[2283]:       at initialize (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/passport/lib/middleware/initialize.js:98:5)
2025-06-27T09:40:56.182907-04:00 xen xo-server[2283]:       at Layer.handle [as handle_request] (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/layer.js:95:5)
2025-06-27T09:40:56.182945-04:00 xen xo-server[2283]:       at trim_prefix (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:328:13)
2025-06-27T09:40:56.182970-04:00 xen xo-server[2283]:       at /opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:286:9
2025-06-27T09:40:56.182994-04:00 xen xo-server[2283]:       at Function.process_params (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:346:12)
2025-06-27T09:40:56.183018-04:00 xen xo-server[2283]:       at next (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:280:10)
2025-06-27T09:40:56.183041-04:00 xen xo-server[2283]:       at urlencodedParser (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/body-parser/lib/types/urlencoded.js:94:7)
2025-06-27T09:40:56.183072-04:00 xen xo-server[2283]:       at Layer.handle [as handle_request] (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/layer.js:95:5)
2025-06-27T09:40:56.183105-04:00 xen xo-server[2283]:       at trim_prefix (/opt/xo/xo-builds/xen-orchestra-202506270848/node_modules/express/lib/router/index.js:328:13) {
2025-06-27T09:40:56.183136-04:00 xen xo-server[2283]:     code: 'DEP0044'
2025-06-27T09:40:56.183166-04:00 xen xo-server[2283]:   }
2025-06-27T09:40:56.183212-04:00 xen xo-server[2283]: }
2025-06-27T09:41:16.131076-04:00 xen xo-server[2352]: 2025-06-27T13:41:16.129Z xo:backups:worker INFO starting backup
2025-06-27T09:41:16.842684-04:00 xen xo-server[2352]: 2025-06-27T13:41:16.842Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
2025-06-27T09:41:16.842762-04:00 xen xo-server[2352]:   path: '/xo-vm-backups/4b999d5b-6bbf-78b5-93f2-83747c879be6/vdis/79a021e6-9637-40f0-871f-ab122f104d65/9aa754da-084f-4cc3-a494-2de58e17387c/.20250627T130633Z.vhd'
2025-06-27T09:41:16.842806-04:00 xen xo-server[2352]: }
2025-06-27T09:41:16.843337-04:00 xen xo-server[2352]: 2025-06-27T13:41:16.843Z xo:backups:MixinBackupWriter INFO deleting temporary VHD {
2025-06-27T09:41:16.843372-04:00 xen xo-server[2352]:   path: '/xo-vm-backups/1c29eb9f-5b4b-2a63-dee2-47b738701058/vdis/79a021e6-9637-40f0-871f-ab122f104d65/a48c797e-39b2-4d9a-84a1-d4323436430f/.20250627T130627Z.vhd'
2025-06-27T09:41:16.843415-04:00 xen xo-server[2352]: }
2025-06-27T09:41:16.847122-04:00 xen xo-server[2352]: 2025-06-27T13:41:16.846Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
2025-06-27T09:41:16.847160-04:00 xen xo-server[2352]:   path: '/xo-vm-backups/4b999d5b-6bbf-78b5-93f2-83747c879be6/vdis/79a021e6-9637-40f0-871f-ab122f104d65/9aa754da-084f-4cc3-a494-2de58e17387c/.20250627T130633Z.vhd'
2025-06-27T09:41:16.847178-04:00 xen xo-server[2352]: }
2025-06-27T09:41:16.847260-04:00 xen xo-server[2352]: 2025-06-27T13:41:16.847Z xo:backups:MixinBackupWriter INFO deleting unused VHD {
2025-06-27T09:41:16.847311-04:00 xen xo-server[2352]:   path: '/xo-vm-backups/1c29eb9f-5b4b-2a63-dee2-47b738701058/vdis/79a021e6-9637-40f0-871f-ab122f104d65/a48c797e-39b2-4d9a-84a1-d4323436430f/.20250627T130627Z.vhd'
2025-06-27T09:41:16.847334-04:00 xen xo-server[2352]: }
2025-06-27T09:41:18.829536-04:00 xen xo-server[2352]: 2025-06-27T13:41:18.828Z xo:backups:worker WARN possibly unhandled rejection {
2025-06-27T09:41:18.829591-04:00 xen xo-server[2352]:   error: TypeError: this.source.readBlock is not a function
2025-06-27T09:41:18.829612-04:00 xen xo-server[2352]:       at XapiDiskSource.readBlock (file:///opt/xo/xo-builds/xen-orchestra-202506270848/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
2025-06-27T09:41:18.829627-04:00 xen xo-server[2352]:       at XapiDiskSource.buildDiskBlockGenerator (file:///opt/xo/xo-builds/xen-orchestra-202506270848/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
2025-06-27T09:41:18.829642-04:00 xen xo-server[2352]:       at buildDiskBlockGenerator.next (<anonymous>)
2025-06-27T09:41:18.829658-04:00 xen xo-server[2352]:       at XapiDiskSource.diskBlocks (file:///opt/xo/xo-builds/xen-orchestra-202506270848/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
2025-06-27T09:41:18.829673-04:00 xen xo-server[2352]:       at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-06-27T09:41:18.829687-04:00 xen xo-server[2352]:       at async Promise.all (index 0)
2025-06-27T09:41:18.829705-04:00 xen xo-server[2352]: }
2025-06-27T09:41:27.192451-04:00 xen xo-server[2352]: 2025-06-27T13:41:27.192Z xo:backups:worker WARN possibly unhandled rejection {
2025-06-27T09:41:27.192549-04:00 xen xo-server[2352]:   error: TypeError: this.source.readBlock is not a function
2025-06-27T09:41:27.192576-04:00 xen xo-server[2352]:       at XapiDiskSource.readBlock (file:///opt/xo/xo-builds/xen-orchestra-202506270848/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:70:28)
2025-06-27T09:41:27.192606-04:00 xen xo-server[2352]:       at XapiDiskSource.buildDiskBlockGenerator (file:///opt/xo/xo-builds/xen-orchestra-202506270848/@xen-orchestra/disk-transform/dist/Disk.mjs:69:28)
2025-06-27T09:41:27.192627-04:00 xen xo-server[2352]:       at buildDiskBlockGenerator.next (<anonymous>)
2025-06-27T09:41:27.192657-04:00 xen xo-server[2352]:       at XapiDiskSource.diskBlocks (file:///opt/xo/xo-builds/xen-orchestra-202506270848/@xen-orchestra/disk-transform/dist/Disk.mjs:27:39)
2025-06-27T09:41:27.192686-04:00 xen xo-server[2352]:       at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
2025-06-27T09:41:27.192715-04:00 xen xo-server[2352]:       at async Promise.all (index 0)
2025-06-27T09:41:27.192750-04:00 xen xo-server[2352]: }

archw

I'm on build d8370 and last night all my backups ran without issue.

itservices

Hi.

I am on commit 2effd and have the same problem.

Snapshots are being created but no transfer is taking place.

Regards,
Marc

marcoi

@itservices
also the same with Xen Orchestra, commit 2effd

olivierlambert

Check with fcefaae

marcoi

looks like Xen Orchestra, commit fcefa might have fixed the backup issue. I am running a backup test now and it showing export step running

RobWhalley

@olivierlambert Sadly, it's not working for me - I've updated to the latest commit and tested a job that should take around 5 minutes; it's been stuck on "Started" for a couple of hours now.

marcoi

@RobWhalley did you try to restart the service or reboot the server running XO? I have a nightly crontab job that updates the OS, updates XO and then reboots the server.

RobWhalley

@marcoi Hi, the service restarted as part of the upgrade process, but I have done a restart of the VM running Orchestra as well, just to be on the safe side.

Please see attached log.txt for additional log information.

git log shows the expected commit:

/opt/xo/xo-builds/xen-orchestra-202506301106# git log
commit fcefaaea85651c3c0bb40bfba8199dd4e963211c (HEAD, origin/master, origin/HEAD)
Author: Pierre Brunet <pierre.demonthelie@gmail.com>
Date:   Mon Jun 30 10:43:41 2025 +0200

    feat: technical release (#8767)

olivierlambert

The rebuild maybe?

marcoi

@RobWhalley to see if it's behaving the same as what I saw on prior versions can you do the following steps?

open tasks in a new browser window so you can monitor it. If you can keep the backup window and task window side by side you can see if the export command comes and goes quickly.
1.a If you go between backup and tasks in the same window, ive noticed the prior task history clears out.
ssh into the VM running XO and monitor the syslog
tail -100f /var/log/syslog
#may need root then try sudo tail -100f /var/log/syslog
In primary window kick off a backup then watch the task window to see if export command comes and goes.
If the task disappears then click on previous tasks and see if you even had export command come up.
reply with the results of how task shows export step and any errors in the syslog file.

Hopefully that helps the devs figure out whats going on in your system.

olivierlambert

Also maybe xo-server -h can give the info we need

RobWhalley

Thanks both for the help, it's much appreciated.

@olivierlambert - apologies, I'm having some difficulty in finding xo-server... the closest I can find is a mention to it in /opt/xo/xo-src/xen-orchestra/packages/xo-server/xo-server.service:

ExecStart=/usr/local/bin/xo-server

# /usr/local/bin/xo-server
bash: /usr/local/bin/xo-server: No such file or directory

However, when I check /etc/systemd/system/xo-server.service, I get something quite different:

ExecStart=/opt/xo/xo-server/dist/cli.mjs

This file does exist, but running it with -h doesn't seem to provide a useful output:

# /opt/xo/xo-server/dist/cli.mjs -h
Usage: xo-server [--safe-mode]

xo-server v5.180.1

I can run the script in place of the service, but in terms of output it only gives me the same as the journalctl -u xo-server output (see previous attachment):

# systemctl stop xo-server
# /opt/xo/xo-server/dist/cli.mjs

2025-06-30T15:00:42.871Z @xen-orchestra/xapi/disks/Xapi WARN can't connect through NBD, fallback to stream export
2025-06-30T15:00:42.892Z xo:backups:worker WARN possibly unhandled rejection {
  error: RequestAbortedError [AbortError]: Request aborted
      at BodyReadable.destroy (/opt/xo/xo-builds/xen-orchestra-202506301106/node_modules/undici/lib/api/readable.js:51:13)
      at XapiVhdStreamSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiVhdStreamSource.mjs:160:22)
      at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:34:29)
      at XapiStreamNbdSource.close (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/XapiStreamNbd.mjs:81:19)
      at #openNbdStream (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:92:21)
      at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
      at async XapiDiskSource.openSource (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/xapi/disks/Xapi.mjs:198:18)
      at async XapiDiskSource.init (file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/disk-transform/dist/DiskPassthrough.mjs:28:41)
      at async file:///opt/xo/xo-builds/xen-orchestra-202506301106/@xen-orchestra/backups/_incrementalVm.mjs:67:5
      at async Promise.all (index 0) {
    code: 'UND_ERR_ABORTED'
  }
}

@marcoi - I didn't see anything appear other than this:

Edited to correct screenshot

marcoi

@RobWhalley I see in the error it mentioning NDB, do your backups have that enabled?

can you try disabling it and testing backup if you do have it on? I think i saw someone else post issue with NDB code in the forums.

RobWhalley

Thanks @marcoi - I don't believe so, though I have no experience of the Network Block Device protocol. The VMs are backed up to a NFS share on a Synology NAS, so pretty low tech in comparison to what a lot of folks here will be using. Based on this page, I'm guessing NBD isn't natively supported on these devices?

This is a sample of the settings, let me know if you need any further information:

marcoi

@RobWhalley Okay you have similar setup to me then. Can you confirm NFS share for backup is working and enabled?
it under settings -->Remote

Make sure the enabled is green and try clicking on the test icon in far right.