@TS79
That makes sense - thanks!

While testing NDB over the last few days, I saw this note ”Remember that it works ONLY with a remote (Backup Repository) configured with "multiple data blocks".” I then saw this note:
“Store backup as multiple data blocks instead of a whole VHD file. (creates 500-1000 files per backed up GB but allows faster merge)”

I set one up and backedup a small VM and, sure enough, with a 5gb VM, it made 15,587 files. I can only imagine a large VM drive would generate a massive amount of files.
Questions:

1. Is there anything of concern with respect to generating all these files on the NFS server vs just a couple of large files?

2. One area said “your remote must be able to handle parallel access (up to 16 write processes per backup)”…how does one know if it will do so?

3. Also, do you still need to do this: “edit your XO config.toml file (near the end of it) with:
   [backups]
   useNbd = true

4. Does anyone use this vs the other methods of backup?

Thanks!

Update:

It’s weird:

• There are three VM’s on this host. The backup works with two but not with the third. It does with the “Body Timeout Error” error.
• Two of the VM’s are almost identical (same drive sizes). The only difference is that one was set up as “Other install media” (came over from esxi) and the one that fails was set up using the “Windows Server 2022” template.
• I normally backup to multiple NFS servers so I changed to try one at a time; both failed.
• After watching it do the backup too many times to thing, I found that, at about the 95% stage, the snapshot stops writing to the NFS share.
• About that time, the file /var/log/xensource.log records this information:
o Feb 26 09:43:33 HOST1 xapi: [error||19977 HTTPS 192.168.1.5->:::80|[XO] VM export R:c14f4c4c1c4c|xapi_compression] nice failed to compress: exit code 70
o Feb 26 09:43:33 HOST1 xapi: [ warn||19977 HTTPS 192.168.1.5->:::80|[XO] VM export R:c14f4c4c1c4c|pervasiveext] finally: Error while running cleanup after failure of main function: Failure("nice failed to compress: exit code 70")
o Feb 26 09:43:33 HOST1 xapi: [debug||922 |xapi events D:d21ea5c4dd9a|xenops] Event on VM fbcbd709-a9d9-4cc7-80de-90185a74eba4; resident_here = true
o Feb 26 09:43:33 HOST1 xapi: [debug||922 |xapi events D:d21ea5c4dd9a|dummytaskhelper] task timeboxed_rpc D:a08ebc674b6d created by task D:d21ea5c4dd9a
o Feb 26 09:43:33 HOST1 xapi: [debug||922 |timeboxed_rpc D:a08ebc674b6d|xmlrpc_client] stunnel pid: 339060 (cached) connected to 192.168.1.6:443
o Feb 26 09:43:33 HOST1 xapi: [debug||19977 HTTPS 192.168.1.5->:::80|[XO] VM export R:c14f4c4c1c4c|xmlrpc_client] stunnel pid: 296483 (cached) connected to 192.168.1.6:443
o Feb 26 09:43:33 HOST1 xapi: [error||19977 :::80|VM.export D:c62fe7a2b4f2|backtrace] [XO] VM export R:c14f4c4c1c4c failed with exception Server_error(CLIENT_ERROR, [ INTERNAL_ERROR: [ Unix.Unix_error(Unix.EPIPE, "write", "") ] ])
o Feb 26 09:43:33 HOST1 xapi: [error||19977 :::80|VM.export D:c62fe7a2b4f2|backtrace] Raised Server_error(CLIENT_ERROR, [ INTERNAL_ERROR: [ Unix.Unix_error(Unix.EPIPE, "write", "") ] ])
• I have no idea if it means anything but the “failed to compress” made me try something. I change “compression” from “Zstd” to “disabled” and that time it worked.

Here are my results:

1. Regular backup to TrueNas, “compression” set to “Zstd”: backup fails.
2. Regular backup to TrueNas, “compression” set to “disabled”: backup is successful.
3. Regular backup to vanilla Ubuntu test VM, “compression” set to “Zstd”: backup is successful.
4. Delta backup to TrueNas: backup is successful.

Sooooo…the $64,000 question is why doesn’t it work on that one VM when compression is on and it a TrueNas box?

@archw
I didn't even know that was a thing....thanks!
Question:

Lets say dummy_remote-1 was where I write data and the mirror job copies the data from dummy_remote-1 to dummy_remote-2.

1. What happens if the backup to dummy_remote-1 fails part of the way through the backup? I don't know what gets left behind in a partial backup but I assume its a bunch of nothingness/worthless data?? Would the mirror copy the failed data over the good data on dummy_remote-2

2. What happens if the backup to dummy_remote-1 got wiped out by a bad actor (erased)? Would the mirror copy the zerored out data over the good data on dummy_remote-2

@olivierlambert
Very cool...thanks for the explanation (but I'll admit I had to google "Node streams")!

Many of my backups are set to go to multiple remotes.When the backup finishes, the status shows up and that they finished at the same time.

One is a faster backup device than the other so how does it work? Does it write parallel data/blocks/etc to one share and then wait for the other to share catch up or is it some other machination?

Just curious.

@TS79

It also backs up two very similar VM's so I went to the status of their last backup and they show this:

VM1
Duration: 8 hours
Size: 1.1 TiB
Speed: 42.2 MiB/s

VM2
Duration: 5 hours
Size: 676.8 GiB
Speed: 41.29 MiB/s

Its a 10GB connection but the storage media is an NFS share made from 7200 RPM SATA drives (I'm jealous of your nvme!).

I dont have a time out set but for the heck of it I just set one at 8 hours. Also, I have the backup set to go to two NFS shares. I also just changed it to only go to one and just started it. If that fails, I'll change to the other and see what happens.

Most of the backup occur with any error but I have a backup that keeps failing with "Body Timeout Error" it says it is dying at the four hour mark.

It is backingup up to two NFS boxes. Liek I said above, it works fine for most of the other VMs however one other one does this every once in a while.

Backup running on Xen Orchestra, commit f18b0

Any ideas?
Thanks!

The log file is as follows:

{
"data": {
"mode": "full",
"reportWhen": "failure"
},
"id": "1740449460003",
"jobId": "2cc39019-5201-43f8-ad8a-d13870e948be",
"jobName": "Exchange-2025",
"message": "backup",
"scheduleId": "1f01cbda-bae3-4a60-a29c-4d88b79a38d2",
"start": 1740449460003,
"status": "failure",
"infos": [
{
"data": {
"vms": [
"558c3009-1e8d-b1e4-3252-9cf3915d1fe4"
]
},
"message": "vms"
}
],
"tasks": [
{
"data": {
"type": "VM",
"id": "558c3009-1e8d-b1e4-3252-9cf3915d1fe4",
"name_label": "VM1"
},
"id": "1740449463052",
"message": "backup VM",
"start": 1740449463052,
"status": "failure",
"tasks": [
{
"id": "1740449472998",
"message": "snapshot",
"start": 1740449472998,
"status": "success",
"end": 1740449481725,
"result": "d73124a9-d08c-4623-f28c-7503fcef9260"
},
{
"data": {
"id": "0d6da7ba-fde1-4b37-927b-b56c61ce8e59",
"type": "remote",
"isFull": true
},
"id": "1740449483456",
"message": "export",
"start": 1740449483456,
"status": "failure",
"tasks": [
{
"id": "1740449483464",
"message": "transfer",
"start": 1740449483464,
"status": "failure",
"end": 1740462845094,
"result": {
"name": "BodyTimeoutError",
"code": "UND_ERR_BODY_TIMEOUT",
"message": "Body Timeout Error",
"stack": "BodyTimeoutError: Body Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/dispatcher/client-h1.js:646:28)\n at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:581:17)\n at process.processTimers (node:internal/timers:519:7)"
}
}
],
"end": 1740462845094,
"result": {
"name": "BodyTimeoutError",
"code": "UND_ERR_BODY_TIMEOUT",
"message": "Body Timeout Error",
"stack": "BodyTimeoutError: Body Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/dispatcher/client-h1.js:646:28)\n at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:581:17)\n at process.processTimers (node:internal/timers:519:7)"
}
},
{
"data": {
"id": "656f82d9-5d68-4e26-a75d-b57b4cb17d5e",
"type": "remote",
"isFull": true
},
"id": "1740449483457",
"message": "export",
"start": 1740449483457,
"status": "failure",
"tasks": [
{
"id": "1740449483469",
"message": "transfer",
"start": 1740449483469,
"status": "failure",
"end": 1740462860953,
"result": {
"name": "BodyTimeoutError",
"code": "UND_ERR_BODY_TIMEOUT",
"message": "Body Timeout Error",
"stack": "BodyTimeoutError: Body Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/dispatcher/client-h1.js:646:28)\n at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:581:17)\n at process.processTimers (node:internal/timers:519:7)"
}
}
],
"end": 1740462860953,
"result": {
"name": "BodyTimeoutError",
"code": "UND_ERR_BODY_TIMEOUT",
"message": "Body Timeout Error",
"stack": "BodyTimeoutError: Body Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/dispatcher/client-h1.js:646:28)\n at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:581:17)\n at process.processTimers (node:internal/timers:519:7)"
}
},
{
"id": "1740463648921",
"message": "clean-vm",
"start": 1740463648921,
"status": "success",
"warnings": [
{
"data": {
"path": "/backup-storage/558c3009-1e8d-b1e4-3252-9cf3915d1fe4/.20250224T232125Z.xva"
},
"message": "unused XVA"
}
],
"end": 1740463649103,
"result": {
"merge": false
}
},
{
"id": "1740463649144",
"message": "clean-vm",
"start": 1740463649144,
"status": "success",
"warnings": [
{
"data": {
"path": "/backup-storage/558c3009-1e8d-b1e4-3252-9cf3915d1fe4/.20250224T232125Z.xva"
},
"message": "unused XVA"
}
],
"end": 1740463651524,
"result": {
"merge": false
}
}
],
"end": 1740463651532,
"result": {
"errors": [
{
"name": "BodyTimeoutError",
"code": "UND_ERR_BODY_TIMEOUT",
"message": "Body Timeout Error",
"stack": "BodyTimeoutError: Body Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/dispatcher/client-h1.js:646:28)\n at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:581:17)\n at process.processTimers (node:internal/timers:519:7)"
},
{
"name": "BodyTimeoutError",
"code": "UND_ERR_BODY_TIMEOUT",
"message": "Body Timeout Error",
"stack": "BodyTimeoutError: Body Timeout Error\n at FastTimer.onParserTimeout [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/dispatcher/client-h1.js:646:28)\n at Timeout.onTick [as _onTimeout] (/opt/xo/xo-builds/xen-orchestra-202502241838/node_modules/undici/lib/util/timers.js:162:13)\n at listOnTimeout (node:internal/timers:581:17)\n at process.processTimers (node:internal/timers:519:7)"
}
],
"message": "all targets have failed, step: writer.run()",
"name": "Error",
"stack": "Error: all targets have failed, step: writer.run()\n at FullXapiVmBackupRunner._callWriters (file:///opt/xo/xo-builds/xen-orchestra-202502241838/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:64:13)\n at async FullXapiVmBackupRunner._copy (file:///opt/xo/xo-builds/xen-orchestra-202502241838/@xen-orchestra/backups/_runners/_vmRunners/FullXapi.mjs:55:5)\n at async FullXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202502241838/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:396:9)\n at async file:///opt/xo/xo-builds/xen-orchestra-202502241838/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38"
}
}
],
"end": 1740463651533
}

This is in the FWIW department. Some of the upgrades (Windows server 2022 to server 2025) have gone through without an issue but two would not upgrade until I removed the Xenserver VM tools (version 9.4). As it was, it would do the upgrade, reboot twice and then sit with a spinning wheel.

Once I removed the Xenserver VM tools, the upgrade went through without issue.

@supportcphl

Many, many happy hours have since transpired

I ended up wiping out the XO vm that was running the process and making a new one. That seems to have fixed it.

With all that said, I got one again last night with backing up the same VM that has caused an issue in the past. I just told the backup to restart so lets see what happens.

FWIW...having the same issue with Server 2025.

Another weird thing I've noticed with Windows Server 2025 on XCP-ng.

The network keeps resetting itself to "Public" (vs domain). This only happens when the VM is a domain controller and it only happens with Server 2025. If you go into the VM's console and disable the NIC and then re-enable it, it returns to a domain network. I've tried the usual trick of change the "Network location awareness" to delayed start but it doesn't help.

@Chemikant784
So with this being patch Tuesday, two updates got installed:

2024-11 Cumulative Update for Microsoft server operating system version 24H2 for x64-based Systems (KB5046617)
2024-11 Cumulative Update for .NET Framework 3.5 and 4.8.1 for Microsoft server operating system version 24H2 for x64 (KB5045934)

For whatever reason, those processes are no longer showing up and I can run things from the scheduler??

If you updated tonight, did yours go away?

@Greg_E
For the heck of it, I set up a new VM with 2025 and made it a domain controller complete with AD DS. It seems to work fine but, like yours, there are thirty something conhost processes running.

I recently upgraded from 8.3 RC2 to 8.3 final. Ever since then I've had a terrible time getting backups...especially delta backups (going to NFS server).

1. Under tasks in XO (Xen Orchestra, commit ff118), it says "[XO] Exporting content of VDI blahblah-drive3 using NBD (on host1) 34%". I have tried rebooting the XO VM (after disabling teh backup task) as well as those that has VM with the data. The task will not go away. I've tried restarting the toolstack on every host....its still stuck.

2. If you go to the backup activity, it says interrupted and then "download logs" you get this:

{
  "data": {
    "mode": "delta",
    "reportWhen": "always"
  },
  "id": "1730477627182",
  "jobId": "7863cc0b-601e-440a-8328-1d1ef2a09972",
  "jobName": "HOST1-VM1-delta",
  "message": "backup",
  "scheduleId": "59590d78-6809-4eb8-97fe-ba938a68939e",
  "start": 1730477627182,
  "status": "interrupted",
  "infos": [
    {
      "data": {
        "vms": [
          "5e01568d-3fa9-11f8-e293-e576f15d5cdb"
        ]
      },
      "message": "vms"
    }
  ],
  "tasks": [
    {
      "data": {
        "type": "VM",
        "id": "5e01568d-3fa9-11f8-e293-e576f15d5cdb",
        "name_label": "HOST1-VM1 (20241014T050050Z)"
      },
      "id": "1730477630143",
      "message": "backup VM",
      "start": 1730477630143,
      "status": "interrupted",
      "tasks": [
        {
          "id": "1730477630162",
          "message": "clean-vm",
          "start": 1730477630162,
          "status": "success",
          "end": 1730477630497,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1730477630164",
          "message": "clean-vm",
          "start": 1730477630164,
          "status": "success",
          "end": 1730477630537,
          "result": {
            "merge": false
          }
        },
        {
          "id": "1730477631151",
          "message": "snapshot",
          "start": 1730477631151,
          "status": "success",
          "end": 1730477648388,
          "result": "e252e0e8-7c26-cd1c-3365-e519829f41e3"
        },
        {
          "data": {
            "id": "0d6da7ba-fde1-4b37-927b-b56c61ce8e59",
            "isFull": true,
            "type": "remote"
          },
          "id": "1730477648389",
          "message": "export",
          "start": 1730477648389,
          "status": "interrupted",
          "tasks": [
            {
              "id": "1730477651581",
              "message": "transfer",
              "start": 1730477651581,
              "status": "interrupted"
            }
          ]
        },
        {
          "data": {
            "id": "656f82d9-5d68-4e26-a75d-b57b4cb17d5e",
            "isFull": true,
            "type": "remote"
          },
          "id": "1730477648390",
          "message": "export",
          "start": 1730477648390,
          "status": "interrupted",
          "tasks": [
            {
              "id": "1730477651599",
              "message": "transfer",
              "start": 1730477651599,
              "status": "interrupted"
            }
          ]
        }
      ],
      "infos": [
        {
          "message": "Transfer data using NBD"
        }
      ]
    }
  ]
}

That happens to me about once every six weeks on one of the XO installs. For what its worth, if I just kill the XO vm and let it reboot, as long as its not out of disk space, it always works the second time.

@stormi
Just rebooted the problematic host. All the VMs autostarted just fine.

Odd!

@stormi
Yes

I've not had a chance to reboot the host since then to see if something else is going on. Will do so tonight.

@stormi
FWIW...one of the boxes where I installed the latest updates (from a few days ago) did not autostart any of the VMs. I had to manually start each of them.

@brezlord

A week or two ago a bunch of updates went out. Were these part of RC2? I updated via yum back then and since this announcement nothing new has come up.

Is there a command line command to run to verify what is installed now is RC2?