I can imagine that a fix could be to send "keepalive" packets in addition to the XCP-ng export-VM-data-stream so that the timeout on XO side does not occur

Also having issue. Didnt see the issue until my Remote SR has ran out of free space. { "data": { "mode": "delta", "reportWhen": "always", "backupReportTpl": "compactMjml", "hideSuccessfulItems": true }, "id": "1762146000010", "jobId": "95ac8089-69f3-404e-b902-21d0e878eec2", "jobName": "Backup Job 1", "message": "backup", "scheduleId": "76989b41-8bcf-4438-833a-84ae80125367", "start": 1762146000010, "status": "success", "infos": [ { "data": { "vms": [ "b25a5709-f1f8-e942-f0cc-f443eb9b9cf3", "5e00a752-177e-33f1-c5f5-ade59826cdab", "836db883-fdca-7458-0308-f225bbecbc29", "ce624d33-8f9c-8587-771b-194a161bdfed", "bce2b7f4-d602-5cdf-b275-da9554be61d3", "e0a3093a-52fd-f8dc-1c39-075eeb9d0314", "f0a673d7-dfb7-5766-86be-6dab83ac264b" ] }, "message": "vms" } ], "tasks": [ { "data": { "type": "VM", "id": "5e00a752-177e-33f1-c5f5-ade59826cdab", "name_label": "XO" }, "id": "1762146003418:0", "message": "backup VM", "start": 1762146003418, "status": "success", "tasks": [ { "id": "1762146003424", "message": "clean-vm", "start": 1762146003424, "status": "success", "end": 1762146003569, "result": { "merge": false } }, { "id": "1762146003933", "message": "snapshot", "start": 1762146003933, "status": "success", "end": 1762146005573, "result": "f8159d5c-c1b3-e4b9-155f-70bc2d2a8325" }, { "data": { "id": "b86b4c88-79f9-4ce8-9fa4-dce78a32ea44", "isFull": false, "type": "remote" }, "id": "1762146005573:0", "message": "export", "start": 1762146005573, "status": "success", "tasks": [ { "id": "1762146009606", "message": "transfer", "start": 1762146009606, "status": "success", "end": 1762146316193, "result": { "size": 25839009792 } }, { "id": "1762146317322", "message": "health check", "start": 1762146317322, "status": "success", "tasks": [ { "id": "1762146317328", "message": "transfer", "start": 1762146317328, "status": "success", "end": 1762146462139, "result": { "size": 0, "id": "bf74ff0e-a2e4-deab-bd59-f51a8985fee3" } }, { "id": "1762146462139:0", "message": "vmstart", "start": 1762146462139, "status": "success", "end": 1762146470513 } ], "end": 1762146472941 }, { "id": "1762146472956", "message": "clean-vm", "start": 1762146472956, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/5e00a752-177e-33f1-c5f5-ade59826cdab/20251103T050009Z.json", "actual": 25839009792, "expected": 25845451264 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1762146473085, "result": { "merge": false } } ], "end": 1762146473086 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:ec4c2c3f-3be7-7533-8a0e-810d36a007a9" }, "message": "Snapshot data has been deleted" } ], "end": 1762146473086 }, { "data": { "type": "VM", "id": "836db883-fdca-7458-0308-f225bbecbc29", "name_label": "Veeam" }, "id": "1762146473089", "message": "backup VM", "start": 1762146473089, "status": "success", "tasks": [ { "id": "1762146473094", "message": "clean-vm", "start": 1762146473094, "status": "success", "end": 1762146473361, "result": { "merge": false } }, { "id": "1762146473566", "message": "snapshot", "start": 1762146473566, "status": "success", "end": 1762146476087, "result": "ae3273f7-8044-63b3-d552-a6c41918c921" }, { "id": "1762147064647", "message": "health check", "start": 1762147064647, "status": "success", "infos": [ { "message": "This VM doesn't match the health check's tags for this schedule" } ], "end": 1762147064647 }, { "data": { "id": "b86b4c88-79f9-4ce8-9fa4-dce78a32ea44", "isFull": false, "type": "remote" }, "id": "1762146476087:0", "message": "export", "start": 1762146476087, "status": "success", "tasks": [ { "id": "1762146479793", "message": "transfer", "start": 1762146479793, "status": "success", "end": 1762147062777, "result": { "size": 88705335296 } }, { "id": "1762147064653", "message": "clean-vm", "start": 1762147064653, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/836db883-fdca-7458-0308-f225bbecbc29/20251103T050759Z.json", "actual": 88705335296, "expected": 88727520256 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1762147065070, "result": { "merge": false } } ], "end": 1762147065071 } ], "warnings": [ { "message": "Backup fell back to a full" }, { "message": "Backup fell back to a full" } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:d50ed8b7-a642-c9ec-29bb-2933401ba26b" }, "message": "Snapshot data has been deleted" }, { "data": { "vdiRef": "OpaqueRef:eb35900c-8e97-7bff-ca5a-ae8aced9897a" }, "message": "Snapshot data has been deleted" } ], "end": 1762147065071 }, { "data": { "type": "VM", "id": "ce624d33-8f9c-8587-771b-194a161bdfed", "name_label": "Bookstack" }, "id": "1762147065075", "message": "backup VM", "start": 1762147065075, "status": "success", "tasks": [ { "id": "1762147065080", "message": "clean-vm", "start": 1762147065080, "status": "success", "end": 1762147065172, "result": { "merge": false } }, { "id": "1762147065258", "message": "snapshot", "start": 1762147065258, "status": "success", "end": 1762147067979, "result": "dd829de3-4fd1-a027-e039-726834458b61" }, { "data": { "id": "b86b4c88-79f9-4ce8-9fa4-dce78a32ea44", "isFull": true, "type": "remote" }, "id": "1762147067979:0", "message": "export", "start": 1762147067979, "status": "success", "tasks": [ { "id": "1762147069665", "message": "transfer", "start": 1762147069665, "status": "success", "end": 1762147176187, "result": { "size": 7098859520 } }, { "id": "1762147177274:0", "message": "health check", "start": 1762147177274, "status": "success", "tasks": [ { "id": "1762147177280", "message": "transfer", "start": 1762147177280, "status": "success", "end": 1762147231441, "result": { "size": 0, "id": "36c43dbc-51b0-f059-3268-224e46a88840" } }, { "id": "1762147231441:0", "message": "vmstart", "start": 1762147231441, "status": "success", "end": 1762147239586 } ], "end": 1762147242037 }, { "id": "1762147242051", "message": "clean-vm", "start": 1762147242051, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/ce624d33-8f9c-8587-771b-194a161bdfed/20251103T051749Z.json", "actual": 7098859520, "expected": 7100725760 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1762147242435, "result": { "merge": false } } ], "end": 1762147242441 } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:deb7a513-2a0f-449a-7112-3ff34a4c527d" }, "message": "Snapshot data has been deleted" } ], "end": 1762147242441 }, { "data": { "type": "VM", "id": "b25a5709-f1f8-e942-f0cc-f443eb9b9cf3", "name_label": "SeedBox" }, "id": "1762146003418", "message": "backup VM", "start": 1762146003418, "status": "success", "tasks": [ { "id": "1762146003423", "message": "clean-vm", "start": 1762146003423, "status": "success", "end": 1762146003667, "result": { "merge": false } }, { "id": "1762146004138", "message": "snapshot", "start": 1762146004138, "status": "success", "end": 1762146006647, "result": "713fce77-e557-d646-9564-09ec8899c7c6" }, { "id": "1762147876503:0", "message": "health check", "start": 1762147876503, "status": "success", "infos": [ { "message": "This VM doesn't match the health check's tags for this schedule" } ], "end": 1762147876505 }, { "data": { "id": "b86b4c88-79f9-4ce8-9fa4-dce78a32ea44", "isFull": false, "type": "remote" }, "id": "1762146006647:0", "message": "export", "start": 1762146006647, "status": "success", "tasks": [ { "id": "1762146009248", "message": "transfer", "start": 1762146009248, "status": "success", "end": 1762147875274, "result": { "size": 81128325120 } }, { "id": "1762147876510", "message": "clean-vm", "start": 1762147876510, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/b25a5709-f1f8-e942-f0cc-f443eb9b9cf3/20251103T050009Z.json", "actual": 81128325120, "expected": 81148396032 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1762147876874, "result": { "merge": false } } ], "end": 1762147876875 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:556f651e-a2f8-0ade-eba9-aa4bf60a932d" }, "message": "Snapshot data has been deleted" } ], "end": 1762147876875 }, { "data": { "type": "VM", "id": "bce2b7f4-d602-5cdf-b275-da9554be61d3", "name_label": "iVentoy" }, "id": "1762147242451", "message": "backup VM", "start": 1762147242451, "status": "success", "tasks": [ { "id": "1762147242490", "message": "clean-vm", "start": 1762147242490, "status": "success", "end": 1762147242740, "result": { "merge": false } }, { "id": "1762147242958", "message": "snapshot", "start": 1762147242958, "status": "success", "end": 1762147244657, "result": "810aa1cf-61ee-a968-3409-2c8721a46832" }, { "data": { "id": "b86b4c88-79f9-4ce8-9fa4-dce78a32ea44", "isFull": false, "type": "remote" }, "id": "1762147244657:0", "message": "export", "start": 1762147244657, "status": "success", "tasks": [ { "id": "1762147248730", "message": "transfer", "start": 1762147248730, "status": "success", "end": 1762147584610, "result": { "size": 71422705664 } }, { "id": "1762147586650", "message": "health check", "start": 1762147586650, "status": "success", "tasks": [ { "id": "1762147586679", "message": "transfer", "start": 1762147586679, "status": "success", "end": 1762148076914, "result": { "size": 0, "id": "f1aa67ee-f368-d459-f93b-bc415ecd6a87" } }, { "id": "1762148076914:0", "message": "vmstart", "start": 1762148076914, "status": "success", "end": 1762148080972 } ], "end": 1762148083243 }, { "id": "1762148083285", "message": "clean-vm", "start": 1762148083285, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/bce2b7f4-d602-5cdf-b275-da9554be61d3/20251103T052048Z.json", "actual": 71422705664, "expected": 71440669184 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1762148084735, "result": { "merge": false } } ], "end": 1762148084739 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:923380c6-4bb9-8741-8a87-03d16bc94151" }, "message": "Snapshot data has been deleted" } ], "end": 1762148084739 }, { "data": { "type": "VM", "id": "f0a673d7-dfb7-5766-86be-6dab83ac264b", "name_label": "UnifiOS" }, "id": "1762148084748", "message": "backup VM", "start": 1762148084748, "status": "success", "tasks": [ { "id": "1762148084782", "message": "clean-vm", "start": 1762148084782, "status": "success", "end": 1762148085288, "result": { "merge": false } }, { "id": "1762148085501", "message": "snapshot", "start": 1762148085501, "status": "success", "end": 1762148087396, "result": "b8e5a515-6028-3dbb-ac3d-24ef7d6c6c96" }, { "data": { "id": "b86b4c88-79f9-4ce8-9fa4-dce78a32ea44", "isFull": false, "type": "remote" }, "id": "1762148087396:0", "message": "export", "start": 1762148087396, "status": "success", "tasks": [ { "id": "1762148091153", "message": "transfer", "start": 1762148091153, "status": "success", "end": 1762148365286, "result": { "size": 24178065408 } }, { "id": "1762148366667:0", "message": "health check", "start": 1762148366667, "status": "success", "tasks": [ { "id": "1762148366673", "message": "transfer", "start": 1762148366673, "status": "success", "end": 1762148547340, "result": { "size": 0, "id": "35638498-aa3a-6f69-de69-47d867b7927f" } }, { "id": "1762148547340:0", "message": "vmstart", "start": 1762148547340, "status": "success", "end": 1762148555601 } ], "end": 1762148558079 }, { "id": "1762148558093", "message": "clean-vm", "start": 1762148558093, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/f0a673d7-dfb7-5766-86be-6dab83ac264b/20251103T053451Z.json", "actual": 24178065408, "expected": 24184232448 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1762148558480, "result": { "merge": false } } ], "end": 1762148558481 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:f053e31e-284a-ae00-741c-eff4e9f97f01" }, "message": "Snapshot data has been deleted" } ], "end": 1762148558481 }, { "data": { "type": "VM", "id": "e0a3093a-52fd-f8dc-1c39-075eeb9d0314", "name_label": "Docker of Things" }, "id": "1762147876878", "message": "backup VM", "start": 1762147876878, "status": "success", "tasks": [ { "id": "1762147876884", "message": "clean-vm", "start": 1762147876884, "status": "success", "end": 1762147877204, "result": { "merge": false } }, { "id": "1762147877403", "message": "snapshot", "start": 1762147877403, "status": "success", "end": 1762147880513, "result": "2753d0e3-1cd5-c34c-2e5e-3c6dba85633d" }, { "data": { "id": "b86b4c88-79f9-4ce8-9fa4-dce78a32ea44", "isFull": false, "type": "remote" }, "id": "1762147880513:0", "message": "export", "start": 1762147880513, "status": "success", "tasks": [ { "id": "1762147884329", "message": "transfer", "start": 1762147884329, "status": "success", "end": 1762148429641, "result": { "size": 65131249664 } }, { "id": "1762148430966:0", "message": "health check", "start": 1762148430966, "status": "success", "tasks": [ { "id": "1762148430991", "message": "transfer", "start": 1762148430991, "status": "success", "end": 1762148812721, "result": { "size": 0, "id": "734681b1-e706-b96b-9efb-168acac01bb5" } }, { "id": "1762148812721:0", "message": "vmstart", "start": 1762148812721, "status": "success", "end": 1762148818820 } ], "end": 1762148820961 }, { "id": "1762148820974", "message": "clean-vm", "start": 1762148820974, "status": "success", "warnings": [ { "data": { "path": "/xo-vm-backups/e0a3093a-52fd-f8dc-1c39-075eeb9d0314/20251103T053124Z.json", "actual": 65131249664, "expected": 65147677184 }, "message": "cleanVm: incorrect backup size in metadata" } ], "end": 1762148821309, "result": { "merge": false } } ], "end": 1762148821310 } ], "warnings": [ { "message": "Backup fell back to a full" } ], "infos": [ { "message": "Transfer data using NBD" }, { "message": "will delete snapshot data" }, { "data": { "vdiRef": "OpaqueRef:68285439-7986-bd76-a88a-5da4a381d548" }, "message": "Snapshot data has been deleted" } ], "end": 1762148821310 } ], "end": 1762148821310 } [image: 1762271220739-screenshot-2025-11-04-100005.png]

@Danp Looking into it looks like VMs were doing full backups everytime when suppose to be delta backups... Let me know where else i can look to verify, but this is what i see when i look at the backups. [image: 1762268475129-screenshot-2025-11-04-095944.png] [image: 1762268480566-screenshot-2025-11-04-100005.png]

@Forza We plan to make it configurable during the upcoming months.

Just connect your XOA to your dedicated network and it will flow to it.

Question for @florent

Whelp had 2 of them get stuck, and one fail from the automated backup this time. The first one where it failed completely doesn't happen very often, so I'm less worried about it. Just including it in case it might be relevant. [image: 1761925465159-98feeae2-c301-4c92-bb33-bb4a029b0bc3-image.png]

@olivierlambert Host started and issue resolved.

Question for @pierrebrunet I assume

@olivierlambert Thanks that worked. Kind of obvious approach, still didn't try that

Are you sure the job isn't already running, as per the error message?

Backup job still failing. I have open a support ticket - Ticket#7746281

Hi @tmk Thanks for the feedback, that's interesting. I'm adding @florent in the loop

@stevewest15 I believe this issue was resolved when the health check system was changed to detect network connectivity at startup so it did not need to wait for then entire VM to boot. Needs the Xen tools to be installed. I have not had an issue since this change.

This look great! Something I was missing. But unfortunately I get an authentication error as well. I do not get it running. I am using PowerShell 7.5 and I am using a cert that is singed by my own CA. This CA is included in the Windows cert store for trusted root certificates. With all browser this works fine. Hopefully someone has an idea?

@olivierlambert minor update... its now been 48+ hours and task is still in the same state, not timed out yet.

Since I'm not used to using git, I was running a few tests on my "test" host. I had made a change in my "one line commando" My memory is not in best shape so I forgot the change and ran it in my update. Now It's sorted out and ran a new update and voila

@olivierlambert Here's the new model. We've tried a few combinations and I think with TrueNAS shared storage this will now work well. [image: 1759787234216-f78178c0-3330-4233-8f9b-debc3c61a3e9-image.png]