You already spotted this over on 12468, so this is just to close the loop here. That one's fixed: Pierre Brunet put the fix on a branch, @Andrew and @abudef both tested it, and it's merged now. Since you're running from source on master, you should be able to pull past that merge rather than wait for a release. One thing to keep in mind, though: the same error string showed up in 11875 for a completely different cause, so if it comes back after you update, it's probably not this bug again. I'd let 12468 be the thread to follow since that's where the people who fixed it are.

@poddingue You're right. The RPU task shown in the list was a previous one. Even that one finished fine. So both RPU tasks didn't show at all today! My bad. The evacuation and remirroring tasks were all shown. Latest XOA 6.7.1

I think I added it after the video ^^ (between the recording and the official release)

@jr-m4 Hi, We have found this issue in another thread, it is now merged

fixed in commit fcefa

Excellent!

@olivierlambert yes, I started to work on friday, and hopefully will have a fix today

@lsouai-vates P.S. This HAS been working a while ago!! I just noticed that it broke when there was a backup job, which had an error and I didn't receive an email. So I can't say when the issue started.

@manilx we will consider it in XO-6 to improve the backup view, thanks for you reply

@manilx yes, that's it we should change the "full backup interval" to "base/complete backup interval", to clarify what is a full . And we will do it.

I just had a backup fail with a similar error. Details: XO Community Xen Orchestra, commit 749f0 Master, commit 749f0 Merge backups synchronously was off on the backup job, going to enable it. [image: 88c76052-e8f1-43dc-a218-5794d46ebaad-image.png] { "data": { "type": "VM", "id": "4f715b32-ddfb-5818-c7bd-aaaa2a77ce70", "name_label": "PROD_SophosXG" }, "id": "1740263738986", "message": "backup VM", "start": 1740263738986, "status": "failure", "warnings": [ { "message": "the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution." } ], "end": 1740263738993, "result": { "code": "ELOCKED", "file": "/run/xo-server/mounts/f992fff1-e245-48f7-8eb3-25987ecbfbd4/xo-vm-backups/4f715b32-ddfb-5818-c7bd-aaaa2a77ce70", "message": "Lock file is already being held", "name": "Error", "stack": "Error: Lock file is already being held\n at /opt/xo/xo-builds/xen-orchestra-202502212211/node_modules/proper-lockfile/lib/lockfile.js:68:47\n at callback (/opt/xo/xo-builds/xen-orchestra-202502212211/node_modules/graceful-fs/polyfills.js:306:20)\n at FSReqCallback.oncomplete (node:fs:199:5)\n at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)\nFrom:\n at NfsHandler.addSyncStackTrace (/opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/fs/dist/local.js:21:26)\n at NfsHandler._lock (/opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/fs/dist/local.js:135:48)\n at NfsHandler.lock (/opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/fs/dist/abstract.js:234:27)\n at IncrementalRemoteWriter.beforeBackup (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:54:34)\n at async IncrementalRemoteWriter.beforeBackup (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:68:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:343:7\n at async callWriter (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:33:9)\n at async IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:342:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38" } }, Email report of failure VM Backup report Global status : failure 🚨 • Job ID: bfadfecc-b651-4fd6-b104-2f015100db29 • Run ID: 1740263706121 • Mode: delta • Start time: Saturday, February 22nd 2025, 5:35:06 pm • End time: Saturday, February 22nd 2025, 5:37:59 pm • Duration: 3 minutes • Successes: 0 / 8 1 Failure PROD_SophosXG Production Sophos Firewall Application • UUID: 4f715b32-ddfb-5818-c7bd-aaaa2a77ce70 • Start time: Saturday, February 22nd 2025, 5:35:38 pm • End time: Saturday, February 22nd 2025, 5:35:38 pm • Duration: a few seconds • ⚠️ the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution. • Error: Lock file is already being held Manually run completed PROD_SophosXG (xcp02) Clean VM directory cleanVm: incorrect backup size in metadata Start: 2025-02-22 18:37 End: 2025-02-22 18:37 Snapshot Start: 2025-02-22 18:37 End: 2025-02-22 18:37 Qnap NFS Backup transfer Start: 2025-02-22 18:37 End: 2025-02-22 18:38 Duration: a minute Size: 12.85 GiB Speed: 185.52 MiB/s Start: 2025-02-22 18:37 End: 2025-02-22 18:44 Duration: 7 minutes Start: 2025-02-22 18:37 End: 2025-02-22 18:44 Duration: 7 minutes Type: delta

@manilx Just happened on our production Pools. Snapshotting Windows Server 2022 crashes our ERP VM! Opening official ticket referencing this thread.

@olivierlambert Great. Posted here and didn't open ticket because this is interesting for all ...

Ping @florent

@florent Looking forward to the docs. Just this one thing (Guess most would be interested): On my example: keeping 7 daily backups, I presume it keeps the last backup from each day (as there are 2 schedules I presume it would keep the backup from 22:00)

@olivierlambert If the issue turns up again.... It was a weird one! As I deleted the previous backup jobs I can't repeat. Just had to move on. Perhaps someone else stumbles upon this and finds this thread...... I will report if it happens again (will try it wit the new backup)

@imaginapix Didn't think of that.... But I guess it's more work than it's worth in this instance.

@julien-f Tried again with today's release Xen Orchestra, commit 3d054 Master, commit 3d054 Same error again https://paste.vates.tech/?4941198dd850ed6d#ArUk6wqkwE2c3CQRbjwAXRERJd2Y4HWF5oq9ysMdkbBp Mirror incremental backup is broken here

@florent Did you find anything? I just retested the job on latest XO build and have the same error.

@Andrew Yes, that offscreen was the problem..... Perhaps place it more to the left.