fixed in commit fcefa

Excellent!

@olivierlambert yes, I started to work on friday, and hopefully will have a fix today

@lsouai-vates P.S. This HAS been working a while ago!! I just noticed that it broke when there was a backup job, which had an error and I didn't receive an email. So I can't say when the issue started.

@manilx we will consider it in XO-6 to improve the backup view, thanks for you reply

@manilx yes, that's it we should change the "full backup interval" to "base/complete backup interval", to clarify what is a full . And we will do it.

I just had a backup fail with a similar error. Details: XO Community Xen Orchestra, commit 749f0 Master, commit 749f0 Merge backups synchronously was off on the backup job, going to enable it. [image: 1740267400062-88c76052-e8f1-43dc-a218-5794d46ebaad-image.png] { "data": { "type": "VM", "id": "4f715b32-ddfb-5818-c7bd-aaaa2a77ce70", "name_label": "PROD_SophosXG" }, "id": "1740263738986", "message": "backup VM", "start": 1740263738986, "status": "failure", "warnings": [ { "message": "the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution." } ], "end": 1740263738993, "result": { "code": "ELOCKED", "file": "/run/xo-server/mounts/f992fff1-e245-48f7-8eb3-25987ecbfbd4/xo-vm-backups/4f715b32-ddfb-5818-c7bd-aaaa2a77ce70", "message": "Lock file is already being held", "name": "Error", "stack": "Error: Lock file is already being held\n at /opt/xo/xo-builds/xen-orchestra-202502212211/node_modules/proper-lockfile/lib/lockfile.js:68:47\n at callback (/opt/xo/xo-builds/xen-orchestra-202502212211/node_modules/graceful-fs/polyfills.js:306:20)\n at FSReqCallback.oncomplete (node:fs:199:5)\n at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)\nFrom:\n at NfsHandler.addSyncStackTrace (/opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/fs/dist/local.js:21:26)\n at NfsHandler._lock (/opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/fs/dist/local.js:135:48)\n at NfsHandler.lock (/opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/fs/dist/abstract.js:234:27)\n at IncrementalRemoteWriter.beforeBackup (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_writers/_MixinRemoteWriter.mjs:54:34)\n at async IncrementalRemoteWriter.beforeBackup (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_writers/IncrementalRemoteWriter.mjs:68:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:343:7\n at async callWriter (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_vmRunners/_Abstract.mjs:33:9)\n at async IncrementalXapiVmBackupRunner.run (file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/_vmRunners/_AbstractXapi.mjs:342:5)\n at async file:///opt/xo/xo-builds/xen-orchestra-202502212211/@xen-orchestra/backups/_runners/VmsXapi.mjs:166:38" } }, Email report of failure VM Backup report Global status : failure 🚨 • Job ID: bfadfecc-b651-4fd6-b104-2f015100db29 • Run ID: 1740263706121 • Mode: delta • Start time: Saturday, February 22nd 2025, 5:35:06 pm • End time: Saturday, February 22nd 2025, 5:37:59 pm • Duration: 3 minutes • Successes: 0 / 8 1 Failure PROD_SophosXG Production Sophos Firewall Application • UUID: 4f715b32-ddfb-5818-c7bd-aaaa2a77ce70 • Start time: Saturday, February 22nd 2025, 5:35:38 pm • End time: Saturday, February 22nd 2025, 5:35:38 pm • Duration: a few seconds • ⚠️ the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution. • Error: Lock file is already being held Manually run completed PROD_SophosXG (xcp02) Clean VM directory cleanVm: incorrect backup size in metadata Start: 2025-02-22 18:37 End: 2025-02-22 18:37 Snapshot Start: 2025-02-22 18:37 End: 2025-02-22 18:37 Qnap NFS Backup transfer Start: 2025-02-22 18:37 End: 2025-02-22 18:38 Duration: a minute Size: 12.85 GiB Speed: 185.52 MiB/s Start: 2025-02-22 18:37 End: 2025-02-22 18:44 Duration: 7 minutes Start: 2025-02-22 18:37 End: 2025-02-22 18:44 Duration: 7 minutes Type: delta

@manilx Just happened on our production Pools. Snapshotting Windows Server 2022 crashes our ERP VM! Opening official ticket referencing this thread.

@olivierlambert Great. Posted here and didn't open ticket because this is interesting for all ...

Ping @florent

@florent Looking forward to the docs. Just this one thing (Guess most would be interested): On my example: keeping 7 daily backups, I presume it keeps the last backup from each day (as there are 2 schedules I presume it would keep the backup from 22:00)

@olivierlambert If the issue turns up again.... It was a weird one! As I deleted the previous backup jobs I can't repeat. Just had to move on. Perhaps someone else stumbles upon this and finds this thread...... I will report if it happens again (will try it wit the new backup)

@imaginapix Didn't think of that.... But I guess it's more work than it's worth in this instance.

@julien-f Tried again with today's release Xen Orchestra, commit 3d054 Master, commit 3d054 Same error again https://paste.vates.tech/?4941198dd850ed6d#ArUk6wqkwE2c3CQRbjwAXRERJd2Y4HWF5oq9ysMdkbBp Mirror incremental backup is broken here

@florent Did you find anything? I just retested the job on latest XO build and have the same error.

@Andrew Yes, that offscreen was the problem..... Perhaps place it more to the left.

@manilx Nice was about to ask this too lol. Glad someone point this out already

@manilx It is not cheaper, but it draws about half the power when idle. When doing work the power still gets up there, they are 60-90 watt computers, which is still less than the 300 watt capable HP servers at full go. If you really stress your lab a lot, real servers are a win as they will draw less power for more performance, but if yours is like mine and it idles a lot, cutting the power is nice. I'm about 200 watts at idle with the above, and that's two labs worth. I was 400 with the single old lab system. Mounts are cheap book ends, drilled and screwed to that specific shelf, and drilled for short m4 thumbscrews to get into the VESA mount on the t740. If anyone reading this wants to duplicate, and they see HP T755 cheap enough, I recommend these because it has 6 cores / 12 threads over the t740 with 4 cores / 8 threads. Both are DDR4 sodimm and top out at 64GB. Both have a PCI 3.0 x8 (x16 slot) that takes half height cards. And if the BIOS is locked, you can write over them with a programmer and enter the data needed to function (see badcaps forum for my guide).

@darkbounce Short answer: it seems that Intel hybrid architecture is finding a way to make something that would result of Xen running the VM with the less "featured" CPU, meaning your VM will never use all P core instructions that aren't on E core, meaning... it will work without disabling one or another. Hard to tell how much is lost in terms of perfs vs the best Intel hybrid scheduler (which is on... Windows) but probably not that much on a machine with a reduced number of P cores like this one.

Question for @florent