the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution.
-
@manilx said in the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution.:
For a few days I'm getting this error every now and then
the writer IncrementalRemoteWriter has failed the step writer.beforeBackup() with error Lock file is already being held. It won't be used anymore in this job execution.The next backup (every 2hrs) works fine again. Nothing has changed in my setup. The affected VM's are random.
Weird.
Basically a backup job (or some other process) has this file already locked and it is unable to complete the backup successfully.
This could be due to an IOPS issue, overlapping backup jobs or even an AV scan that might be scanning your VM's repo.
-
@DustinB Yes BUT there are no concurrent backup jobs, no AV installed anywhere. And backups are 200Mb/s and take only 2-10 mins.
So this is why I reported this.
-
Can you try to enable "Merge backups synchronously" in the advanced backup settings for your job? And report if you still have the issue then
-
@olivierlambert I have it enabled all the time. And I'm now getting these errors now each day. Most of the time on the 1st backup of the day.
I get it on Delta, Mirror incremental backup
-
@manilx This has started 2 weeks go or so. But it's getting worse. I'm pretty sure it started with one of the XO updates..... As nothing else was changed.
-
Are you on XOA or XO sources? Which version?
-
@olivierlambert XO Xen Orchestra, commit 987e9
Master, commit ef636 -
I'm not sure to understand, why two different commit numbers?
-
@olivierlambert That's the last update I did yesterday.....
Just updated now to latest: Xen Orchestra, commit ef636
Master, commit ef636 -
Do you have the issue on XOA?
-
@olivierlambert @Business with XOA no. Only HomeLab with XO but it has been rockstable until this.
-
this happens when 2 backup jobs are working on the same VM , or a backup and a background merge worker .
If it's the first case, you can use a job sequence to ensure the backup run one after anotherIf it's the latter, you have a check box to ensure the merge is done directly inside the job (bottom of the advanced settings of the backup job ), instead of doing this in a background job. It will slow down a little the job, but ensure it's completely done.
-
@florent No other backup is running as I mentioned!
And "Merge backups synchronously" is on as mentioned.
This is not it.....
-
@manilx are there multiple job running on the same VM ?
-
@florent Only one single backup job every 2hrs (multiple VM's). Running fore more than 1year without issues (not counting the initianl cbt stuff).
-
@florent This make no sense. There has to be some leftover sometimes of a lock file.
-
So I started getting this a few days ago as well. My backups have been fine for over a year until now, they are constantly failing at random. They will usually succeed just fine if I restart the failed backup. I am currently on commit 494ea. I haven't had time to delve into this though. I just found this post when initially googling.
-
That's interesting feedback, maybe a recently introduced bug. Could you try to go on an older commit to see if you can identify the culprit?
-
@SudoOracle Thx for chiming in!
Glad to know I'm not alone.
-
@olivierlambert I went back to my prior version, which was commit 04dd9. And the very next backup finished successfully. Not sure if this was a fluke, but I have backups going every 2 hours and this was the first to finish successfully today without me clicking the button to retry the backup. I'll report tomorrow on whether they continue to work or not.
