Warnings & errors in journalctl

Danp

Dec 08 00:00:13 xo-server[237842]: 2021-12-08T06:00:13.158Z xo:backups:MixinBackupWriter WARN the parent /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211125T060012Z.vhd of the child /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211126T060036Z.vhd is unused
Dec 08 00:00:13 xo-server[237842]: 2021-12-08T06:00:13.160Z xo:backups:MixinBackupWriter WARN merging /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211126T060036Z.vhd into /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211125T060012Z.vhd
Dec 08 00:00:23 xo-server[237842]: 2021-12-08T06:00:23.160Z xo:backups:MixinBackupWriter WARN merging /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211126T060036Z.vhd: 126/219
Dec 08 00:00:33 xo-server[237842]: 2021-12-08T06:00:33.161Z xo:backups:MixinBackupWriter WARN merging /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211126T060036Z.vhd: 184/219
Dec 08 00:00:38 xo-server[237842]: 2021-12-08T06:00:38.238Z xo:backups:MixinBackupWriter WARN the parent /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211125T060019Z.vhd of the child /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211126T060054Z.vhd is unused
Dec 08 00:00:38 xo-server[237842]: 2021-12-08T06:00:38.239Z xo:backups:MixinBackupWriter WARN merging /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211126T060054Z.vhd into /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211125T060019Z.vhd
Dec 08 00:00:48 xo-server[237842]: 2021-12-08T06:00:48.239Z xo:backups:MixinBackupWriter WARN merging /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211126T060054Z.vhd: 73/181
Dec 08 00:00:58 xo-server[237842]: 2021-12-08T06:00:58.240Z xo:backups:MixinBackupWriter WARN merging /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211126T060054Z.vhd: 137/181
Dec 08 00:01:06 xo-server[237842]: 2021-12-08T06:01:06.258Z xo:backups:MixinBackupWriter WARN the parent /xo-vm-backups/2e5c9076-c1ae-4c3b-6f56-940aef6451b0/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/07ac8fda-39eb-48e3-be35-2db747d1bb5f/20211023T050017Z.vhd of the child /xo-vm-backups/2e5c9076-c1ae-4c3b-6f56-940aef6451b0/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/07ac8fda-39eb-48e3-be35-2db747d1bb5f/20211024T050036Z.vhd is unused
Dec 08 00:01:07 xo-server[237922]: 2021-12-08T06:01:07.770Z xo:backups:mergeWorker INFO starting
Dec 08 00:01:08 xo-server[237922]: 2021-12-08T06:01:08.030Z xo:backups:mergeWorker INFO the parent /xo-vm-backups/2e5c9076-c1ae-4c3b-6f56-940aef6451b0/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/07ac8fda-39eb-48e3-be35-2db747d1bb5f/20211023T050017Z.vhd of the child /xo-vm-backups/2e5c9076-c1ae-4c3b-6f56-940aef6451b0/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/07ac8fda-39eb-48e3-be35-2db747d1bb5f/20211024T050036Z.vhd is unused
Dec 08 00:01:08 xo-server[237922]: 2021-12-08T06:01:08.031Z xo:backups:mergeWorker INFO merging /xo-vm-backups/2e5c9076-c1ae-4c3b-6f56-940aef6451b0/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/07ac8fda-39eb-48e3-be35-2db747d1bb5f/20211024T050036Z.vhd into /xo-vm-backups/2e5c9076-c1ae-4c3b-6f56-940aef6451b0/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/07ac8fda-39eb-48e3-be35-2db747d1bb5f/20211023T050017Z.vhd
Dec 08 00:01:12 xo-server[237842]: 2021-12-08T06:01:12.306Z xo:backups:MixinBackupWriter WARN the VHD /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211126T060036Z.vhd is unused
Dec 08 00:01:12 xo-server[237842]: 2021-12-08T06:01:12.308Z xo:backups:MixinBackupWriter WARN deleting unused VHD /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211126T060036Z.vhd
Dec 08 00:01:17 xo-server[237842]: 2021-12-08T06:01:17.234Z xo:backups:MixinBackupWriter WARN the VHD /xo-vm-backups/821044bd-8e41-009c-abeb-23bc347a2833/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/f21b0b14-924d-4b62-ba23-23792d34a3f3/20211126T060016Z.vhd is unused
Dec 08 00:01:17 xo-server[237842]: 2021-12-08T06:01:17.236Z xo:backups:MixinBackupWriter WARN deleting unused VHD /xo-vm-backups/821044bd-8e41-009c-abeb-23bc347a2833/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/f21b0b14-924d-4b62-ba23-23792d34a3f3/20211126T060016Z.vhd
Dec 08 00:01:18 xo-server[237922]: 2021-12-08T06:01:18.032Z xo:backups:mergeWorker INFO merging /xo-vm-backups/2e5c9076-c1ae-4c3b-6f56-940aef6451b0/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/07ac8fda-39eb-48e3-be35-2db747d1bb5f/20211024T050036Z.vhd: 78/111
Dec 08 00:01:53 xo-server[237842]: 2021-12-08T06:01:53.056Z xo:backups:MixinBackupWriter WARN the VHD /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211126T060054Z.vhd is unused
Dec 08 00:01:53 xo-server[237842]: 2021-12-08T06:01:53.058Z xo:backups:MixinBackupWriter WARN deleting unused VHD /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211126T060054Z.vhd
Dec 08 00:02:14 xo-server[237842]: 2021-12-08T06:02:14.179Z xo:backups:MixinBackupWriter WARN the VHD /xo-vm-backups/83139da5-48d5-33f6-79f9-79ea91a7163c/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/629f9dc6-2dea-4b17-a376-762fa0237ddb/20211126T060112Z.vhd is unused
Dec 08 00:02:14 xo-server[237842]: 2021-12-08T06:02:14.180Z xo:backups:MixinBackupWriter WARN deleting unused VHD /xo-vm-backups/83139da5-48d5-33f6-79f9-79ea91a7163c/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/629f9dc6-2dea-4b17-a376-762fa0237ddb/20211126T060112Z.vhd
Dec 08 00:03:02 xo-server[237922]: 2021-12-08T06:03:02.353Z xo:backups:mergeWorker INFO bye :-)
Dec 08 00:05:24 xo-server[237842]: 2021-12-08T06:05:24.610Z xo:backups:MixinBackupWriter WARN the VHD /xo-vm-backups/6b264102-de29-ef95-3ebf-d3323034c22f/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/eb356f12-12c0-4d90-aa20-eb48d98bfff6/20211126T060028Z.vhd is unused
Dec 08 00:05:24 xo-server[237842]: 2021-12-08T06:05:24.611Z xo:backups:MixinBackupWriter WARN deleting unused VHD /xo-vm-backups/6b264102-de29-ef95-3ebf-d3323034c22f/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/eb356f12-12c0-4d90-aa20-eb48d98bfff6/20211126T060028Z.vhd
Dec 08 00:06:25 xo-server[237842]: 2021-12-08T06:06:25.128Z xo:backups:MixinBackupWriter WARN the VHD /xo-vm-backups/1cd1077d-08ff-3947-2a2e-25b93cf4cc95/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/8fd0a5c9-73d0-4b59-9031-0f6c913651c1/20211126T060230Z.vhd is unused
Dec 08 00:06:25 xo-server[237842]: 2021-12-08T06:06:25.130Z xo:backups:MixinBackupWriter WARN deleting unused VHD /xo-vm-backups/1cd1077d-08ff-3947-2a2e-25b93cf4cc95/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/8fd0a5c9-73d0-4b59-9031-0f6c913651c1/20211126T060230Z.vhd
Dec 08 00:06:30 xo-server[237842]: 2021-12-08T06:06:30.305Z xo:backups:MixinBackupWriter WARN the parent /xo-vm-backups/a176cb04-6720-1f52-6f03-9ccbeb6f6134/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/1dbba180-b219-4916-a631-b69ca84e26dc/20211126T060139Z.vhd of the child /xo-vm-backups/a176cb04-6720-1f52-6f03-9ccbeb6f6134/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/1dbba180-b219-4916-a631-b69ca84e26dc/20211127T061120Z.vhd is unused
Dec 08 00:06:33 xo-server[238029]: 2021-12-08T06:06:33.419Z xo:backups:mergeWorker INFO starting
Dec 08 00:06:33 xo-server[238029]: 2021-12-08T06:06:33.686Z xo:backups:mergeWorker INFO the parent /xo-vm-backups/a176cb04-6720-1f52-6f03-9ccbeb6f6134/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/1dbba180-b219-4916-a631-b69ca84e26dc/20211126T060139Z.vhd of the child /xo-vm-backups/a176cb04-6720-1f52-6f03-9ccbeb6f6134/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/1dbba180-b219-4916-a631-b69ca84e26dc/20211127T061120Z.vhd is unused
Dec 08 00:06:33 xo-server[238029]: 2021-12-08T06:06:33.686Z xo:backups:mergeWorker INFO merging /xo-vm-backups/a176cb04-6720-1f52-6f03-9ccbeb6f6134/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/1dbba180-b219-4916-a631-b69ca84e26dc/20211127T061120Z.vhd into /xo-vm-backups/a176cb04-6720-1f52-6f03-9ccbeb6f6134/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/1dbba180-b219-4916-a631-b69ca84e26dc/20211126T060139Z.vhd
...
Dec 08 00:08:24 xo-server[237842]: 2021-12-08T06:08:24.966Z xo:backups:MixinBackupWriter WARN the parent /xo-vm-backups/ae4366fa-704c-2bcd-fd4a-d2e29fcae51b/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/fc977501-76a5-452b-8538-190fd0e200e9/20211126T060018Z.vhd of the child /xo-vm-backups/ae4366fa-704c-2bcd-fd4a-d2e29fcae51b/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/fc977501-76a5-452b-8538-190fd0e200e9/20211127T060014Z.vhd is unused
...
Dec 08 00:08:53 xo-server[238029]: 2021-12-08T06:08:53.691Z xo:backups:mergeWorker INFO merging /xo-vm-backups/a176cb04-6720-1f52-6f03-9ccbeb6f6134/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/1dbba180-b219-4916-a631-b69ca84e26dc/20211127T061120Z.vhd: 1033/1038
Dec 08 00:08:55 xo-server[238029]: 2021-12-08T06:08:55.424Z xo:backups:mergeWorker INFO the parent /xo-vm-backups/ae4366fa-704c-2bcd-fd4a-d2e29fcae51b/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/fc977501-76a5-452b-8538-190fd0e200e9/20211126T060018Z.vhd of the child /xo-vm-backups/ae4366fa-704c-2bcd-fd4a-d2e29fcae51b/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/fc977501-76a5-452b-8538-190fd0e200e9/20211127T060014Z.vhd is unused
Dec 08 00:08:55 xo-server[238029]: 2021-12-08T06:08:55.425Z xo:backups:mergeWorker INFO merging /xo-vm-backups/ae4366fa-704c-2bcd-fd4a-d2e29fcae51b/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/fc977501-76a5-452b-8538-190fd0e200e9/20211127T060014Z.vhd into /xo-vm-backups/ae4366fa-704c-2bcd-fd4a-d2e29fcae51b/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/fc977501-76a5-452b-8538-190fd0e200e9/20211126T060018Z.vhd
Dec 08 00:09:05 xo-server[238029]: 2021-12-08T06:09:05.425Z xo:backups:mergeWorker INFO merging /xo-vm-backups/ae4366fa-704c-2bcd-fd4a-d2e29fcae51b/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/fc977501-76a5-452b-8538-190fd0e200e9/20211127T060014Z.vhd: 95/3675
...
Dec 08 00:09:49 xo-server[237842]: 2021-12-08T06:09:49.258Z xo:backups:MixinBackupWriter WARN the VHD /xo-vm-backups/6cd7298f-d8e6-3446-5989-cec8e3d0a0ef/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/89f98ead-7220-4639-a1fd-38b4041976eb/20211126T060131Z.vhd is unused
Dec 08 00:09:49 xo-server[237842]: 2021-12-08T06:09:49.260Z xo:backups:MixinBackupWriter WARN deleting unused VHD /xo-vm-backups/6cd7298f-d8e6-3446-5989-cec8e3d0a0ef/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/89f98ead-7220-4639-a1fd-38b4041976eb/20211126T060131Z.vhd
Dec 08 00:09:49 xo-server[237842]: 2021-12-08T06:09:49.260Z xo:backups:MixinBackupWriter WARN the VHD /xo-vm-backups/6cd7298f-d8e6-3446-5989-cec8e3d0a0ef/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/45d0cc4e-5aa7-4e82-9849-9ed6a5d28af3/20211126T060131Z.vhd is unused
Dec 08 00:09:49 xo-server[237842]: 2021-12-08T06:09:49.260Z xo:backups:MixinBackupWriter WARN deleting unused VHD /xo-vm-backups/6cd7298f-d8e6-3446-5989-cec8e3d0a0ef/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/45d0cc4e-5aa7-4e82-9849-9ed6a5d28af3/20211126T060131Z.vhd
Dec 08 00:09:55 xo-server[238029]: 2021-12-08T06:09:55.427Z xo:backups:mergeWorker INFO merging /xo-vm-backups/ae4366fa-704c-2bcd-fd4a-d2e29fcae51b/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/fc977501-76a5-452b-8538-190fd0e200e9/20211127T060014Z.vhd: 631/3675
...
Dec 08 00:13:05 xo-server[238029]: 2021-12-08T06:13:05.430Z xo:backups:mergeWorker INFO merging /xo-vm-backups/ae4366fa-704c-2bcd-fd4a-d2e29fcae51b/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/fc977501-76a5-452b-8538-190fd0e200e9/20211127T060014Z.vhd: 3632/3675
Dec 08 00:14:51 xo-server[238029]: 2021-12-08T06:14:51.188Z xo:backups:mergeWorker INFO bye :-)

Danp

Should I be concerned by these warnings / errors? Recommendations on how to further troubleshoot / resolve?

Edit: XCP-ng 8.2 fully patched and XO from sources (backups were run on commit ba0f7df. Updated to cb2a34c earlier this morning )

julien-f

First, the INFO entries are normal, they indicate the work the merge worker is doing.

The open/EEXIST error may be because more than one task are created at the same time, which appears unlikely but not impossible, we'll handle this.

The rename/ENOENT error is not normal, I don't get how it can happen, it means that the source file right after the parent directory was listed, and I don't understand how that could be. The related code:

let taskFiles
while ((taskFiles = await listRetry()) !== undefined) {
  const taskFileBasename = min(taskFiles)
  const taskFile = join(CLEAN_VM_QUEUE, '_' + taskFileBasename)

  // move this task to the end
  await handler.rename(join(CLEAN_VM_QUEUE, taskFileBasename), taskFile)

The merging warning should not be a real problem, it can occur because for now XO only knows to merge one delta VHD at a time which, in certain cases may not be enough, we are working to handle this

Danp

@julien-f said in Warnings & errors in journalctl:

the INFO entries are normal

I knew this. I just left them in for context.

The rename/ENOENT error is not normal

Could this be caused by the same issue you mentioned above where more than one task is running?

Let me know if there is anything I can do to assist further, such as capturing more details logs.

Thanks for your quick response!

Regards, Dan

Danp

@julien-f What about this one?

Dec 07 00:00:11 xo-server[209172]: 2021-12-07T06:00:11.341Z xo:backups:MixinBackupWriter WARN the parent /xo-vm-backups/821044bd-8e41-009c-abeb-23bc347a2833/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/f21b0b14-924d-4b62-ba23-23792d34a3f3/20211124T060033Z.vhd of the child /xo-vm-backups/821044bd-8e41-009c-abeb-23bc347a2833/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/f21b0b14-924d-4b62-ba23-23792d34a3f3/20211125T060112Z.vhd is unused

julien-f

@danp It means that there is a VHD left to merge, this is normal and we'll try to remove these useless warnings

Danp

@julien-f Here's a new one from today's backup --

Dec 09 00:00:38 xo-server[268840]: 2021-12-09T06:00:38.235Z xo:backups:MixinBackupWriter WARN the parent /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211128T060025Z.vhd of the child /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211129T060035Z.vhd is unused
Dec 09 00:00:40 xo-server[268840]: 2021-12-09T06:00:40.718Z xo:backups:MixinBackupWriter WARN the parent /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211128T060021Z.vhd of the child /xo-vm-backups/12fb9fcb-edd2-a349-c1a3-af0b39a4bee3/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/afdb2817-02ef-4003-b5cd-a54eef270b69/20211129T060015Z.vhd is unused
Dec 09 00:00:41 xo-server[268913]: 2021-12-09T06:00:41.320Z xo:backups:mergeWorker INFO starting
Dec 09 00:00:41 xo-server[268913]: 2021-12-09T06:00:41.463Z xo:backups:mergeWorker INFO the parent /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211128T060025Z.vhd of the child /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211129T060035Z.vhd is unused
Dec 09 00:00:41 xo-server[268913]: 2021-12-09T06:00:41.464Z xo:backups:mergeWorker INFO merging /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211129T060035Z.vhd into /xo-vm-backups/787ada5e-e49d-716f-bdf9-ac7d3fb76195/vdis/dfe75d18-94ac-4bd1-a1c7-fb2170e75154/33e87f85-1afd-4d49-be23-8d88e9ce6bd9/20211128T060025Z.vhd
Dec 09 00:00:41 xo-server[268920]: 2021-12-09T06:00:41.625Z xo:backups:mergeWorker INFO starting
Dec 09 00:00:41 xo-server[268920]: 2021-12-09T06:00:41.654Z xo:backups:mergeWorker FATAL Lock file is already being held {
Dec 09 00:00:41 xo-server[268920]:   error: Error: Lock file is already being held
Dec 09 00:00:41 xo-server[268920]:       at /opt/xen-orchestra/node_modules/proper-lockfile/lib/lockfile.js:68:47
Dec 09 00:00:41 xo-server[268920]:       at callback (/opt/xen-orchestra/node_modules/graceful-fs/polyfills.js:299:20)
Dec 09 00:00:41 xo-server[268920]:       at FSReqCallback.oncomplete (node:fs:199:5)
Dec 09 00:00:41 xo-server[268920]:       at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17) {
Dec 09 00:00:41 xo-server[268920]:     code: 'ELOCKED',
Dec 09 00:00:41 xo-server[268920]:     file: '/run/xo-server/mounts/remote-2/xo-vm-backups/.queue/clean-vm'
Dec 09 00:00:41 xo-server[268920]:   }
Dec 09 00:00:41 xo-server[268920]: }

Danp

@julien-f said in Warnings & errors in journalctl:

The rename/ENOENT error is not normal

Had this occur again today.

olivierlambert

Do you have multiple XOs accessing the same remote?

julien-f

@danp Latest version contains the fix for multiple tasks at the same second (commit), I would love to see if that helps with it.

Danp

@julien-f Don't think so as I was already on commit f3eeeef.

julien-f

@danp

Do you have an XOA from which I could investigate?

Danp

@julien-f Yes, the free version. Check your PM for support channel details.

julien-f

@danp Just to be clear, the ELOCKED error is not a big deal, it just means that another merge worker is handling this remote, I'm going to change this error to warning instead of fatal

julien-f

@Danp Regarding the rename error, unfortunately I don't see anything on your remote and as this appliance is not the one running the backup or the merge, I don't think I'll be able to gather more info.

Danp

@olivierlambert said in Warnings & errors in journalctl:

Do you have multiple XOs accessing the same remote?

There's only one backup job running from my XO VM. There is an active instance of XOA, but it is just used for testing and is sitting idle.