Backup mergeWorker stuck

hannes

Hi everyone,

we are testing an xcp-ng setup to hopefully replace a vmware system soon. We tested with delta backups and an actual set of vms with about 3-5 TB storage.

We noticed that backups get stuck on one vm every once in a while. The situation resolves most of the times as soon as we restart the xoa vm. Being stuck, new snapshots for backups don't even start. The status is only "orange" in xoa.

The error log of the xoa shows the following:

Mar  3 09:26:50 ktmr-vm-xoa xo-server[20439]: 2023-03-03T08:26:50.509Z xo:backups:mergeWorker INFO merge in progress {
Mar  3 09:26:50 ktmr-vm-xoa xo-server[20439]:   done: 68,
Mar  3 09:26:50 ktmr-vm-xoa xo-server[20439]:   parent: 'xo-vm-backups/22676691-e76d-58fa-e73d-0cc9ec0d7150/vdis/4c1f1db8-b70f-4e1c-aca7-74b457edacb2/0793f530-5
2f5-4d6f-8bb8-5710117e8eb3/20230227T180042Z.vhd',
Mar  3 09:26:50 ktmr-vm-xoa xo-server[20439]:   progress: 0,
Mar  3 09:26:50 ktmr-vm-xoa xo-server[20439]:   total: 21554
Mar  3 09:26:50 ktmr-vm-xoa xo-server[20439]: }

The messages has been repeating itself for about 12 hours now. The progress remains 0 and the done at 68.

Did anyone notice anything similar, yet? Any suggestions how to resolve this?

Thanks,
Hannes

olivierlambert

Hi!

In which release channel of XOA are you?

hannes

@olivierlambert We have been using the master channel. We pulled two files about two weeks ago.

olivierlambert

So you are not using XOA but XO from the sources, right?

hannes

@olivierlambert Yes - we have compiled it from sources for the lab tests.

olivierlambert

In general, it's easier to start on XOA (you can have a free trial with all features) to discover the ecosystem. Because XOA is more tested and also comes with an updater AND a way to give us access remotely to investigate a problem. So please deploy an XOA and report if it's better
On which commit are you right now? Are you on the right version of node? (that's the questions you wouldn't have to answer if you were on XOA for example )