Backup fails with "Body Timeout Error", "all targets have failed, step: writer.run()"

JB

@marcoi So the problem is with my infrastructure.

JB

@marcoi I see that you use NFS to store the backup. I use SMB.

JB

This post is deleted!

JB

marcoi

i switched one of the backups for my extra pool from nfs to smb and its been sitting so i expect it to fail. might be a smb issue. ill update this post with results once it either fails or works

i did test smb and it works.

back to nfs works

MajorP93

@olivierlambert said:

Hi! Can you try to replicate on the latest commit?

Hello!
Yes I was able to replicate this issue after updating my Xen Orchestra.

The thing is that the "XO-config-and-pool-metadata" does not always fail.
It fails from time to time then it looks exactly like shown in the screenshot.
Attached logs please find above in my earlier post.

This exact job has been working for a long time before.
After some Xen Orchestra commit this issue got introduced.

I use the same backup remotes for multiple backup jobs.
Only the "XO-config-and-pool-metadata" job is failing with this body timeout error.

Best regards

pierrebrunet

@MajorP93 Hi,
Thank you, thank you! I think you found it! It seems to be SMB, @acebmxer can you confirm that you also use smb please?
If this is the case, the bad news is I think we will need to update our docs to not advise SMB at all: it was already said for delta but maybe we will have to extend this to all kind of BR

MajorP93

@pierrebrunet Hello, thank you for working on this issue.

Are you completely sure that this isn't something that got introduced within Xen Orchestra code?

Since I started using XCP-ng / Xen Orchestra I have been using these backup repositories via SMB.
All backup jobs (including this one) have been working fine for almost a year.

This repeated body timeout error is a rather new issue which is why I was wondering if this could have been introduced by XO code changes?

I did not switch to a different linux kernel major version or similar. So the SMB client kernel module is at the same version number.

Best regards

acebmxer

@pierrebrunet

I use nfs for my remotes and still get the error occasionally.

Edit - Note i am only seeing this error in my work production environment. I am not seeing this issue in my home lab. Both use NFS for remote storage and vm storage.

MajorP93

@pierrebrunet Oh one more thing that I just remembered.
If I recall correctly the first time I saw this issue was right after I patched my XCP-ng hosts without updating Xen Orchestra.
So maybe this is caused by XAPI code changes?

JB

commit 20bed

SMB

pierrebrunet

@acebmxer ok, too bad... I thought we had it...
@majorp93 this can be a lead. I will ask xapi team if they changed something in pool metadata export.

acebmxer

@pierrebrunet

No worries,
Mine seems to be intermittent and restarting that pools tool stack resolves the issue for now.

JB

@acebmxer

Exactly! I restarted the pools tool stack and made 3 backups. If I restart it again, I'll be able to make more backups.

GregBinSD

@jb
You have hit on something that ties into what I have discovered. Since July 1, on my XCP-ng / XO-CE cluster, the metadata and xo-config backup job only runs successfully when I have all 3 hosts in my cluster active, and after I have reset the tool stack on the Master host.

Because I am running the XCP-ng and XO-CE on an off-grid solar/battery system, independent of my electric bill, I try to limit total power to 250 watts for all devices (network switches, iSCSI NAS, XCP-ng hosts, etc). Consequently, I run only one host on a regular basis. When it is necessary to update XCP-ng, I turn on all hosts and do a rolling pool update. If I manually perform a metadata and xo-config backup while all of the hosts are on, only then will the backup be successful.

Before July 1, the metadata and xo-config back was always successful, whether it was scheduled or manual.

Hopefully, this will turn on a light bulb in the head of a Vates engineer, and he or she will solve this issue!

olivierlambert

Are you sure it really worked? Because we fixed something that swallowed the error before. Maybe it was successful while in fact not working. When you have some hosts down in a pool, you cannot safely export metadata because XAPI can't tell if the other hosts are dead, disconnected or simply stopped.

acebmxer

@JB @gregbinsd @olivierlambert

Yes restarting the tool stack works for me. I currently do not have any host offline at least not intentionally. If there is a network glitch that i am not aware of that might cause the issue if that is the actual cause.

As @olivierlambert stated maybe we were having issues all along and they just changed something that is making the issue visible now. The support tunnel should still be open on my environment if you or the team want to look at logs.

GregBinSD

@jb
@olivierlambert
Oliver, you may be right, wondering whether they really worked. How can I prove that the metadata and xo-config backups were truly successful?

Below, I pasted in the GUI backup log from XO5.

JB

All my hosts are online, yet the error still occurs. However, if I restart, the backup works for two or three runs.

bogikornel

The problem is that my VM backups don't run either, because I still get a timeout. This morning out of 11 VM backups, 2 failed, on Tuesday 1 failed, 20 were good.