Potential bug with Windows VM backup: "Body Timeout Error"

poddingue

I'm picking this one up alongside the other Body Timeout thread (9002), since they look (to me, at least) like the same underlying issue.
Pierre from the XO team is on it and has tested metadata backups on the XO side without hitting it, so the current thinking is it may be coming from the XCP-ng side rather than XO.
What would move it along fastest is a /var/log/xensource.log from the pool master covering one failed run's window, so if anyone here can reproduce and grab that slice, it'd really help line the timeout up against what xapi was doing.
I know it's especially frustrating when the thing that breaks is your backups, thanks for bearing with us while we chase it.
A mention to @Team-Storage might help route the storage-layer angle.

ph7

Had a failed job this night (commit 3bc70)

Logged in to my host and saw time was 30-40 sec's off.
The time in XOCE VM was OK
Had a few power outage yesterday.
after starting up the host again. I guess the cmos clock was a little slow.
Could that be the culprit?
Adjusted time and ran a backup manually and it worked fine.

As requested below here are the logs
https://paste.vates.tech/?314d454cd7c52552#4ocXwQksjayBbvUxxxddJRwAQ8YJ3KqvKspJGak8yHF4

ph7

This post is deleted!

ph7

This post is deleted!

ph7

This post is deleted!

ph7

This post is deleted!

ph7

This post is deleted!

MajorP93

@ph7 As a suggestion: it would improve readibility if you paste your logs on Vates official pastebin tool: https://paste.vates.tech/ and share the links here instead of posting the logs in multiple messages.

ph7

@MajorP93
I'l do that, didn't know about it.

poddingue

Thirty to forty seconds of clock drift after a power outage, and then the backup goes through once you fix the time. That could be coincidence, but it might not be.
I don't know enough about how the XAPI client handles time skew to say whether it would surface as a BodyTimeoutError rather than an auth or TLS failure, so I could easily be wrong here.
If it fails again, could you look at what the host clock is doing before you correct it? @pierrebrunet from the XO team is already on this one, and a second data point either way would tell him whether it's worth pulling on. Thanks for putting the logs on the pastebin, much easier to read that way.