Potential bug with Windows VM backup: "Body Timeout Error"

ph7

This post is deleted!

ph7

This post is deleted!

ph7

This post is deleted!

MajorP93

@ph7 As a suggestion: it would improve readibility if you paste your logs on Vates official pastebin tool: https://paste.vates.tech/ and share the links here instead of posting the logs in multiple messages.

ph7

@MajorP93
I'l do that, didn't know about it.

poddingue

Thirty to forty seconds of clock drift after a power outage, and then the backup goes through once you fix the time. That could be coincidence, but it might not be.
I don't know enough about how the XAPI client handles time skew to say whether it would surface as a BodyTimeoutError rather than an auth or TLS failure, so I could easily be wrong here.
If it fails again, could you look at what the host clock is doing before you correct it? @pierrebrunet from the XO team is already on this one, and a second data point either way would tell him whether it's worth pulling on. Thanks for putting the logs on the pastebin, much easier to read that way.

CodeMercenary

I checked my hosts and the clocks all seem to be very close in time, certainly within a second of each other. The one difference is that the host that is often the target for the backups is set to UTC and the other two are set to MST. Clearly an oversight on my part, unless it is purely a display issue. When I ssh into dom0 and use date, those two show MST and the other shows UTC. If that has ever changed, it would most likely have been in the upgrade from 8.2 to 8.3, which I performed on Jan 28.

All that said, this body timeout issue happens to me every day on random VMs within my backup. Occasionally a backup will complete with none of these errors on any VM but that's uncommon. All of the failed backups are from VMs or configs on the two MST hosts and are stored on the UTC host. (To be clear, they are also stored elsewhere.) I also have an XO instance on one of the MST hosts that performs delta backups and those have never failed. I don't know if that's because they are delta or if it's because they don't touch the UTC host.

The recent failed backups started on 6/26 for my VM full backups and on 6/29 for the config backups. Prior to that the config backups had never had a problem and the full backups had been fine back into February. From February 21 to March 12, I had a smattering of backup failures due to body timeout error for the full backups.

poddingue

Thanks for checking. That's useful, even if it points away from where I was looking.
Clocks within a second of each other means drift probably isn't your problem, and I'd guess the MST/UTC difference is just how dom0 displays it, though I'm not sure.
What I keep coming back to is that your full backups fail while the delta jobs on the same hosts never do. That's the same split in https://github.com/vatesfr/xen-orchestra/issues/9181, where full backups hit BodyTimeoutError on VMs with big disks or a lot of free space and the deltas are fine.
If your failing VMs look like that, your dates and the MST/UTC detail would do more good on that issue than buried in here.

MajorP93 created this issue in vatesfr/xen-orchestra

open Full backup fails on large VMs / VMs with lots of free disk space #9181

Greg_E

@poddingue

My three hosts are all EDT, drift on mine is pretty small because I have a local NTP server with GNSS.

So it could be a timezone issue and might be worth looking into.

poddingue

Thanks Greg, that's a useful data point.
If your clocks are within a second across all three hosts and you're still seeing it, that makes me doubt the timezone angle as the root cause, even if the way dom0 displays the time is confusing.
The thing I keep coming back to is the split you and I both see: full backups fail while the delta jobs on the same VMs never do. That's the same pattern in https://github.com/vatesfr/xen-orchestra/issues/9181, which points at large VMs or VMs with a lot of free disk space rather than anything clock-related.
I'm not sure that's your case, but it might be worth checking whether the VMs that fail are the ones carrying the most free space inside the guest.

MajorP93 created this issue in vatesfr/xen-orchestra