Error: invalid HTTP header in response body
-
Hi @manilx.
thank you for the quick reply. I indeed do have it enabled.
I am just surprised since the exact same job worked previously and it does not make sense to me to keep it. Nevertheless I have to admit, that my knowledge is limited to these types of snapshots and its impact upon storage usage. Just FYI, there is this discussion I found: https://xcp-ng.org/forum/topic/10400/purge-snapshot-data-when-using-cbt-why-wouldn-t-i-enable-this.
However, your point is very good. I just disabled it in order to validate your thesis. If so, I think upon our system behavior we could then report some sort of "bug". Did you backup did not work from the start either or was it also after "some sort of modifications"?
-
@FritzGerald This error happened only on my backups to a remore (a bit slower location but also mounted as nfs like to the local one). It started out of the blue after running perfectly and didn't happen all the time. No rime.
I read about disabling this option in this thread and tried it. Seems to have worked and it is most definitely a bug. -
@manilx okay. I will wait and see. Backup runs tonight.
-
@manilx said in Error: invalid HTTP header in response body:
@peo I don't have this setting set. The errors appear inconsistently.
So you confirmed my "solution" to the problem, even if you did not had that setting enabled when I suggested turning it off.
It's great that more people have this problem and it gets resolved with the same "solution". I was starting to think I imagined the problems week after week before I first reported it.
-
@peo I had the setting turned on and turned it off! Seems to have helped.
-
Hi everyone, just FYI. During my delta backup testing, I ran out of space on my NAS (although it should have had enough?!?). It must have created more data then I expected. Therefore I had other priorities, I removed the backups and set up new delta backups. However, thereby I could not dive into further exploring the problems.
-
Hi everyone again. I pretty much got back to square one. What I can observer is, that all my VMs where I added additional disc run into the above error code. So 1 disc per VM works fine for 6 backups, the two others VMs (on with 2 disc, one with 5 discs) fail. Does CBT based delta backups only work if there is no disc attached? I really appreciate any help.
-
@FritzGerald It has nothing to do with the number of disks attached to the VM. It just fails every second time:
https://xcp-ng.org/forum/post/93508The "solution" (until there is a real solution) is in the reply below the linked one: turn off "Purge snapshot data when using CBT" under advanced settings for all backup jobs.
-
@peo Hi, thank you for your quick reply. Since I had this storage issues after disabling it, I am a little bit careful. My knowledge is really limited about CBT based backups, can you tell me, what it means in terms of storage use. To my understanding it will keep the snapshots and thereby significantly increase space usage, or do I miss something? And have you heard about whether the bug is officially known and worked on?
-
@FritzGerald The snapshots (left on the disk until next backup) will only consist of the differences between the previous backup and the current one.
BUT.. when you do the first backup of a machine, the snapshot will use the full (used) size of each disk attached to the machine (this might be what happened at your first attempt).If you have the space for it, just do one backup at the time with snapshot deletion disabled, then do another one when it's finished. The snapshots will then be reduced to only the difference between the first and second backup.
-
@peo
Hi thank you. Your are most likely right about the backup storage overflow.Just two more questions:
- have you every experienced this problem on a VM with only one disc attached? I am asking since at my site only delta backups fail with additional discs attached.
- Is this somehow addressed as a bug or shall I officially report a bug, since now quite a few users experience this problem.
-
@FritzGerald I have not had any problems like this since I disabled the deletion of the in-between-backup snapshots. I have for example a couple of machines with 50GB+ disks (one with a 100+ GB, mostly unused now, so the snapshot in between the backups takes less than a MB).
All backups were failing (most of my VMs have more than one disk, a trick I use to lock it to a specific host) until I disabled the deletion of the snapshot. Not at once, but more and more of them until all..I still have the other "imaginary problem" with my VM for Docker (but that's a completely other problem which have not yet been acknowledged - backups "fail" but I'm able to restore them to a fully working new VM)
-
@peo
Hi. Sorry for bothering you. I may have expressed in-precise in my question.For reporting a bug we need to get the most specific information possible. That helps developers to pinpoint a bug. Since I have only trouble with VMs having additional virtual discs (disc created on a local SR on the same host as the VM) attached to it, I would like to know if you have a counter example. Or in other words: Do all your VMs only have a single virtual disc?
@manilx feedback would be very interesting as well.
Thank you for your support.
-
@FritzGerald As I replied, most of my VMs have at least two virtual disks (also mixed locations: some on local NVMe, some on local SSD and some on the NAS over NFS).
I would say this is not a reportable issue until you have the backups running again (without deletion of the snapshots). When you activate the snapshot deletion again, the problem will appear on every second run of a backup job (the first is "full" and it succeeds, the second attempt will be "delta" and it fails, the third attempt will again be "full" and will succeed). This is (was at least for me) independent of the number of attached disks per VM.
-
@peo
I am sorry, I must have missed your statement about the discs.Okay. We can both confirm:
- CBT based delta backups fail on some VMs, causing an iterating success / failure backup behavior.
- removing "Purge snapshot data when using CBT" flag works around it, but at the costs of additional "doubling" storage space.
- In addition only I observed, that it always and only happens to me, when there is are additional discs involves. Delta backup of VMs within the same backup job but different machines work at my place.
I will observe this during this week and then file a bug report.
-
Just jumping in here,
Running XOCE. Updated on the 21st June, never had backup issues but now all jobs fail as above. Don't have (and never have had) "Purge snapshot data when using CBT" enabled.
Rolled back to the update i did on the 4th June and all backups working as expected.
Thanks god for a decent rollback plan.
For those running XO from sources, you can "upgrade" again and select an older version. May get you out of the woods.
-
Things were fixed in
masterrecently, so please try again
-
@olivierlambert I just updated to ('a348ce07d') and the problem remains:
(this backup job was set up just as a test yesterday, only local VDI in the VM and storing backups to NAS over NFS)
As before, it happens on every second manual run of the job (when the last line of the error says "Type: Delta")
Also, a slight UI problem, but that's not there (or at least was not yesterday) in the real version.
-
@olivierlambert Also the same in XOA, switched "latest" in the "release channel" before updating:
Initial backup succeeds, the second one fails, the third one succeeds (but is transferred as "full" again, as the first one)
-
Yes but I was talking about the sources.
latestisn'tmaster.
