Keep getting "HTTP connection has timed out" on my Delta Backup

ToasterUwU

Some info before I explain in detail:

The external Server is a Synology Diskstation which I use with SMB
The Xen Orchestra version I have installed is from sources with the newest commit
I gave the Xen Orchestra VM 4GB of RAM and 30GB of Storage
I have had this issue for many months and tried many things (switching to NFS, manually mounting and using mount as local in Xen Orchestra, reinstalling Xen Orchestra completely and again NFS and manual mount)

Above you can see what I mean. I keep getting this error on like half of the VMs when 6 AM arrives and the backup runs. It's not the same VMs every time, it is not the same amount every time, retrying the failed backups sometimes work, but sometimes it doesn't.
My current solution is to retry the failed ones over and over until all of them worked. But that sometimes takes 5 tries or more.

Does anyone have an idea why this could happen? And what exactly does that mean? Is that meaning that the HTTP connection to the Xen API on my Host timed out or something else?

If there is anything i can do to give more info, let me know.

julien-f

Hi, you can try this work-around, just create this file /etc/xo-server/config.httpInactivityTimeout.toml:

# Work-around HTTP timeout issue during backups
[xapiOptions]
httpInactivityTimeout = 1800000 # 30 mins

olivierlambert

Check the detailed log for the error. It might be due to a flaky connection between some elements in the chain.

ToasterUwU

@olivierlambert Here is what the error log gives as reason for the error:

Error: HTTP connection has timed out\n at IncomingMessage.emitAbortedError (/opt/xo/xo-builds/xen-orchestra-202204122029/node_modules/http-request-plus/index.js:83:19)\n at Object.onceWrapper (events.js:519:28)\n at IncomingMessage.emit (events.js:400:28)\n at IncomingMessage.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202204122029/@xen-orchestra/log/configure.js:120:17)\n at TLSSocket.socketCloseListener (_http_client.js:432:11)\n at TLSSocket.emit (events.js:412:35)\n at TLSSocket.patchedEmit [as emit] (/opt/xo/xo-builds/xen-orchestra-202204122029/@xen-orchestra/log/configure.js:120:17)\n at net.js:686:12\n at TCP.done (_tls_wrap.js:564:7)\n at TCP.callbackTrampoline (internal/async_hooks.js:130:17)

I have no real clue what that means in human language, but that's all the log can offer.

The connection between the Host and the NAS is a switch with LAN. The connection is not unstable, its not slow or anything. I use the same switch and same cables to move files to my PC and I never got any kind of issue.

Forza

@olivierlambert

This looks like an error we had. The support team ( @julien-f ) fixed a timeout setting on the XOA instance and it has been working since then.

ToasterUwU

@Forza Is there a way I could set the timeout to something higher myself for my installation? Or anything I can do in general to don't have that issue anymore?

Forza

@ToasterUwU said in Keep getting "HTTP connection has timed out" on my Delta Backup:

@Forza Is there a way I could set the timeout to something higher myself for my installation? Or anything I can do in general to don't have that issue anymore?

I do not know how it was done. We have a support contract so the support team helped through the support tunnel.

ToasterUwU

@Forza Damn. Being a broke student with a hobby and interest in IT things sucks.

Thanks anyways, lets hope @olivierlambert knows what to do.

olivierlambert

It means that one end isn't answering fast enough probably due to connection timeout (as I said, a flaky connection for example).

I don't remember the config, I think @julien-f did make a change. Let's see if he's around

julien-f

Hi, you can try this work-around, just create this file /etc/xo-server/config.httpInactivityTimeout.toml:

# Work-around HTTP timeout issue during backups
[xapiOptions]
httpInactivityTimeout = 1800000 # 30 mins

ToasterUwU

@julien-f does it have to be /etc/xo-server/ or do you just mean the xo-server folder where all the files for XO are?

olivierlambert

It can be in the xo-server folder

ToasterUwU

@olivierlambert perfect, I tried it and it works! Thank you very much. Finally I'm free of this issue

vbits

@ToasterUwU Hi,

where to create this file?
I have XOCE self compiled and I have several routes with xo-server:

/opt/xo/xo-builds/xen-orchestra-202208011909/packages/xo-server/
/opt/xo/xo-src/xen-orchestra/packages/xo-server/
/var/lib/xo-server
/opt/xo/xo-server (I think here "systemd")

I have 2 very large VMs of 400GB that whenever I try to configure a backup I get "time out" and during the time it tries there is a lot of instability in the network / RAC.
Most of the VMs and physical servers lose connectivity.
With NFS I never could
With SMB it works with truenas being in another RAC but the network problems always appear.
This time I set up a SMB on synology in the same RAC thinking that the problem would be due to the amount of transfer between RACs but even being in the same RAC gives me the same error.

olivierlambert

How did you install it? If you did yourself, then go where you cloned the repo If you used a 3rd party script, ask the 3rd party provider

ronivay

@vbits looks like you used XenOrchestraInstallerUpdater so i can just reply here.

If you used default user in the scripts config, you can put this timeout override config to /root/.config/xo-server directory and restart xo-server service. If you used other user than root, then same thing but /root -> /users/homedirectory

vbits

@ronivay thank you
I kept having network problems... many servers were losing connectivity.
After realizing that the backup traffic goes through xoce first, I found the problem as the VMs (RAC1) were going through xoce (RAC2) and then back to RAC1 (Synology).
I created an XOCE clone VM on RAC1 and the backups now run without "HTTP connection has timed out" and no network flapping.

olivierlambert

Hi,

Thanks for the feedback. So basically your network couldn't handle the traffic?

vbits

@olivierlambert Yes, it seems so. The switches are of different brands between RACs and since a year ago the datacenter was divided in 2 companies and we are not sure how they have organized the networks.
Then I had another error but with "rescan all disks" on the host it was solved.