Keep getting "HTTP connection has timed out" on my Delta Backup

Forza

This looks like an error we had. The support team ( @julien-f ) fixed a timeout setting on the XOA instance and it has been working since then.

ToasterUwU

@Forza Is there a way I could set the timeout to something higher myself for my installation? Or anything I can do in general to don't have that issue anymore?

Forza

@ToasterUwU said in Keep getting "HTTP connection has timed out" on my Delta Backup:

@Forza Is there a way I could set the timeout to something higher myself for my installation? Or anything I can do in general to don't have that issue anymore?

I do not know how it was done. We have a support contract so the support team helped through the support tunnel.

ToasterUwU

@Forza Damn. Being a broke student with a hobby and interest in IT things sucks.

Thanks anyways, lets hope @olivierlambert knows what to do.

olivierlambert

It means that one end isn't answering fast enough probably due to connection timeout (as I said, a flaky connection for example).

I don't remember the config, I think @julien-f did make a change. Let's see if he's around

julien-f

Hi, you can try this work-around, just create this file /etc/xo-server/config.httpInactivityTimeout.toml:

# Work-around HTTP timeout issue during backups
[xapiOptions]
httpInactivityTimeout = 1800000 # 30 mins

ToasterUwU

@julien-f does it have to be /etc/xo-server/ or do you just mean the xo-server folder where all the files for XO are?

olivierlambert

It can be in the xo-server folder

ToasterUwU

@olivierlambert perfect, I tried it and it works! Thank you very much. Finally I'm free of this issue

vbits

@ToasterUwU Hi,

where to create this file?
I have XOCE self compiled and I have several routes with xo-server:

/opt/xo/xo-builds/xen-orchestra-202208011909/packages/xo-server/
/opt/xo/xo-src/xen-orchestra/packages/xo-server/
/var/lib/xo-server
/opt/xo/xo-server (I think here "systemd")

I have 2 very large VMs of 400GB that whenever I try to configure a backup I get "time out" and during the time it tries there is a lot of instability in the network / RAC.
Most of the VMs and physical servers lose connectivity.
With NFS I never could
With SMB it works with truenas being in another RAC but the network problems always appear.
This time I set up a SMB on synology in the same RAC thinking that the problem would be due to the amount of transfer between RACs but even being in the same RAC gives me the same error.

olivierlambert

How did you install it? If you did yourself, then go where you cloned the repo If you used a 3rd party script, ask the 3rd party provider

ronivay

@vbits looks like you used XenOrchestraInstallerUpdater so i can just reply here.

If you used default user in the scripts config, you can put this timeout override config to /root/.config/xo-server directory and restart xo-server service. If you used other user than root, then same thing but /root -> /users/homedirectory

vbits

@ronivay thank you
I kept having network problems... many servers were losing connectivity.
After realizing that the backup traffic goes through xoce first, I found the problem as the VMs (RAC1) were going through xoce (RAC2) and then back to RAC1 (Synology).
I created an XOCE clone VM on RAC1 and the backups now run without "HTTP connection has timed out" and no network flapping.

olivierlambert

Hi,

Thanks for the feedback. So basically your network couldn't handle the traffic?

vbits

@olivierlambert Yes, it seems so. The switches are of different brands between RACs and since a year ago the datacenter was divided in 2 companies and we are not sure how they have organized the networks.
Then I had another error but with "rescan all disks" on the host it was solved.