Backblaze B2 as a backup remote

andrewreid

I'm consistently seeing timeout errors at the six minute mark when attempting to use Backblaze B2 as a remote for VM backups. This doesn't happen with smaller backups (i.e., metadata), only the larger backups of VMs.

The relevant error seems to boil down to (full log) :

Error calling AWS.S3.upload: HTTP connection has timed out

I'm not sure if this is a bug per se, or a misconfiguration issue; I don't feel like it's a connectivity issue, as I'm able to make smaller backups without a problem. It's either a Backblaze thing, or there's an implementation problem in XO around the writing of large files.

I've read previous forum posts about the XO server running out of RAM and confirmed it's not – watched it closely during the backup job and it doesn't swap.

Grateful for any pointers!

Cheers,

— andrew

olivierlambert

@florent is our new recruit, revamping entirely the S3 code. We think we might have an entirely new code to test for the end of the month, also orders of magnitude faster (not just working). See https://xen-orchestra.com/blog/xen-orchestra-5-63/#amazons3backup

andrewreid

@olivierlambert Ah! I did read your blog post. That's fantastic. I'm in no screaming hurry to implement S3 backups, but it's great that you've got some promising work in the pipeline!

florent

@andrewreid Yes, I can't wait to share it with everybody.

shorian

@florent Can't wait!

Andrew

I'm having the same timeout problem using wasabi S3. It will be a great feature when it's working. This will make off-site backups very easy to use for everyone.

I also tried making S3 storage (fuse s3fs) as local directory on the XO VM and running it that way. Same problem...

Small backups (a few gigs) work sometimes but larger ones time out.

Error: HTTP connection has timed out

"result": {
            "canceled": false,
            "method": "GET",
            "url": "https://xcp1/export_raw_vdi/?format=vhd&vdi=OpaqueRef%3Aaf1c8105-6fec-4f41-99b3-2828c4220f61&session_id=OpaqueRef%3Abbecbeb8-3c2d-4111-a621-3c36d13261ba&task_id=OpaqueRef%3Af6740a15-f294-4ca2-9dad-2a4b2e5b98ab",
            "timeout": true,
            "message": "HTTP connection has timed out",
            "name": "Error",
            "stack": "Error: HTTP connection has timed out
    at IncomingMessage.emitAbortedError (/opt/xo/xo-builds/xen-orchestra-202111081508/node_modules/http-request-plus/index.js:83:19)
    at Object.onceWrapper (node:events:509:28)
    at IncomingMessage.emit (node:events:390:28)
    at IncomingMessage.patchedEmit (/opt/xo/xo-builds/xen-orchestra-202111081508/@xen-orchestra/log/configure.js:118:17)
    at IncomingMessage.emit (node:domain:475:12)
    at IncomingMessage._destroy (node:_http_incoming:179:10)
    at _destroy (node:internal/streams/destroy:102:25)
    at IncomingMessage.destroy (node:internal/streams/destroy:64:5)
    at TLSSocket.socketCloseListener (node:_http_client:407:11)
    at TLSSocket.emit (node:events:402:35)"
          }

I have no problems doing migrate and copy over slow links (ie. 10Mbps) but backups are not happy even on faster links (ie. 100Mbps).

olivierlambert

Just hold on for this month release