I found a weird thing, I'm trying to find the cause.

@shorian Ah sorry, I think S3 is completely broken at the moment.

@shorian Thank you.

I will look into the memory issue, that seems to be the clue into a hidden issue, thank you.

thanks, I've been inching along with external help, I just managed to compile the package a few minutes ago.
I will try to report here with an exact script to compile the RPM from nothing.

I am extremely sorry, but it seems to works on my machine

BUT, the error I see in your logs also exists at another customer site, and they also trigger it with a publicly available OVA.
If you can and want, would you mind trying to import 'watchguard-dimension_2_2.ova' https://software.watchguard.com/SoftwareDownloads?current=true&familyId=a2RF00000009On4MAE
and check if you get the same error ? you don't have to start the VM, we are just looking for vhd-tool output: vhd-tool: Non-zero size required in xensource.log

We will confer internally. in the mean time.

Nicolas.

yes, I'll have a look

Hi, sorry for the delay, I reproduce the error with chr-6.48.4.ova, I'm now starting to diagnose it.

@shorian thanks. If you could just try no concurrency as a proof that the code work, then your normal concurrency, to check if the issue is fixed that would be great.

This code is meant to be more vulnerable to losing the connection to S3, but I think it's a low risk. It's now using straight HTTP POST.

@shorian I would like to abuse your patience again, by asking you to test this branch: https://github.com/vatesfr/xen-orchestra/tree/nr-s3-fix-big-backups2

The concept is that the backup upload will happen without any sort of smart upload system or queue.

Thank you, Nico

I just had a getaddrinfo EAI_AGAIN error after 3h when trying a big backup towards AWS S3. According to stack overflow, it's generally a timeout on a DNS query. That seems close to your timeout error on the server side. I guess the immediate hypothesis is that the network side of XOA VM is choking on things.

In my last change I added support for keepalive in the AWS library, I will look in this direction.

@ITJamie I'm sorry I can't answer about the node version until we get the bug.

I suspect there is some kind of compounding, where an error happens somewhere, but is lost by improper error handling, and no all we see is a corrupted state long after the error happened.

@ITJamie No, we don't have any hierarchical log system on the front side AFAIK.

@ITJamie oh I see you're building from sources, even better, would you mind being my eyes in Chrome debugger?

when you click on the 'import' button (maybe for a failing VMDK disk, it's the simplest code path that fails) there should be a POST request to a weird URL , if you could tell me the content-length of the request (not the response), and if you see anything untowards in the payload.

here is a capture of a normal thing on my computer:
https://gfycat.com/aggressivedisgustinggopher (there is an HD button at the bottom right)
the payload size should be a few KB and at the bottom there should be the file name. No private information transits here, it's the addresses of the the blocks in the file, not their content.

Thank you for your help.

@ITJamie thanks! it's the first clue ever we get on this problem.

and it is front side, which makes it a bit easier to find.

It's not a slam dunk, because it works here, on a mac with chrome, mut we have a lead.

I am extremely sorry, but it seems to works on my machine

BUT, the error I see in your logs also exists at another customer site, and they also trigger it with a publicly available OVA.
If you can and want, would you mind trying to import 'watchguard-dimension_2_2.ova' https://software.watchguard.com/SoftwareDownloads?current=true&familyId=a2RF00000009On4MAE
and check if you get the same error ? you don't have to start the VM, we are just looking for vhd-tool output: vhd-tool: Non-zero size required in xensource.log

We will confer internally. in the mean time.

Nicolas.

yes, I'll have a look

@sidsanders Hi, I gave a look at your issue, it seems that it is often associated with the ssl certificate missing the middle of the Merkle chain.

here are a couple of explanations:

• https://serverfault.com/questions/693033/unable-to-use-builtin-ca-bundle-to-verify-godaddy-sha2-ssl-certificate
• https://www.npmjs.com/package/ssl-root-cas

I suspect that the certificate you gave to minio was incomplete. Note that the chain of certificate should not include the root certificate, only all the certificates in the chain minus the root.

Nico.

I think the issue is solved, you can follow the fix here: https://github.com/vatesfr/xen-orchestra/pull/5903

nraynaud opened this pull request in vatesfr/xen-orchestra

closed fix(xo-server/{disk,vm}.import): fix import of very small VMDK files #5903

I found a weird thing, I'm trying to find the cause.

Hi, sorry for the delay, I reproduce the error with chr-6.48.4.ova, I'm now starting to diagnose it.