Offline
@ITJamie I'm sorry I can't answer about the node version until we get the bug.
I suspect there is some kind of compounding, where an error happens somewhere, but is lost by improper error handling, and no all we see is a corrupted state long after the error happened.
@ITJamie No, we don't have any hierarchical log system on the front side AFAIK.
@ITJamie oh I see you're building from sources, even better, would you mind being my eyes in Chrome debugger?
when you click on the 'import' button (maybe for a failing VMDK disk, it's the simplest code path that fails) there should be a POST request to a weird URL , if you could tell me the content-length of the request (not the response), and if you see anything untowards in the payload.
here is a capture of a normal thing on my computer:
https://gfycat.com/aggressivedisgustinggopher (there is an HD button at the bottom right)
the payload size should be a few KB and at the bottom there should be the file name. No private information transits here, it's the addresses of the the blocks in the file, not their content.
Thank you for your help.
@ITJamie thanks! it's the first clue ever we get on this problem.
and it is front side, which makes it a bit easier to find.
It's not a slam dunk, because it works here, on a mac with chrome, mut we have a lead.
I am extremely sorry, but it seems to works on my machine
BUT, the error I see in your logs also exists at another customer site, and they also trigger it with a publicly available OVA.
If you can and want, would you mind trying to import 'watchguard-dimension_2_2.ova' https://software.watchguard.com/SoftwareDownloads?current=true&familyId=a2RF00000009On4MAE
and check if you get the same error ? you don't have to start the VM, we are just looking for vhd-tool output: vhd-tool: Non-zero size required in xensource.log
We will confer internally. in the mean time.
Nicolas.
yes, I'll have a look
@sidsanders Hi, I gave a look at your issue, it seems that it is often associated with the ssl certificate missing the middle of the Merkle chain.
here are a couple of explanations:
I suspect that the certificate you gave to minio was incomplete. Note that the chain of certificate should not include the root certificate, only all the certificates in the chain minus the root.
Nico.
I think the issue is solved, you can follow the fix here: https://github.com/vatesfr/xen-orchestra/pull/5903
I found a weird thing, I'm trying to find the cause.
Hi, sorry for the delay, I reproduce the error with chr-6.48.4.ova, I'm now starting to diagnose it.
@jensolsson-se Can you use SMB, NFS or local backups? I don't think S3 has ever worked for anyone.
@jensolsson-se yes, we have though of complicated solutions too, but we haven't yet really dug into it, because this is a backup situation, we'd like the state of things and failure modes to be manageable.
@jensolsson-se Hi Jens, you are on the right track. The other thing is that 5TB/10000 leads to big fragment size that are a bit much to keep in memory.
@jensolsson-se that is weird, there is no real error in the logs you posted, and the testing function actually writes on the S3 bucket, so I really think a lot of things went right.
I am trying to think of a bug that would have gone around the error reporting system. The websocket message is a bit suspicious.
@jensolsson-se hello, sorry the notification slipped through the cracks.
the current fashion in S3 bugs at the moment is in the backups, and the stacktrace clearly shows our code:
"stack": "Error: Error calling AWS.S3.upload: write ECONNRESET\n
at rethrow (/home/xoa/xen-orchestra/node_modules/@sullux/aws-sdk/index.js:254:24)\n
at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n
at async S3Handler._outputStream (/home/xoa/xen-orchestra/@xen-orchestra/fs/dist/s3.js:100:5)\n
at async S3Handler.outputStream (/home/xoa/xen-orchestra/@xen-orchestra/fs/dist/abstract.js:250:5)\n
at async RemoteAdapter.outputStream (/home/xoa/xen-orchestra/@xen-orchestra/backups/RemoteAdapter.js:509:5)\n
at async /home/xoa/xen-orchestra/@xen-orchestra/backups/writers/FullBackupWriter.js:69:7"
I am extremely puzzled by the stacktrace you posted, the "websocket" keyword almost suggest that the server tried to answer to the browser, but the TCP socket had been closed.
what do you have in the backup log ?
This circled button copies a very long detail of the error to the clipboard, you can paste it in a text editor and have a look.
@olivierlambert no I have never encountered this error, but the "websocket" keyword make me suspect it's a UI error between XO and the browser,
Hi @Meth0d , I had a look at your file, I see that it's VMFS ( https://en.wikipedia.org/wiki/VMware_VMFS ) inside, which seems to be a real file system and not a disk image. I'm unfamiliar with esxi, but I think you might not have exported the virtual machine the way other users do.
@shorian Ah sorry, I think S3 is completely broken at the moment.
@shorian thanks. If you could just try no concurrency as a proof that the code work, then your normal concurrency, to check if the issue is fixed that would be great.
This code is meant to be more vulnerable to losing the connection to S3, but I think it's a low risk. It's now using straight HTTP POST.
@shorian I would like to abuse your patience again, by asking you to test this branch: https://github.com/vatesfr/xen-orchestra/tree/nr-s3-fix-big-backups2
The concept is that the backup upload will happen without any sort of smart upload system or queue.
Thank you, Nico