XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    ova import failing silently. logs show timeout against xcp-ng node

    Scheduled Pinned Locked Moved Xen Orchestra
    featurein backlog
    33 Posts 5 Posters 6.8k Views 3 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • ITJamieI Offline
      ITJamie
      last edited by marcungeschikts

      XOA / XCP import failing
      XOA built from source, master branch. ref: bf194f5c8503e4a09e8f6d84c0619d669e7dd172
      Node 17.5.0
      Yarn 1.22.17
      Debian 11

      xcp-ng: 8.2.0. fully patched through XOA and rebooted

      I confirmed that the xcp-ng node URL below is reachable from XOA (self signed cert, but it is working)
      I saw another forum post about a similar issue https://xcp-ng.org/forum/topic/2853/import-ova-silently-fails. ive added 10gb of ram to the XOA server but it still made no difference.
      the xcp-ng server has 8gb of memory assigned to the control domain.

      Any ideas?

      xoa logs:

      2022-02-15T16:03:58.130Z xo:main INFO + WebSocket connection (::ffff:<IP REMOVED>)
      2022-02-15T16:09:11.247Z xo:xo-server ERROR unhandled error event {
        error: Error: HTTP connection has timed out
            at IncomingMessage.emitAbortedError (/opt/xen-orchestra/node_modules/http-request-plus/index.js:83:19)
            at Object.onceWrapper (node:events:645:28)
            at IncomingMessage.emit (node:events:526:28)
            at IncomingMessage.emit (node:domain:475:12)
            at IncomingMessage.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:118:17)
            at IncomingMessage._destroy (node:_http_incoming:179:10)
            at _destroy (node:internal/streams/destroy:109:25)
            at IncomingMessage.destroy (node:internal/streams/destroy:71:5)
            at TLSSocket.socketCloseListener (node:_http_client:402:11)
            at TLSSocket.emit (node:events:538:35)
            at TLSSocket.emit (node:domain:475:12)
            at TLSSocket.patchedEmit [as emit] (/opt/xen-orchestra/@xen-orchestra/log/configure.js:118:17)
            at node:net:687:12
            at TCP.done (node:_tls_wrap:580:7)
            at TCP.callbackTrampoline (node:internal/async_hooks:130:17) {
          canceled: false,
          method: 'PUT',
          url: 'https://<IP REMOVED>/import_raw_vdi/?format=vhd&vdi=OpaqueRef%3Ac3f705d6-773d-4b23-a00e-30948cb0df34&session_id=OpaqueRef%3A035a6aea-bfe7-48a2-9e4f-40c33e6adf65&task_id=OpaqueRef%3A2fd4af46-8ced-46c4-9f68-d63efe15c5f0',
          timeout: true
        }
      }
      

      xcp-ng logs (all logs from start to 5 minute timeout): https://pastebin.pl/view/007f62a3

      1 Reply Last reply Reply Quote 0
      • olivierlambertO Offline
        olivierlambert Vates πŸͺ Co-Founder CEO
        last edited by

        Anything in the browser log?

        ITJamieI 1 Reply Last reply Reply Quote 0
        • ITJamieI Offline
          ITJamie @olivierlambert
          last edited by

          @olivierlambert nothing shows up in the console log in dev tools in chrome.
          i tried it from another pc on our network. same thing.

          i got the cli upload command to work from a linux vm. so at least i was able to get unblocked. but would be nice to be able to use the UI for this

          1 Reply Last reply Reply Quote 0
          • olivierlambertO Offline
            olivierlambert Vates πŸͺ Co-Founder CEO
            last edited by olivierlambert

            Well, OVA import is normally working, but there's always "some" OVAs that are different, and those might break.

            It's hard to guess why until we got access to the OVA itself.

            ITJamieI 1 Reply Last reply Reply Quote 0
            • ITJamieI Offline
              ITJamie @olivierlambert
              last edited by

              @olivierlambert
              heres the exact one im importing
              https://cloud-images.ubuntu.com/focal/20220215/focal-server-cloudimg-amd64.ova

              1 Reply Last reply Reply Quote 0
              • olivierlambertO Offline
                olivierlambert Vates πŸͺ Co-Founder CEO
                last edited by

                Perfect, this will help us to test it on our side πŸ™‚ We'll keep you posted!

                1 Reply Last reply Reply Quote 0
                • olivierlambertO Offline
                  olivierlambert Vates πŸͺ Co-Founder CEO
                  last edited by

                  @nraynaud any chance you can take a look?

                  1 Reply Last reply Reply Quote 0
                  • nraynaudN Offline
                    nraynaud XCP-ng Team
                    last edited by

                    yes, I'll have a look

                    1 Reply Last reply Reply Quote 1
                    • olivierlambertO Offline
                      olivierlambert Vates πŸͺ Co-Founder CEO
                      last edited by

                      Thanks!

                      1 Reply Last reply Reply Quote 0
                      • nraynaudN Offline
                        nraynaud XCP-ng Team
                        last edited by

                        I am extremely sorry, but it seems to works on my machine

                        Capture d’écran 2022-02-17 aΜ€ 10.00.28.png

                        BUT, the error I see in your logs also exists at another customer site, and they also trigger it with a publicly available OVA.
                        If you can and want, would you mind trying to import 'watchguard-dimension_2_2.ova' https://software.watchguard.com/SoftwareDownloads?current=true&familyId=a2RF00000009On4MAE
                        and check if you get the same error ? you don't have to start the VM, we are just looking for vhd-tool output: vhd-tool: Non-zero size required in xensource.log

                        We will confer internally. in the mean time.

                        Nicolas.

                        ITJamieI 1 Reply Last reply Reply Quote 1
                        • ITJamieI Offline
                          ITJamie @nraynaud
                          last edited by

                          @nraynaud said in ova import failing silently. logs show timeout against xcp-ng node:

                          Non-zero size required

                          confirmed i do get the same error. I have not managed to successfully upload any ova or vmdk through the gui

                          for additional context, i am running this on Version 98.0.4758.80 (Official Build) (arm64) chrome on an m1x mac.
                          ive also tried it through latest chrome on a windows machine

                          I saw very little network throughput from the point i pressed "upload" to the point the error appeared in logs. I have a feeling the vmdk is just not being converted in the browser. Can we enable additional logging in the vmdk to vhd code?

                          1 Reply Last reply Reply Quote 0
                          • olivierlambertO Offline
                            olivierlambert Vates πŸͺ Co-Founder CEO
                            last edited by

                            Could you also check with Firefox, just to be sure?

                            Also, what kind of storage (SR) is the target on your XCP-ng? (local ext, LVM, iSCSI?)

                            ITJamieI 2 Replies Last reply Reply Quote 0
                            • ITJamieI Offline
                              ITJamie @olivierlambert
                              last edited by

                              @olivierlambert local ext and nfs. ive tried both. ive also tried it against different xcp-ng hosts,all installed from the same 8.2.0 iso

                              tried with firefox. same deal. i noticed in osx's activity monitor it only sent 10MB total

                              ITJamieI 1 Reply Last reply Reply Quote 0
                              • ITJamieI Offline
                                ITJamie @ITJamie
                                last edited by

                                @ITJamie Screenshot 2022-02-17 at 11.57.00.png

                                after messing around re-attempting uploads. i managed to get this to appear in debugger console logs...

                                1 Reply Last reply Reply Quote 1
                                • ITJamieI Offline
                                  ITJamie @olivierlambert
                                  last edited by

                                  @olivierlambert
                                  firefox (latest after i upgraded it) on windows just worked!
                                  managed to upload the ubuntu image and the watchguard image!

                                  so nether firefox nor chrome worked on mac.
                                  firefox (97.0) did work on windows.
                                  firefox uploads worked to both local storage and nfs

                                  chrome didnt work on windows when i tried yesterday. however today when i rdp'd in i noticed it was awaiting an upgrade. post upgrade it did manage to upload (im not sure how out of date it was, i dont use windows much)

                                  so currently it means we can work with xoa uploader on windows. but not mac (at least, not m1x arm64 macs, ill try pull out one of my older macs later and see if it works on x86-64 bit mac)

                                  if theres any additional tests you need me to perform to try help that other user @nraynaud mentioned id be happy to.

                                  nraynaudN 2 Replies Last reply Reply Quote 0
                                  • nraynaudN Offline
                                    nraynaud XCP-ng Team @ITJamie
                                    last edited by nraynaud

                                    @ITJamie thanks! it's the first clue ever we get on this problem.

                                    and it is front side, which makes it a bit easier to find.

                                    It's not a slam dunk, because it works here, on a mac with chrome, mut we have a lead.

                                    ITJamieI 1 Reply Last reply Reply Quote 0
                                    • ITJamieI Offline
                                      ITJamie @nraynaud
                                      last edited by

                                      @nraynaud
                                      if there is a way to quickly enable a lot of console logging for this im happy to swap out the xoa build and re-test and give you logs

                                      nraynaudN 1 Reply Last reply Reply Quote 0
                                      • nraynaudN Offline
                                        nraynaud XCP-ng Team @ITJamie
                                        last edited by

                                        @ITJamie oh I see you're building from sources, even better, would you mind being my eyes in Chrome debugger?

                                        when you click on the 'import' button (maybe for a failing VMDK disk, it's the simplest code path that fails) there should be a POST request to a weird URL , if you could tell me the content-length of the request (not the response), and if you see anything untowards in the payload.

                                        here is a capture of a normal thing on my computer:
                                        https://gfycat.com/aggressivedisgustinggopher (there is an HD button at the bottom right)
                                        the payload size should be a few KB and at the bottom there should be the file name. No private information transits here, it's the addresses of the the blocks in the file, not their content.

                                        Thank you for your help.

                                        ITJamieI 1 Reply Last reply Reply Quote 0
                                        • nraynaudN Offline
                                          nraynaud XCP-ng Team @ITJamie
                                          last edited by

                                          @ITJamie No, we don't have any hierarchical log system on the front side AFAIK.

                                          ITJamieI 1 Reply Last reply Reply Quote 0
                                          • ITJamieI Offline
                                            ITJamie @nraynaud
                                            last edited by

                                            @nraynaud So this is interesting.
                                            there is no payload at all, and it takes quite awhile for any of the tabs (including headers) to show any data.
                                            the response and request section are completely empty.

                                            heres the "copied as curl"

                                            curl 'http://<ip>/api/xwE65b7EvmOp46LgogDxER7co_8Jl1P2NktYW_oszOs' -X POST -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:96.0) Gecko/20100101 Firefox/96.0' -H 'Accept: */*' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate' -H 'Referer: http://<ip>/' -H 'Content-Type: multipart/form-data; boundary=---------------------------274750692132175193903430548728' -H 'Origin: http://<ip>' -H 'Content-Length: 960097718' -H 'Connection: keep-alive' -H 'Cookie: <removed>'
                                            
                                            ITJamieI 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post