XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Exports OVA Timeout...

    Scheduled Pinned Locked Moved Xen Orchestra
    37 Posts 7 Posters 7.8k Views 6 Watching
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • C Offline
      chelle_belle @olivierlambert
      last edited by

      @olivierlambert - okay - one thing haven't tried is XVA export via XOA. If that falls, I can export to some other format - and zip up and ping across...

      C 1 Reply Last reply Reply Quote 0
      • C Offline
        chelle_belle @chelle_belle
        last edited by

        @chelle_belle - So i tried using an XVA export from XOA and that works like a dream - immediately see the file grow and the transfer starts immediately...

        A 1 Reply Last reply Reply Quote 0
        • olivierlambertO Offline
          olivierlambert Vates 🪐 Co-Founder CEO
          last edited by

          @florent will come here to ask some question 🙂

          1 Reply Last reply Reply Quote 0
          • A Offline
            Andrew Top contributor @chelle_belle
            last edited by

            @chelle_belle @florent I have the same result. XVA export works great.

            1 Reply Last reply Reply Quote 0
            • florentF Offline
              florent Vates 🪐 XO Team @chelle_belle
              last edited by florent

              @chelle_belle so the ova is a tar containing many files, (one 4KB ovf file containing the metadata and one vmdk file per disk containing the data)

              since it's a tar we need to know the size of each file to be able to stream it

              but the most supported subformat of vmdk is the stream optimized , which mandates to compress each 64KB part of the disk. Since we can't compute the compressed size of the vhd transformed to vmdk we need to transfert the disk, compress it to vmdk , keep track of the compressed size, then start again, this time sending the data

              Since we can't store the vmdk on xo-server disks, we need to transfer and transform the disk to vmdk twice. And the transform process is quite slow.

              As I write this, I am wondering if I can create a less efficient vmdk with empty spaces at the end so that I can do it in one pass.

              I will test it and let you know the result

              florentF 1 Reply Last reply Reply Quote 0
              • florentF Offline
                florent Vates 🪐 XO Team @florent
                last edited by

                @chelle_belle can you test this branch : fix_ova_speed

                there will still have a pause after the 5kb and after each disk, but it should speed up things . I will look for other performance optimization during this sprint.

                A 1 Reply Last reply Reply Quote 0
                • A Offline
                  Andrew Top contributor @florent
                  last edited by Andrew

                  @florent That branch is a LOT faster for OVA export but does not compress the data as much.

                  • XVA/NoComp 40 seconds to complete. 1760MB
                  • XVA/Zstd 30 seconds to complete. 560MB
                  • XVA/gzip 90 seconds to complete. 586MB
                  • New OVA 40 seconds to start, 60 seconds to download. 1570MB
                  • Old OVA 6 minutes to start, 6 minutes to download. 597MB

                  It still leaves a zombie task....

                  [XO] VM OVA export (TestSmall7-64 on xcp1) 0%

                  1 Reply Last reply Reply Quote 1
                  • olivierlambertO Offline
                    olivierlambert Vates 🪐 Co-Founder CEO
                    last edited by

                    Thanks for your feedback @Andrew as usual 🙂

                    1 Reply Last reply Reply Quote 0
                    • olivierlambertO Offline
                      olivierlambert Vates 🪐 Co-Founder CEO
                      last edited by

                      Ah I see, @florent just disabled compression entirely to see the diff, so maybe using compression level to 1 will be better while keeping that speed 🙂

                      A 1 Reply Last reply Reply Quote 0
                      • A Offline
                        Andrew Top contributor @olivierlambert
                        last edited by

                        @olivierlambert I see that now...

                        I guess the best choice would to have an option during export like with XVA. Quick-NoComp OR Slow-Compression. It could also be applied to export disk function.

                        I tried 1 but it did not seem to change the size of time. Maybe I did not change it correctly.

                        I guess technically the no-compression option could stream the data without the pre-calculation work for compression that takes time but is no longer needed.

                        1 Reply Last reply Reply Quote 0
                        • olivierlambertO Offline
                          olivierlambert Vates 🪐 Co-Founder CEO
                          last edited by

                          Maybe try 6? Don't forget to rebuild after that 🙂

                          A 1 Reply Last reply Reply Quote 0
                          • A Offline
                            Andrew Top contributor @olivierlambert
                            last edited by Andrew

                            @olivierlambert @florent My mistake.... I did not rebuild XO correctly.

                            Using level 1 is about 1 minute to start and about 1 minute to download and was 637MB. Seems to be a reasonable tradeoff for a significant increase in speed (over level 9) and a minimal decrease in compression.

                            So level 0 (no compression) actually does not help unless the code is modified to just stream the data without the pre-calculation and would cut the export time in half (no scanning/calculation needed).

                            Level 1 is reasonable if it the current procedure is maintained. Level 9 should be optional but not forced. Level 9 is very slow.... If compression is left in place then it would be nice to have some options.... 1..3..6..9... Or some names (min=1, fast=3, standard=6, max=9).

                            The quickest change is to just make it level 1. That still offers compression and still takes time but is MUCH faster and I think that's what most people want. speed...speed....speed....

                            1 Reply Last reply Reply Quote 0
                            • olivierlambertO Offline
                              olivierlambert Vates 🪐 Co-Founder CEO
                              last edited by

                              Sounds a good modification to do in the short term until make it editable indeed 🙂

                              @chelle_belle if you need assistance so we can configure it this way on your XOA, let me know 🙂

                              C 1 Reply Last reply Reply Quote 0
                              • C Offline
                                chelle_belle @olivierlambert
                                last edited by

                                @olivierlambert - I'm struggling to switch to the channel and do the update. When I got to set it I get this error message:

                                06/06/2022, 17:07:33: Start updating...
                                06/06/2022, 17:07:33: No manifest found for fix_ova_speed channel.

                                I assumed I had to switch to "Unlisted Channel" and then input fix_ova_speed as the Unlisted channel name...

                                florentF 1 Reply Last reply Reply Quote 0
                                • olivierlambertO Offline
                                  olivierlambert Vates 🪐 Co-Founder CEO
                                  last edited by

                                  @chelle_belle a release channel is not the branch on Github, which can be used only if you installed XO from the sources (https://xen-orchestra.com/docs/installation.html#from-the-sources).

                                  In theory, we could build packages to match such a release channel, but it's not trivial on our side. It might be easier to just install XO from the sources and not using master branch but fix_ova_speed. If you need assistance on doing that, I'm here 🙂

                                  1 Reply Last reply Reply Quote 0
                                  • florentF Offline
                                    florent Vates 🪐 XO Team @chelle_belle
                                    last edited by

                                    @chelle_belle If you can open a support tunnel I can deploy the fix today ( with a more reasonable compression level of 1 😅 , thanks @Andrew )

                                    Can you open a support ticket with the tunnel information ?

                                    1 Reply Last reply Reply Quote 0
                                    • A Offline
                                      Andrew Top contributor
                                      last edited by

                                      @florent @julien-f Trying the new OVA export (XO source commit c11e0)..... I exported a VM quickly and without a zombie task left over (good), but then I tried to import it and got no error when clicking import but it did nothing (failure without notice). The logs do show an error:

                                      HTTP handler of vm.import
                                      undefined
                                      {
                                        "code": "INTERNAL_ERROR",
                                        "params": [
                                          "(Failure \"Expected string, got 'N'\")"
                                        ],
                                        "call": {
                                          "method": "VM.create",
                                          "params": [
                                            {
                                              "actions_after_crash": "restart",
                                              "actions_after_reboot": "restart",
                                              "actions_after_shutdown": "destroy",
                                              "affinity": null,
                                              "HVM_boot_params": {
                                                "order": "cdn"
                                              },
                                              "HVM_boot_policy": "BIOS order",
                                              "is_a_template": false,
                                              "memory_dynamic_max": 4294967296,
                                              "memory_dynamic_min": 4294967296,
                                              "memory_static_max": 4294967296,
                                              "memory_static_min": 4294967296,
                                              "other_config": {
                                                "vgpu_pci": "",
                                                "base_template_name": "Other install media",
                                                "install-methods": "cdrom"
                                              },
                                              "PCI_bus": "",
                                              "platform": {
                                                "timeoffset": "0",
                                                "nx": "true",
                                                "acpi": "1",
                                                "apic": "true",
                                                "pae": "true",
                                                "hpet": "true",
                                                "viridian": "true"
                                              },
                                              "PV_args": "",
                                              "PV_bootloader_args": "",
                                              "PV_bootloader": "",
                                              "PV_kernel": "",
                                              "PV_legacy_args": "",
                                              "PV_ramdisk": "",
                                              "recommendations": "<restrictions><restriction field=\"memory-static-max\" max=\"137438953472\" /><restriction field=\"vcpus-max\" max=\"32\" /><restriction property=\"number-of-vbds\" max=\"255\" /><restriction property=\"number-of-vifs\" max=\"7\" /><restriction field=\"has-vendor-device\" value=\"false\" /></restrictions>",
                                              "user_version": 1,
                                              "VCPUs_at_startup": 2,
                                              "VCPUs_max": 2,
                                              "VCPUs_params": {},
                                              "blocked_operations": {},
                                              "has_vendor_device": false,
                                              "HVM_shadow_multiplier": 1,
                                              "name_description": "",
                                              "name_label": "NginX Test (ova)",
                                              "order": 0,
                                              "shutdown_delay": 0,
                                              "start_delay": 0,
                                              "version": 0
                                            }
                                          ]
                                        },
                                        "message": "INTERNAL_ERROR((Failure \"Expected string, got 'N'\"))",
                                        "name": "XapiError",
                                        "stack": "XapiError: INTERNAL_ERROR((Failure \"Expected string, got 'N'\"))
                                          at Function.wrap (/opt/xo/xo-builds/xen-orchestra-202206091200/packages/xen-api/src/_XapiError.js:16:12)
                                          at /opt/xo/xo-builds/xen-orchestra-202206091200/packages/xen-api/src/transports/json-rpc.js:37:27
                                          at AsyncResource.runInAsyncScope (node:async_hooks:202:9)
                                          at cb (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/util.js:355:42)
                                          at tryCatcher (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/util.js:16:23)
                                          at Promise._settlePromiseFromHandler (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/promise.js:547:31)
                                          at Promise._settlePromise (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/promise.js:604:18)
                                          at Promise._settlePromise0 (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/promise.js:649:10)
                                          at Promise._settlePromises (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/promise.js:729:18)
                                          at _drainQueueStep (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/async.js:93:12)
                                          at _drainQueue (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/async.js:86:9)
                                          at Async._drainQueues (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/async.js:102:5)
                                          at Immediate.Async.drainQueues [as _onImmediate] (/opt/xo/xo-builds/xen-orchestra-202206091200/node_modules/bluebird/js/release/async.js:15:14)
                                          at processImmediate (node:internal/timers:466:21)
                                          at process.callbackTrampoline (node:internal/async_hooks:130:17)"
                                      }
                                      
                                      julien-fJ 2 Replies Last reply Reply Quote 0
                                      • julien-fJ Offline
                                        julien-f Vates 🪐 Co-Founder XO Team @Andrew
                                        last edited by

                                        @Andrew This problem does not happen before the last 3 commits?

                                        I believe the issue is the affinity: null, but I don't see how it can be related to these changes. 🤔

                                        julien-fJ 1 Reply Last reply Reply Quote 0
                                        • julien-fJ Offline
                                          julien-f Vates 🪐 Co-Founder XO Team @julien-f
                                          last edited by

                                          Probably related to https://github.com/vatesfr/xen-orchestra/commit/2af5328a0f69912f5c03f5a8894fd75633bd3655

                                          0 julien-f committed to vatesfr/xen-orchestra
                                          chore(xo-server): use @xen-orchestra/xapi/VM_create
                                          1 Reply Last reply Reply Quote 0
                                          • julien-fJ Offline
                                            julien-f Vates 🪐 Co-Founder XO Team @Andrew
                                            last edited by

                                            @Andrew Should be fixed, please test and keep me posted 🙂

                                            https://github.com/vatesfr/xen-orchestra/commit/83690a4dd4457daf9ea402e623c9f3fe9ce9ab7f

                                            0 julien-f committed to vatesfr/xen-orchestra
                                            fix(xo-server/_importOvaVm): fix VM creation
                                            
                                            Fixes https://xcp-ng.org/forum/post/49920
                                            
                                            Introduced by 2af5328a0f69912f5c03f5a8894fd75633bd3655
                                            A 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post