Slow XOA VM download (~250KBps)

bryan_v

I've been trying to get XOA setup on a new device for about a day and the web page and cli install methods both would keep hanging at the "Importing XOA VM..." stage ("Deploying..." if using the web version).

I think I finally tracked down the culprit to a slow download from XOA directly. Specifically the command wget http://xoa.io:8888/ reveals the download speed is between 250KB/s and 350KB/s with an ETA of 1hr to pull down the whole VM.

Is there another mirror I can use that might be faster?

I'm on a 1Gbps WAN connection, and I ran a speedtest just now to double check.

olivierlambert

Hi,

Where are you based?

bryan_v

olivierlambert Downtown Toronto, Canada

olivierlambert

I suspect it's the peering issue between your ISP and where the file is hosted. We do not have mirrors right now, but this might change in the future. Have you retried to download at different hours of the day?

bryan_v

olivierlambert Yeup, you're right, just tested it with EC2 in us-east-1, and I get ~12MB/s.

I also tested yesterday afternoon and evening, same issue.

Where is the XOA VM hosted? I'm trying to track exactly where chain the bottleneck is.

And can we put something like Cloudflare in front of the host so there's no direct cost and it gets edge cached?

olivierlambert

It's hosted in Lyon, France, with Jaguar Network AS.

bryan_v

olivierlambert I've dug into the issue a bit more, and it just gets stranger.

Downloading to my Macbook on the same network gets ~4MB/s. This means there is something strange going on with the control domain around remote downloads.

When I test the disk speed (which is an Samsung NVMe), the speed is 1.7GB/s:

[14:31 xcp-host ~]# dd if=/dev/zero of=testfile oflag=direct bs=1M count=5000 status=progress
4998561792 bytes (5.0 GB) copied, 3.000607 s, 1.7 GB/s
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 3.15049 s, 1.7 GB/s

When I test the network speed with Ookla's speedtest-cli it confirms a 1G connection, and a 1G transit to Europe (checked this twice because 1G Atlantic transits are rare):

[14:44 xcp-host ~]# speedtest -s 2023

   Speedtest by Ookla

     Server: LaFibre.info - Lyon (id = 2023)
        ISP: Bell Canada
    Latency:   108.05 ms   (0.49 ms jitter)
   Download:   960.47 Mbps (data used: 1.5 GB )
     Upload:   172.11 Mbps (data used: 293.4 MB )
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/b8f1da9f-5c57-4ad8-ad3b-729e1625affe

On my Macbook (on 802.11ac) I can download an Ubuntu image at ~14MB/s

➜ wget http://ubuntu.mirrors.ovh.net/ubuntu-releases/21.10/ubuntu-21.10-desktop-amd64.iso
--2021-11-16 14:49:41--  http://ubuntu.mirrors.ovh.net/ubuntu-releases/21.10/ubuntu-21.10-desktop-amd64.iso
Resolving ubuntu.mirrors.ovh.net (ubuntu.mirrors.ovh.net)... 213.32.5.7
Connecting to ubuntu.mirrors.ovh.net (ubuntu.mirrors.ovh.net)|213.32.5.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3116482560 (2.9G) [application/octet-stream]
Saving to: ‘ubuntu-21.10-desktop-amd64.iso’

ubuntu-21.10-desktop-amd64.iso       3%[=>           ] 107.27M  14.4MB/s    eta 4m 14s

However on the control domain the download speed is considerably lower:

[14:52 xcp-host ~]# wget http://ubuntu.mirrors.ovh.net/ubuntu-releases/21.10/ubuntu-21.10-desktop-amd64.iso
--2021-11-16 14:53:03--  http://ubuntu.mirrors.ovh.net/ubuntu-releases/21.10/ubuntu-21.10-desktop-amd64.iso
Resolving ubuntu.mirrors.ovh.net (ubuntu.mirrors.ovh.net)... 213.32.5.7
Connecting to ubuntu.mirrors.ovh.net (ubuntu.mirrors.ovh.net)|213.32.5.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3116482560 (2.9G) [application/octet-stream]
Saving to: ‘ubuntu-21.10-desktop-amd64.iso.1’

 3% [==>           ] 106,910,815 3.20MB/s  eta 16m 47s

Have you ever see something like this before?

I saw you had some discussion on another thread regarding slow writes in a VM, but this looks like it affects the control domain as well:
https://xcp-ng.org/forum/topic/4865/slow-disk-writes-inside-vm-on-8-2