Hi, I'm pretty sure I need to build a newer version of the i40e drivers (2.17.4) from source so that VFs are properly supported on Intel X710 cards. It looks like the latest version in XCP is 2.9.21 and VF support via sysfs was only fixed in 2.14.13. Right now, the 2.9.21 driver tries to initialize support for 64 VFs, when the actual limit is 32, causing the entire subsystem to panic and blocking the creation of any VFs, and no matter what modprobe setting I change, it always tries to initialze 64 VFs. (I assume because the original version of this controller, the XL710, had support for 64 VFs per port)

No one would really notice this, unless they tried to connect a VM via SR-IOV VFs rather than a traditional VIF.

Three quick questions:

1. Is there a guide to build XCP-ng drivers from source?
2. I assume https://github.com/xcp-ng/xcp-ng-build-env is still the official build environment?
3. I assume I need to roll the driver into an RPM, them move it back to my system for install?
4. Is there a reference repo for the i40e driver that I can fork and update with the new 2.17.4 driver?

Some of this is from this old forum post: https://xcp-ng.org/forum/topic/212/build-network-drivers-from-sources/2

Proper use of the X710's VF functionality via SR-IOV should offload all the bridge work to card, instead of doing it via VIFs and OVS (I don't know how to max out the 32 VFs per port limit but that's a good problem to have).

And yes, I know that the VIFs technically have a higher bandwidth than the X710's 10G ports, but why waste CPU cycles when you don't have to? Plus aren't you curious to see if an SR-IOV VF bridge is faster than a VIF OVS bridge?

@olivierlambert I've dug into the issue a bit more, and it just gets stranger.

Downloading to my Macbook on the same network gets ~4MB/s. This means there is something strange going on with the control domain around remote downloads.

When I test the disk speed (which is an Samsung NVMe), the speed is 1.7GB/s:

[14:31 xcp-host ~]# dd if=/dev/zero of=testfile oflag=direct bs=1M count=5000 status=progress
4998561792 bytes (5.0 GB) copied, 3.000607 s, 1.7 GB/s
5000+0 records in
5000+0 records out
5242880000 bytes (5.2 GB) copied, 3.15049 s, 1.7 GB/s

When I test the network speed with Ookla's speedtest-cli it confirms a 1G connection, and a 1G transit to Europe (checked this twice because 1G Atlantic transits are rare):

[14:44 xcp-host ~]# speedtest -s 2023

   Speedtest by Ookla

     Server: LaFibre.info - Lyon (id = 2023)
        ISP: Bell Canada
    Latency:   108.05 ms   (0.49 ms jitter)
   Download:   960.47 Mbps (data used: 1.5 GB )
     Upload:   172.11 Mbps (data used: 293.4 MB )
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/b8f1da9f-5c57-4ad8-ad3b-729e1625affe

On my Macbook (on 802.11ac) I can download an Ubuntu image at ~14MB/s

➜ wget http://ubuntu.mirrors.ovh.net/ubuntu-releases/21.10/ubuntu-21.10-desktop-amd64.iso
--2021-11-16 14:49:41--  http://ubuntu.mirrors.ovh.net/ubuntu-releases/21.10/ubuntu-21.10-desktop-amd64.iso
Resolving ubuntu.mirrors.ovh.net (ubuntu.mirrors.ovh.net)... 213.32.5.7
Connecting to ubuntu.mirrors.ovh.net (ubuntu.mirrors.ovh.net)|213.32.5.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3116482560 (2.9G) [application/octet-stream]
Saving to: ‘ubuntu-21.10-desktop-amd64.iso’

ubuntu-21.10-desktop-amd64.iso       3%[=>           ] 107.27M  14.4MB/s    eta 4m 14s

However on the control domain the download speed is considerably lower:

[14:52 xcp-host ~]# wget http://ubuntu.mirrors.ovh.net/ubuntu-releases/21.10/ubuntu-21.10-desktop-amd64.iso
--2021-11-16 14:53:03--  http://ubuntu.mirrors.ovh.net/ubuntu-releases/21.10/ubuntu-21.10-desktop-amd64.iso
Resolving ubuntu.mirrors.ovh.net (ubuntu.mirrors.ovh.net)... 213.32.5.7
Connecting to ubuntu.mirrors.ovh.net (ubuntu.mirrors.ovh.net)|213.32.5.7|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3116482560 (2.9G) [application/octet-stream]
Saving to: ‘ubuntu-21.10-desktop-amd64.iso.1’

 3% [==>           ] 106,910,815 3.20MB/s  eta 16m 47s

Have you ever see something like this before?

I saw you had some discussion on another thread regarding slow writes in a VM, but this looks like it affects the control domain as well:
https://xcp-ng.org/forum/topic/4865/slow-disk-writes-inside-vm-on-8-2

@olivierlambert Yeup, you're right, just tested it with EC2 in us-east-1, and I get ~12MB/s.

I also tested yesterday afternoon and evening, same issue.

Where is the XOA VM hosted? I'm trying to track exactly where chain the bottleneck is.

And can we put something like Cloudflare in front of the host so there's no direct cost and it gets edge cached?

@olivierlambert Downtown Toronto, Canada

I've been trying to get XOA setup on a new device for about a day and the web page and cli install methods both would keep hanging at the "Importing XOA VM..." stage ("Deploying..." if using the web version).

I think I finally tracked down the culprit to a slow download from XOA directly. Specifically the command wget http://xoa.io:8888/ reveals the download speed is between 250KB/s and 350KB/s with an ETA of 1hr to pull down the whole VM.

Is there another mirror I can use that might be faster?

I'm on a 1Gbps WAN connection, and I ran a speedtest just now to double check.