[XCP-ng 8] max_grant_frames reached
-
Hello,
got the following while started a VyOS router:
[ 2.072983] xen:grant_table: xen/grant-table: max_grant_frames reached cur=32 extra=1 limit=32 gnttab_free_count=191 req_entries=65 [ 2.072985] xen_netfront: can't alloc rx grant refs [ 2.092487] vif vif-7: only created 3 queuesDo you know who is limiting domU to 32 pages while dom0 has 64 per default?
11:14 xs1 ~]# xen-diag gnttab_query_size 0 domid=0: nr_frames=1, max_nr_frames=64 [11:14 xs1 ~]# xen-diag gnttab_query_size 1 domid=1: nr_frames=32, max_nr_frames=32Thanks
Christian -
What kind of template are you using? Other media install?
-
Yes, Other media
-
@olivierlambert: I can't find a restriction of this kind in the template.
-
But is it causing any issue in the VM?
-
@olivierlambert: Are you asking for issues with the VM? Yes, the issue is that I get round about 7.5 of 9 vif if I have 4 vcpu due to the lack of grants.
I've freshly installed and updated OL7 and Debian9 using the provided templates.
13:19 xs1 ~]# xl list Name ID Mem VCPUs State Time(s) Domain-0 0 1840 4 r----- 4892.2 ER1-de.egr.lan 1 512 4 -b---- 86.5 Debian Stretch 9.0 (1) 7 511 4 -b---- 14.5 Oracle Linux 7 (1) 9 4095 4 -b---- 71.2 [13:19 xs1 ~]# xen-diag gnttab_query_size 0 domid=0: nr_frames=1, max_nr_frames=64 [13:19 xs1 ~]# xen-diag gnttab_query_size 1 domid=1: nr_frames=32, max_nr_frames=32 [13:19 xs1 ~]# xen-diag gnttab_query_size 7 domid=7: nr_frames=5, max_nr_frames=32 [13:19 xs1 ~]# xen-diag gnttab_query_size 9 domid=9: nr_frames=5, max_nr_frames=32Grants for domU are always 32. What is causing the grants limit?
-
@olivierlambert: I've read since 4.10 it is a domU configuration via max_grant_frames parameter. Since XCP-ng has no config files per domU, how can I set domU parameter?
I've tried it via
xe vm-param-set platform:max_grant_frames=64 uuid=b61297e6-a413-135a-fa1e-647b547247efbut no success.
-
- Please use Markdown syntax for code/console blocks (I edited your post)
- Sorry I don't understand the problem you have, can you rephrase?
-
@olivierlambert: The problem is that there are not enough grants for creating queues for 9 vif on a system running 4 vcpu. Trying to create the 4th queue for vif-7 raises a max_grant_frames reached error. The netfront driver could create only 3 of 4 queues for vif-7 and 0 queues for vif-8, which is configured in Xen but therefore not present in the guest.
That is not a problem of the "Other media install" template, all created VMs (different templates, different OSs) have the same issue, max_nr_frames=32, which seems a default value for XCP-ng. Since Xen version 4.10 max_grant_frames isn't a global config parameter issued to the hypervisor at boot time anymore. Since version 4.10 it is a parameter per domU.
The question is, since XCP-ng has no config files per domU, how can I set domU parameter?
-
@cfischer thanks for your feedback. I might have a solution. Note this is a really complex problem in the end (interactions between xenops/xenguest and XAPI)
I might have an patch idea to see if it works. Stay tuned.
-
Due to the complicated modification, we'd like to have a support ticket created please, so we can continue to discuss those details in depth
-
@olivierlambert: How can I open a support ticket without buying support?
-
-
@olivierlambert: Thanks, I wasn't aware the possibility submitting tickets. Ticket created.
-
This is a free support without any commitment to answer, but it's a good way to discuss more in depth details when we can.
-
Does not having all the grants requested result in network slowdown?
-
@rjt
The November 2019 security, bugfix, and enhancement updates address this issue in the section "Enable increasing max_grant_frames for guests."Indicates that with not enough grant frames, the guest VM will not be able to use the all the virtual NICs that it expects. Still wonder if increasing max_grant_frames would speed up existing network interfaces even when they have enough working vNICS.
