[XCP-ng 8] max_grant_frames reached

cfischer

Hello,

got the following while started a VyOS router:

[    2.072983] xen:grant_table: xen/grant-table: max_grant_frames reached cur=32 extra=1 limit=32 gnttab_free_count=191 req_entries=65
[    2.072985] xen_netfront: can't alloc rx grant refs
[    2.092487] vif vif-7: only created 3 queues

Do you know who is limiting domU to 32 pages while dom0 has 64 per default?

11:14 xs1 ~]# xen-diag gnttab_query_size 0
domid=0: nr_frames=1, max_nr_frames=64
[11:14 xs1 ~]# xen-diag gnttab_query_size 1
domid=1: nr_frames=32, max_nr_frames=32

Thanks
Christian

olivierlambert

What kind of template are you using? Other media install?

cfischer

Yes, Other media

cfischer

@olivierlambert: I can't find a restriction of this kind in the template.

olivierlambert

But is it causing any issue in the VM?

cfischer

@olivierlambert: Are you asking for issues with the VM? Yes, the issue is that I get round about 7.5 of 9 vif if I have 4 vcpu due to the lack of grants.

I've freshly installed and updated OL7 and Debian9 using the provided templates.

13:19 xs1 ~]# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  1840     4     r-----    4892.2
ER1-de.egr.lan                               1   512     4     -b----      86.5
Debian Stretch 9.0 (1)                       7   511     4     -b----      14.5
Oracle Linux 7 (1)                           9  4095     4     -b----      71.2
[13:19 xs1 ~]# xen-diag gnttab_query_size 0
domid=0: nr_frames=1, max_nr_frames=64
[13:19 xs1 ~]# xen-diag gnttab_query_size 1
domid=1: nr_frames=32, max_nr_frames=32
[13:19 xs1 ~]# xen-diag gnttab_query_size 7
domid=7: nr_frames=5, max_nr_frames=32
[13:19 xs1 ~]# xen-diag gnttab_query_size 9
domid=9: nr_frames=5, max_nr_frames=32

Grants for domU are always 32. What is causing the grants limit?

cfischer

@olivierlambert: I've read since 4.10 it is a domU configuration via max_grant_frames parameter. Since XCP-ng has no config files per domU, how can I set domU parameter?

I've tried it via

xe vm-param-set platform:max_grant_frames=64 uuid=b61297e6-a413-135a-fa1e-647b547247ef

but no success.

olivierlambert

Please use Markdown syntax for code/console blocks (I edited your post)
Sorry I don't understand the problem you have, can you rephrase?

cfischer

@olivierlambert: The problem is that there are not enough grants for creating queues for 9 vif on a system running 4 vcpu. Trying to create the 4th queue for vif-7 raises a max_grant_frames reached error. The netfront driver could create only 3 of 4 queues for vif-7 and 0 queues for vif-8, which is configured in Xen but therefore not present in the guest.

That is not a problem of the "Other media install" template, all created VMs (different templates, different OSs) have the same issue, max_nr_frames=32, which seems a default value for XCP-ng. Since Xen version 4.10 max_grant_frames isn't a global config parameter issued to the hypervisor at boot time anymore. Since version 4.10 it is a parameter per domU.

The question is, since XCP-ng has no config files per domU, how can I set domU parameter?

olivierlambert

@cfischer thanks for your feedback. I might have a solution. Note this is a really complex problem in the end (interactions between xenops/xenguest and XAPI)

I might have an patch idea to see if it works. Stay tuned.

olivierlambert

Due to the complicated modification, we'd like to have a support ticket created please, so we can continue to discuss those details in depth

cfischer

@olivierlambert: How can I open a support ticket without buying support?

olivierlambert

Here: https://xen-orchestra.com/#!/member/support

cfischer

@olivierlambert: Thanks, I wasn't aware the possibility submitting tickets. Ticket created.

olivierlambert

This is a free support without any commitment to answer, but it's a good way to discuss more in depth details when we can.

rjt

Does not having all the grants requested result in network slowdown?

rjt

@rjt
The November 2019 security, bugfix, and enhancement updates address this issue in the section "Enable increasing max_grant_frames for guests."

Indicates that with not enough grant frames, the guest VM will not be able to use the all the virtual NICs that it expects. Still wonder if increasing max_grant_frames would speed up existing network interfaces even when they have enough working vNICS.