XCP-ng
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Unable to create more than 7 VIFs for a VM / Adding Trunk Network to VM

    Scheduled Pinned Locked Moved
    Management
    2
    5
    381
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      Morpheus0x
      last edited by Morpheus0x

      Hello, I am experiencing an issue with Xen Orchestra (self compiled) not being able to add a new VIF if the VM has a total of 7 VIFs already attached.

      I get the following error trying to create the new Network Device:

      could not find an allowed VIF device

      This is only after updating Xen Orchestra to the latest commit (771b04a), before the error was:

      INTERNAL_ERROR(Server_helpers.Dispatcher_FieldNotFound("device"))

      I have tested adding an 8th VIF to multiple VMs using different Networks, but nothing makes a difference. The reason I need that "many" is basically only for the Firewall VM since I am adding a VIF for every Subnet/Vlan in my network. A workaround would be to just attach the trunk to the Firewall VM. However, I would be better if I could attach more than 7 VIFs.

      I found this XCP-ng guide stating that 7 VIFs is some limitation in Xen. I think this should be highlighted in the normal Network documentation as well. Can this restriction be changed? Does adding 8+ VIFs via CLI introduce issues like the guide states?
      Is there any maximum limitation on how many VIFs are allowed or recommended for a VM in XCP-ng or is this a Xen Orchestra limitation?

      The same guide also states that if I want to use the trunk interface inside a VM, that I need to change the MTU of the PIF to 1504. Why is that the case? I changed the 7th VIF (#6) of my firewall VM to a network without any VLAN set (trunk) and now in OPNsense I created a VLAN interface on that network interface. However, that VLAN interface is down and unable to communicate with a VM using a Xen Orchestra network with the same VLAN set. Is this due to the MTU issues? Where exactly do I need to change the MTU, beside on the network switch? I have a LACP (802.3ad) with two SFP+ ports. Do I set 1504 MTU for each of its members, eth0 and eth1, and/or for the bond0 PIF?

      Edit: I resolved my last issue, now I can successfully use the trunk port inside OPNsense without having to change the MTU. The issue was me not correctly setting up OPNsense, but now I can ping the firewall from the VM as described in the setup above. Should I still change MTU?

      A 1 Reply Last reply Reply Quote 0
      • A
        Andrew Top contributor @Morpheus0x
        last edited by

        @Morpheus0x You can look at this post.

        M 1 Reply Last reply Reply Quote 0
        • M
          Morpheus0x @Andrew
          last edited by

          @Andrew thanks, I know that it is possible to add additional interfaces using the XCP-ng CLI. I was looking for information on if doing this causes any issues. From the thread you linked, I would assume that the 7 VIF limit exists only because some operating systems have issues with too many virtual Xen network interfaces. Are there any other considerations when adding additional VIFs?

          I've got the trunk port working in the firewall VM and since OPNsense is based on FreeBSD, I will use that instead of adding additional VIFs from the CLI.

          However, the MTU question still remains. Do I need to change it? It works for now, but I don't know if there could be any issues, e.g. limited network throughput when not changing MTU...

          A 1 Reply Last reply Reply Quote 0
          • A
            Andrew Top contributor @Morpheus0x
            last edited by

            @Morpheus0x FreeBSD will crash if you add too many VIFs.

            As for the MTU... You can assume that the normal 1500 MTU is needed BEFORE a VLAN tag is added. On a normal interface, the VLAN tag is added AFTER the MTU in the OS by the hardware interface (adds 4 bytes+the packet).

            If you are connecting a Xen hardware interface (without VLANs) to a VM interface and then trying to do VLANs in the VM, yes, you will need to add 4 to the Xen interface MTU because the VM is adding the VLAN tag and the Xen interface sees 1504 bytes in the packet (VLAN tag+data). You'll also need your switch port to handle tagged frames. Note, many switches won't accept tagged frames that match the native VLAN for the port.

            Most physical interfaces can deal with baby giants that are MTUs a little larger than 1500 but smaller than Jumbo frames (ie. 9000).

            M 1 Reply Last reply Reply Quote 2
            • M
              Morpheus0x @Andrew
              last edited by Morpheus0x

              @Andrew awesome, thank you for your explanation. After reading a bit more about MTU (see this) I get it now. I looked up MTU in OPNsense for the VLAN interfaces and was surprised that it automatically uses 1496 MTU, which explains why it works and means I don't have to adjust my host's MTU settings, which I really didn't want to do.

              Now, as far as I understand, keeping it this way could introduce some split packets and therefore potentially limiting throughput. To fix this, I would need to increase the MTU to 1504 or larger. However, after some more reading (thanks for the keyword "baby giants"), it seems that today, MTU is just a "guideline" for LAN. I will keep the host MTU at 1500 for now.

              1 Reply Last reply Reply Quote 0
              • First post
                Last post