Preventing "new network" detection on different XCP-NG hosts

Zevgeny

Hi, we are running two XCP-NG hosts on two different locations. One site backs up to the 2nd site for disaster recovery. Our network admin has created a VLAN for us, which all the VMs are configured to access.

In the event of a failure in the primary site, the network admin will move the VLAN over to the 2nd site, and we should theoretically be able to make a copy of the DR-ed VMs, start them up, and they should keep the same static IP address and start working without realizing they are on a different host.

However, in practice there seems to be a problem. When the VMs are started on the 2nd host, they detect it as a new ethernet network, and lose their static IP addresses. On Linux, this seems to be related to the fact that the copied VM has a different MAC address. On Windows Server 2019, this seems to be related to the Network Location Awareness service. I'm not sure what triggers it to recognise a new network.

I am investigating ways to disable this functionality on the OS level, but I want to approach it from the XCP side as well in case no solution can be found on the OS side.

One idea is to stop using static IP addresses, and instead use DHCP to bind a fixed IP address to a VM's MAC address. However, is there any way to configure XO or XCP to preserve the same MAC address when copying a VM?

There's also something like this:
https://www.groovypost.com/howto/change-mac-address-windows-10-why/

However, the setting "Locally Administered Address" doesn't exist in the advanced properties of the "XCP-ng PV Network Device".

Any other ideas or suggestions would be much appreciated, thanks!

olivierlambert

Maybe @fohdeesha will get some ideas

Forza

Perhaps make a note in the VM description that the mac address needs to be aa:bb... And make it part of the routine to check this before starting a vm.

auspugs

@zevgeny
Solutions inside windows itself:

One option would be a script that would check if the static IP is assigned to an interface and if it is not, then assign it to the first interface of the Windows Server, you could use a VBscript, and use wscript or cscript to run the code.

You could also use a netsh script (I know Microsoft claims netsh is depreciated it has been included in windows 11), or a powershell script would also work.

On the Linux VM's

Depending on the distro and the version you are using, netplan can be setup so it applies the static IP address using the order of the device, mac address, label, or even the card and port of the network device. If it is an older version, then ifupdown can also be setup to allocate the IP based on the order the interfaces appear, or via a label.

However I would not do it that way.

I dont fully understand how you are doing HA, and ensuring the VM that starts in the second site is up to date? If you are moving the VM after the site goes down, does that mean there is a second link, that joins the two sites?

If there is a second link, do you have the VM on a shared storage? If you do, then you should be able to migrate the VM over, and therefore the mac will stay the same. iSCSI can work between sites, it really depends on if the services you are running are hard drive intense or memory intense. Alternatively, if there is a second link, can you route the site through the second link, and have the VM's running from the first site?

Assuming you can not do a migrate, and you somehow get a copy of the VM to the other site. Then when you create the VM in the secondary site, you can always create the VIF that you attach to the new VM. Using the command below, in a simple script, can keep the mac the same on the new VM, I would suggest to script the whole VM creation in this case.

xe vif-create vm-uuid=uuid_of_the_vm device=0 (or the number of the physical interface you want to use on the xcp box) network-uuid=uuid_of_network_to_connect_to mac=mac_address

Keep in mind with a windows server, spinning up a VM that has a snapshot, or a image of a server that is not current at the time the other Server was taken offline, then you will be creating issues with Active Directory (assuming you are using AD), and this could lead to corruption within the AD that you will not be able to fix without MS help.

The ideally solution in my view, would be having VM running at both sites, in a active passive setup. Again depending on the services you have on the servers, and the network equipment you have. You can setup a VRRP (Virtual Router Redundancy protocol) or do a reverse proxy with something like NGINX to direct the traffic to the second site if the main link is down.

I understand you may not be able to share too much information in a public forum due to security, If you want to send me a private with more details I can help you out more with reference to your setup.

Zevgeny

@auspugs

Thanks for the reply. Our setup isn't that fancy so probably ok to talk about it.

We're just doing a simple "Disaster Recovery Backup" from XO Community Edition, every night from the primary to backup site. Our data isn't so crucial so the loss of part of a day is ok. We are not using Active Directory. The VMs are stored on internal harddrives on each host.

When the primary site goes down, we make a fast clone of the DR backups in the backup site, and then start the VMs. We contact our network admin, who moves the VLAN over to the backup site, redirects external IPs etc.

The level of sysadmin knowledge is not high in my organisation (although I am a programmer, at least) so I don't want to introduce anything too complex in case I am not around.

Does XO allow us to run a script after the DR backup process? Am I able to set the MAC address on the backup VM directly? When we fast clone the DR backup to boot the VMs, can the fast clone operation be set to preserve the Mac address? Or automatically run a script to do so?

I'm trying to avoid the situation where our DR procedure requires staff to manually set the IP addresses in the VMs, or the MAC addresses in the XO / XCP Centre interface. But worst case it's possible, I guess.

auspugs

@zevgeny

That does make it easier, I would say the easiest way to do what you are after is to use a NFS/SMB share for the VM storage. Have the VM setup on the second site with the current mac address.

If you have that already setup, then all you need to do ensure the clone from your DR is exported to the correct location and named correctly, then you can start the VM and you are all set.

Just some extra information for others people who may have a similar issue, the reason I am suggesting a SMB/NFS share, is both these SR, are file based storage, therefore you have the VHD listed as a file on the system. Unlike Block storage ext, ext4, iscsi, FC etc, these SR use block addresses to separate the different drives.

The other option would be to use the ovf format of the VM. If you export the current VM in the OVF format then add in the network section as per below replacing the 88:88:88:88:88 with your mac address.

With this setup you would replace the vhd listed int he OVF with the cloned device from your DR, then import the OVF. If you DR system output the VHD to a certain location, you can simple store the ovf in that location, removing a step.

<VirtualHardwareSection>
      <Item>
        <rasd:AddressOnParent>7</rasd:AddressOnParent>
        <rasd:AutomaticAllocation>true</rasd:AutomaticAllocation>
        <rasd:Connection>Network 1</rasd:Connection>
        <rasd:Address>88:88:88:88:88:88</rasd:Address>
        <rasd:Description>E1000 ethernet adapter on "Network 1"</rasd:Description>
        <rasd:ElementName>Network adapter 1</rasd:ElementName>
        <rasd:InstanceID>8</rasd:InstanceID>
        <rasd:ResourceSubType>E1000</rasd:ResourceSubType>
        <rasd:ResourceType>10</rasd:ResourceType>
      </Item>
</VirtualHardwareSection>

Zevgeny

@auspugs

Hmm, so are you saying to have the VMs already existing on the backup site, and when we do the DR backup we only copy the VHD file to a SMB share?

When we need to boot the VM, we'd have to copy the VHD file to the host backup machine, replacing the one attached to the VM at the time?

That sounds more complex than just typing the MAC address before starting the backup VM, though. And I'm not sure if it's possible to use XO's DR backup feature to copy the VHD files only...

fohdeesha

@Zevgeny As you suspected, this is caused by the VMs booting with new & fresh MAC addresses they've never seen before, which from their point of view means an entirely new NIC. The "clean" solution here would be to have an option or toggle inside of XOA backup jobs, that allows MAC addresses to be preserved - this way the replicated DR VMs on the backup site still have the same MAC addresses. This could be something you could file as a feature request on our github

However note that pretty much any VM copy/replicate action will trigger xen to generate a new MAC address (for safety, duplicated MACs are usually bad). This means that even if the MAC is preserved to the DR VMs, when your admin copies the DR VMs and boots them, the new copies will have newly generated MACs. I suppose it would be possible to add a "preserve MAC" checkbox for copy operations too, but I'm not sure if XAPI currently exposes such functionality.

Zevgeny

@fohdeesha

Thanks for the reply. It truly sounds like the easiest and simplest solution is to require the admin to manually enter the required MAC address before booting the backup VM.

Just a question about MAC addresses, in XCP-NG and its underlying technologies, are there any restrictions on valid MAC addresses? I mean over and above the IEEE standard.

I'd like to specify some easy to type and remember mac addresses for my VMs. From reading the wikipedia article and other articles, it seems like something like 02-11-11-11-11-11, 02-22-22-22-22-22, etc would be valid? As long as I have 02 at the beginning?

Is there a range of MAC addresses where I can guarantee won't collide with the automatically generated ones? Do you have any advice on a nicely human-readable set of MAC addresses I can use? I only need a range of about 10 in total...

fohdeesha

@Zevgeny there's no special reserved MACs or special validation in XCP-ng, as it just uses linux networking via OpenVswitch underneath, so if your custom MACs adhere to IEEE standards they will work. Of course, make sure you don't collide with the MAC addresses on the physical adapter in the server (plus maybe 3 or 4 sequentially above that) as XCP-ng uses those in dom0

Zevgeny

@fohdeesha Great, thank you!

swtrse

Is there a Solution yet to make MAC address for a VM static?
Because this is a showstopper for me.
I never hat this kind of problem with VMWare till I was forced to switch to another hypervisor.
My whole setup depends on the DHCP to give out correct static leases.

olivierlambert

Hi @swtrse

What do you mean exactly? Can you illustrate what you want to achieve with an example?

nikade

@fohdeesha said in Preventing "new network" detection on different XCP-NG hosts:

@Zevgeny As you suspected, this is caused by the VMs booting with new & fresh MAC addresses they've never seen before, which from their point of view means an entirely new NIC. The "clean" solution here would be to have an option or toggle inside of XOA backup jobs, that allows MAC addresses to be preserved - this way the replicated DR VMs on the backup site still have the same MAC addresses. This could be something you could file as a feature request on our github

However note that pretty much any VM copy/replicate action will trigger xen to generate a new MAC address (for safety, duplicated MACs are usually bad). This means that even if the MAC is preserved to the DR VMs, when your admin copies the DR VMs and boots them, the new copies will have newly generated MACs. I suppose it would be possible to add a "preserve MAC" checkbox for copy operations too, but I'm not sure if XAPI currently exposes such functionality.

Yea, keeping the mac adress would really solve this.
We actually edit the mac adress manually on the copied VM's incase we need to start the DR copy up for testing.