@mietek I can assure you there's no malicious intent to mislead you, why we would do that to our users is beyond me. Olivier is simply one of the most busy people I've worked with, and he still takes the time to come here and answer free users when he can. He might not have the time to scour the internet and github for the relevant patches and news for niche threads like this. We were not made aware of the cloud-init fix until just a few days ago when a patch was submitted. Cloud-init has been notoriously hard to support and document because the upstream project is constantly doing things like this, and as you noticed it can affect how it works on one OS version versus another very differently. If it were up to me we would drop built-in support for it because of this mess (and a lot of large projects have dropped it entirely and moved to Ignition, like CoreOS) but a lot of users still find it very useful so we continue to support it as best we can, baring with the mess going on upstream.

I'm not sure if you're aware, but Olivier is the founder and CEO of Vates, who is behind both XCP-ng and XOA. I welcome you to go to the ESXI forum and try and get the CEO of VMware to personally answer your questions, as a free user to top it off.

Why is it these people can never spell?

+1 for avoiding 3rd party install scripts, the official install doc is like, 10 commands? I think it takes maybe 10 minutes or less the last time I did it on a fresh debian system, and you know it's always the correct instructions (which is NOT the case with 3rd party scripts, as you'll see in this forum when even the slightest architectural change is made to the XO sources). You also get to learn at least a little bit about the architecture of what you're installing and running instead of pressing go on a script you grabbed from some guy's github and hoping the XO web interface appears. I thought the whole point of "homelab" (for which sources are intended and primarily used) was learning and developing skills in the first place?

No disrespect intended to the people that create and maintain said scripts, it just seems to me like it bypasses the point of sources a little bit

I can start adding to this this weekend probably - would a "guide to installing pfsense" be useful? I know there's many guides out there already, but more than half of them have useless (or worse than useless) steps telling people to turn off things that don't need to be turned off

@s-pam If you don't mind about power usage, high performance (that you've already selected) is the way to go. It will avoid entering C states as often and leave the CPU clocked high among other things so there is no latency for workloads waiting on the CPU existing C states etc.

Leave x2APIC enabled, it helps to distribute interrupts between multiple cores/CPUs. There's no downside to having it enabled unless the hypervisor doesn't support it (and you would know immediately by having errors).

Regarding your last setting, also referred to as CCX as NUMA, this changes how the CPU cores are presented to the hypervisor (either as one NUMDA domain, or multiple, one for each set of cores that share cache). From the benchmarks I can find, leaving this on gains maybe 3 to 4 percent improvement https://blogs.vmware.com/performance/2020/05/app-perf-vmware-vsphere-amd-epyc-rome.html

@olivierlambert indeed, I believe it's just:

xe host-management-reconfigure pif-uuid=<PIF_UUID>

with the PIF_UUID being the UUID of the new PIF interface (bond, vlan, etc) you would like to move the existing management interface to

@station-already Hi! This can be done in XOA, but it requires deleting the existing management interface, then creating a new one on the new PIF you've created (a vlan 10 on eth0 PIF for example). You say you can't lose access to the host however, so that is probably not an option. XCP-ng center, the windows client, has a bit of a workaround, it has a dropdown box that will allow you to change the PIF of an existing management interface/IP address to a new interface. In XCP-ng center just go to the Networking tab of the host, then hit configure at the bottom beneath the management interfaces section.

It will have a new pop up window with a dropdown for where you want the existing management interface to exist. Assuming you've already created the vlan network you want in xcp-ng, it should show up in this list. just choose it and hit ok:

@olivierlambert I believe he means be able to rename / map a hosts physical interfaces to different namings. For instance on host 1, "network 0 / eth0" would be mapped to the physical eth0 interface, but on another host in the pool with a different NIC configuration, he could map the same pool "network 0 / eth0" to the server's physical eth1.

@S-Pam this certainly isn't currently supported, I believe it would take a lot of rewriting the network backend to make it possible.

Hey again:

1. With the latest debian and the latest cloud-init, no there should not be any extra steps now that the weird drive detection bug has been patched upstream. The important thing is that cloud-init inside your template VM is set to look for the nocloud datatype among others, and in the default config, it's already in the list of types of sources to look for so it shouldn't need changing

2. No at this time I am not aware of any plans to add ignition support as there hasn't been any demand (we prioritize features added by demand mostly). Personally, pretending I'm a non-employee, I would like to see Ignition support included as it would be quite handy, so it may be something we can approach down the road

If it wasn't already mentioned, the preferred data source now is NoCloud: https://cloudinit.readthedocs.io/en/latest/topics/datasources/nocloud.html

That's what XOA passes to VMs, a NoCloud config drive. Note that there was a serious upstream bug in cloud-init that was stopping OS's like debian from seeing this drive - as Olivier linked above it seems it was finally just now fixed.

I just confirmed with tests that Disaster Recovery jobs do indeed set the blocked-operations parameter for backed up VMs. As Julien said, did you ever force-start this VM? That would have removed start from the blocked-operations field, which is why we don't see it in your param-list, and why it's now being allowed to autostart.

@thorcraftit seems like perhaps a permissions error, what does your share properties on truenas look like for this host? Is the allowed IP or denied IP boxes populated at all? Attach a screenshot if you can. Here's what mine looks like:

Note that the mapall user and group must have permissions on the shared folder, I typically create a user on the freenas system just for xen shares, and give it ownership of the folder/dataset that's being shared. this ensures you don't have any weird permissions issues. I think in your case the key boxes to examine are the authorized networks however.

What does your Services > NFS > Settings look like? here's mine:

Lastly and perhaps most importantly, can you run "yum install nmap" and then "nmap 172.16.60.200" on both a working host, and then this non-working host? Paste the results of both here - if they differ, for instance if the non-working server doesn't see any RPC services running like I suspect, that will heavily point towards a firewall/network/ACL issue

@station-already hehe glad you got it figured out Indeed, management interface changes are pretty "violent" in terms of how much they change and interrupt, so a reboot helps quite often - same for changing the virtual adapter/PIF assigned to a VM - alway a good idea to reboot the VM

@olivierlambert indeed, I believe it's just:

xe host-management-reconfigure pif-uuid=<PIF_UUID>

with the PIF_UUID being the UUID of the new PIF interface (bond, vlan, etc) you would like to move the existing management interface to

@station-already Hi! This can be done in XOA, but it requires deleting the existing management interface, then creating a new one on the new PIF you've created (a vlan 10 on eth0 PIF for example). You say you can't lose access to the host however, so that is probably not an option. XCP-ng center, the windows client, has a bit of a workaround, it has a dropdown box that will allow you to change the PIF of an existing management interface/IP address to a new interface. In XCP-ng center just go to the Networking tab of the host, then hit configure at the bottom beneath the management interfaces section.

It will have a new pop up window with a dropdown for where you want the existing management interface to exist. Assuming you've already created the vlan network you want in xcp-ng, it should show up in this list. just choose it and hit ok:

@garrya Hi, it sounds like you've set up VLAN interfaces correctly on the XCP-ng side, but the switch they are connected to also needs to be aware of VLAN tags to pass them, or else it will only pass untagged traffic as you noticed. You'll need a smart/managed switch with VLAN capability to do so

@slynch OK, that's very good news then! Debian was still having issues where any configuration applied via cloud-init would just be added to the existing default DHCP config, unless you took care to build the VM without the default interfaces config beforehand.

@slynch Interesting! since you left the default DHCP netplan, does the VM also have a DHCP config and grab a DHCP address in addition to your cloud-init config?

Also, you mean when there is netplan content, cloud-init does finally apply netplan apply like it's supposed to? interesting

@slynch that end of the issue sounds like a cloud-init bug/issue, which doesn't surprise me (I've been fighting with them since I started here at Vates). I suppose you could manually work around it, somewhere in the cloud-init options/parameters is optional commands to pass and run in the VM on first run, I suppose you could pass "netplan apply" there.

@s-pam If you don't mind about power usage, high performance (that you've already selected) is the way to go. It will avoid entering C states as often and leave the CPU clocked high among other things so there is no latency for workloads waiting on the CPU existing C states etc.

Leave x2APIC enabled, it helps to distribute interrupts between multiple cores/CPUs. There's no downside to having it enabled unless the hypervisor doesn't support it (and you would know immediately by having errors).

Regarding your last setting, also referred to as CCX as NUMA, this changes how the CPU cores are presented to the hypervisor (either as one NUMDA domain, or multiple, one for each set of cores that share cache). From the benchmarks I can find, leaving this on gains maybe 3 to 4 percent improvement https://blogs.vmware.com/performance/2020/05/app-perf-vmware-vsphere-amd-epyc-rome.html