@brm Hmm, I actually am not sure if we ever added support for this specifically (specifying an IP from IP pools in a cloud-init configuration). I've never seen IP variables used or referenced so I don't think it's currently possible. @olivierlambert who was it on the team that implemented the IP Pools feature?
@MrXeon So, the actual root issue here I believe, is opnsense installs come with an IP and dhcp server already assigned and enabled on the lan interface (I believe it's 192.168.1.1, but don't quote me). If your existing home network already uses 192.168.1.x/24 and already has a dhcp server, booting an opnsense install with it's virtual lan nic set to your existing home lan, there will be a lot of conflicts. Virtual nic order can be whatever you'd like (you can change and move around assignments in opnsense), but if it's preconfigured lan interface gets set to your preexisting lan network, there will be conflicts 
Also note the text at the top of your screenshot: to continue you need to select a boot device. There might be a way in that menu (or partition creation submenu) to mark that created partition as bootable, or maybe you just need to highlight/select the partition under "used devices" before hitting "done"
@encryptblockr yup, welcome to cloud-init hell. Your issue is definitely ubuntu related though, if I had to guess, the installer wants/requires a swap partition. Just create a 1 or 2gb swap partition as well, but put it first in the partition table, so the root partition after it has room to grow. You'll also run into some network issues probably when trying to use your new template, as ubuntu has moved to new netplan crap to manage networking in the OS, and cloud-init has a ton of bugs with it
@furyflash777 I'm assuming you're on Ubuntu? Indeed as Olivier said this is tested on Debian and doesn't cause issues, but it seems on the newer Ubuntu versions with cloud-init, the new Netplan based network manager and how it interacts with cloud-init breaks/gets wiped if no cloud-init drive is found. Yet another cloud-init bug to track down
@jcdick1 I run opnsense on xcp-ng personally as well and use their packaged tools without issue, the only time I've gotten this behavior is when I hot-added interfaces and it changed the order of interfaces. If that's not it, I'm really not sure what would be causing this
One last thing you can try in case it's a weird cash issue is (inside XOA) go to settings > servers, click the green connected button next to your xcp-ng server to disconnect it from xoa - then wait a couple seconds and click it again to reconnect it
@jcdick1 Hi, have you hot-added any new network interfaces to this VM by chance? I've noticed when doing this with *bsd based guests like the *sense projects, the order can get quite messed up, if you've added any new interfaces, changed any MAC addresses, etc, can you please shut the VM down entirely (not just issue a reboot) - once the power state of the VM is completely off, start it again.
Note that if you did hot-add interfaces and hadn't rebooted yet, the interface order will probably change into its "final" order (the ordering seems to be affected when hot adding interfaces, eg when I hot add interfaces into *bsd VMs, sometimes the new interface will show up as xn0 in the VM, so the existing xn0 will get moved to xn3 etc). I've avoided this by just no longer hot adding interfaces and doing it when the VM is off instead
@tekwendell xen carefully manages CPU power management to match VM load and vCPU count, I would not manually try to adjust things with xenpm in the meantime as it's likely you'll make things worse (don't try to outsmart xen power management unless you have a VERY specific use case). Xen is designed for paralleled workloads (more than a single VM), so there's many tunables for VMs that are set with this in mind (like CPU affinity). So by default I'm sure the CPU affinity for your single windows VM is still set somewhere in the "middle", so it's not going to be allowed to schedule the full CPU time versus what dom0 is also using.
I'm not an expert in AMD/Epyc power management, but I believe it's pretty typical that CPU power/clock management boosts based on overall CPU load, and running a benchmark on only a single VM using something like 8 cores on a 64 core processor is not going to demand a lot CPU time, so I'm not surprised to see it's not boosting very far. Spin up 6 more of those VMs and benchmark them all at the same time, I wouldn't be surprised if you see it start boosting higher
475 cpu-z versus 501 bare metal is very good and indicates pretty clearly there's no issue here, you're getting 94% bare metal performance on windows under a large virtualization stack (historically the OS with the most overhead to virtualize). I would be very happy about this
If you really want to dig further, ensure your bios power management is set to "OS-controlled", this will hand more control over turbo and c-states to the xen power manager and is what is recommended on AMD processors, and then you can use some commands listed here to check actual turbo status. But again, note that I won't be surprised if you can't get a 64-core processor to enter its highest turbo states when only stressing 1/10th of its cores: https://support.citrix.com/article/CTX200390/power-settings-in-citrix-hypervisor-cstates-turbo-and-cpu-frequency-scaling
@olivierlambert I vaguely remember @pdonias and I discussing which of these behaviors would be best and we decided on adding it to the smallest matching prefix, I'm not sure why the behavior is the opposite
@jmishal As @tjkreidl says, you can do this quite easily through the management console in your screenshot. But be aware if this is a pool, change the master first, then each slave. This won't affect VM traffic on the network or running VMs.
@encryptblockr The issue is the cloud-init project is so disorganized and is constantly introducing more and more OS-dependent workarounds, bugs, and oddities that any documentation will become useless in a few months. I maintain our XOA Hub cloud-init builds, and I have to personally rewrite my own documentation every single time we push out a new image because cloud-init has broken something new, changed or removed config options without documenting it, or the underlying OS has changed how it deals with networking files etc and cloud-init has not been updated to deal with that. It's a nightmare, and cloud-init is the worst project I've ever had to deal with at Vates.
For debian/ubuntu, this is my rough process currently, but it's not worth putting in an official blog or document guide because it will be rendered useless and just cause people frustration the next release cloud-init puts out, the next debian or ubuntu version change, etc. You'll note none of this is related to XCP-ng at all (except for xen tools obviously), so the below is documentation cloud-init should really be publishing themselves - I have a feeling they also know it would quickly be rendered useless as well so they don't bother. Anyway:
#create a new debian or ubuntu VM in XCP-ng with a ~10gb disk. During the install ISO process, you'll have to choose manual partitioning, and either remove the swap partition completely from the partition layout, or move it to the beginning of the disk. If you choose auto partitioning, it will put a swap partition after the data partition, so the data partition won't be able to be expanded on your template. Once it's installed and running, install xcp-ng guest utilities. To get the latest version, use our guest-utilities ISO. Once all that xen-specific stuff is done, you can start the cloud-init setup: Start with installing cloud-init in the first place:
apt-get install -y cloud-init cloud-utils cloud-initramfs-growroot
Set the root and default user to random passwords, then disable password logins (this is done on our templates so only pubkey based logins are accepted, skip this optionally):
echo 'root:dfgdfgdfg' | chpasswd
echo 'ubuntu:dfgdfg' | chpasswd
passwd -l root
passwd -l ubuntu
nano /etc/ssh/sshd_config #set permitrootlogin to "no"
Clean networking stuff that the VM picked up via dhcp from your current network, they will be re-populated with current info whenever the template is deployed:
nano /etc/resolv.conf #remove all
nano /etc/hosts # remove everything but localhost
Now the important part, set the main cloud init config. Edit /etc/cloud/cloud.cfg - What exactly you need to edit here is impossible to document, as the default values in this file change across every OS, and are changed across every cloud-init version, with no documentation indicating they have done so. The actual behavior of a given option has also been changed out of nowhere. So I'll try to summarize. You want to remove any datasource_list or datasource blocks, and replace them with these values:
datasource_list: [ NoCloud, ConfigDrive ]
datasource:
ConfigDrive:
dsmode: local
NoCloud:
fs_label: cidata
You'll need to find and change (or add) these vars as well so networking is properly handled:
manage_resolv_conf: true
manage_etc_hosts: true
preserve_hostname: false
Also find the default_user block, and change it to whatever username you set up during the install ISO (the user aside from root). On our templates we name this user after the OS, so the non-root user on the ubuntu template is "ubuntu", so that part of the cloud-init config looks like this:
default_user:
name: ubuntu
If your file doesn't have the following somewhere in it, add it (I've had to add it on some builds, on other builds it was magically there already):
users:
- default
disable_root: true
On some older cloud-init versions, I had to manually re-order the module init staging order, because cloud-init could not be bothered to get this right themselves. I think (???) this has been resolved in later builds, but again it changes constantly so there's no way to know. Basically, ensure these three modules are under the first "cloud_init_modules" that run during the init stage, not under the later config or final stage:
- set_hostname
- update_hostname
- update_etc_hosts
Save the config file, and hopefully you're done with that. I can't outline just how unreliable documenting this file is - the defaults, the behavior of certain options, or even the presence of certain options changes entirely across OS's and cloud-init versions, so I can't just keep a "master copy" of the config file and expect it to work. You have to examine the default file you get line by line and work towards the values above until the template works properly. When it stops working properly 3 months later, browse the cloud-init mailing lists and github issues to figure out which option's behavior they changed without any warning, and repeat.
Now remove all the random config files that cloud-init occasionally decides to install that will override your own config and render it useless. This list of files changes whenever cloud-init builds feel like it, so just look in the parent directory to see what's there on your specific cloud-init version:
rm -rf /etc/cloud/cloud.cfg.d/99-installer.cfg /etc/cloud/cloud.cfg.d/90_dpkg.cfg /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg
Remove any stuff the VM picked up that you don't want in all your templates. This also cleans any cloud-init runs that might have occurred if you rebooted the VM, so the template you end up with will be "fresh". We also remove the command history for the root user and default user
rm -rf /etc/ssh/ssh_host_*
cloud-init clean --logs
su - ubuntu
cat /dev/null > ~/.bash_history && history -c && exit
cat /dev/null > ~/.bash_history && history -c && exit
Now you can shut the VM down and convert it into a template. Whenever deploying, it will already have xen guest tools, and a disk that will automatically be expanded to whatever you set the disk size to when deploying. This post will no longer lead to a working cloud-init deployment within 3-6 months based on my previous experience, enjoy it while you can
@ember1205 it's not a matter of not being "well optimized" or xen arbitrarily removing support for old CPUs (we don't do that), it's a matter of the CPUs he's using being so old they're missing entire instruction sets that are required for proper virtualization. The Z800 systems you mention use xeons a full generation newer than the processors he's using, and were the first intel xeon generation to have proper full IOMMU/VT-d/etc instruction set support, so of course I'm sure they worked just great. The Z800 line is the same xeon generation as the 11th gen dells I mentioned, which is why I stated that is the oldest I would comfortably use - any generations older than these (like the poweredges zxcp is using) are missing core instruction sets xen relies upon.
@MilitaryNerd Have you restarted the host since removing this bond? It's reporting that bond0 is still attached to an OVS bridge, so it may need a reboot. You can also ensure the bond is really gone by running xe bond-list, it should return nothing. You could also still have the bond PIF existing or other networks using the old bond, check that you don't have any PIFs with device ( RO): bond0 listed when running xe pif-list
@zxcp PowerEdge 1950 and R900 are far too old for virtualization with xen. This era of Xeons (more than 12 years old) are missing a ton of extensions that help with virtualization, with no IOMMU subsystem to be used by xen etc as you noticed in dmesg (I'm honestly surprised our installer even worked on these). The bare minimum I would consider for virtualization would be 11th gen dell stuff (R610, R710, etc)
@zxcp Something is very broken - with customers and most users we typically see around ~90% of bare metal perfs (in things like passmark and real use). This is with modern virtualization using PVHVM - which is what most of the templates should default to if your hardware supports it. What hardware is this exactly? is it running the latest bios/firmware and you've ensured it has VT-D / IOMMU / etc enabled? In simple cases of poor virtualization overhead we may expect to see something like ~30% worse performance compared to bare metal, so maybe an 8 minute ubuntu install instead of 5. Three hours means something is very broken somewhere - could be storage related as well as installs are usually very write heavy. If you attach a dmesg from XCP-ng it should make it clear what your hardware supports virt wise
@jivanpal I think trying to get OpenVswitch (our underlying network layer) managing and using a wifi dongle for core hypervisor networking would be a nightmare, and you'd probably need to install some extra packages to handle the authentication / WPA management etc - then of course there's zero guarantee OVS wouldn't overwrite or revert any of this custom stuff for said interface (OVS was not designed to work with wifi)
Not to mention you'd have to have a relatively hackable AP, one that will allow the dongle to source a lot of different MAC addresses, which is pretty rare. From OVS's own docu:
@Andi79 the guest RX messages are normal, some OSs like debian etc put unused interfaces in some type of "sleep" mode if I remember correctly and this causes these harmless messages
@Andi79 it's impossible to say without more testing, it could be a firmware issue on these boards causing IO stalls or similar, the only way to be sure now is to run some perf tests inside dom0 to confirm it's not VM related, once you get that confirmation, install (or live boot) plain CentOS on one of the machine and test there as well. Are both the SSD and the HDD underneath an mdadm software raid? I suppose it could be a software raid configuration issue as well, but it's hard to say without further testing
@olivierlambert definitely doesn't ring any bells but definitely smells like a storage issue, I would test disk performance directly in dom0 on the RAID SR using something like FIO, although we don't provide this in our repos so that would be difficult to get in dom0. perhaps a simple dd speed test under the SR mount directly, you should be able to find it by running df -h. Just be sure you don't overwrite any VM disk files