@stormi thanks, will look forward to try this out when the update drops.

@MrXeon So, the actual root issue here I believe, is opnsense installs come with an IP and dhcp server already assigned and enabled on the lan interface (I believe it's 192.168.1.1, but don't quote me). If your existing home network already uses 192.168.1.x/24 and already has a dhcp server, booting an opnsense install with it's virtual lan nic set to your existing home lan, there will be a lot of conflicts. Virtual nic order can be whatever you'd like (you can change and move around assignments in opnsense), but if it's preconfigured lan interface gets set to your preexisting lan network, there will be conflicts 🙂

@gnpcx What did you end up doing for this? The docs dont seem to have any guidance.

Its not solving it, but you can run

echo 3 > /proc/sys/vm/drop_caches

to release some of the cache again, without interfering with running processes.

[root@host2 ~]# free -m
total used free shared buff/cache available
Mem: 15958 3308 158 8 12491 2355
Swap: 1023 177 846
[root@host2 ~]# echo 3 > /proc/sys/vm/drop_caches
[root@host2 ~]# free -m
total used free shared buff/cache available
Mem: 15958 3308 2598 10 10051 2751
Swap: 1023 177 846

Agreed, but the downside is to update the template from time to time. We'll probably create a new Packer plugin to work with XO API, so it will be easier to use and rely on 🙂 (Packer goal is to prep' a template from scratch and up to date for you, that you can use for solution 2. then)

@epretorious I would add that you have to be careful about overprovisioning when NUMA/vNUMA kicks in, that is when you allocate more VCPUs to exceed the number of physical CPUs of a bank of them as well as the associated physical memory (assume, for the sake of argument, you have two banks of physical CPUs and each has directly accessible to it one of two banks of memory) then things get inefficient because a CPU may need to go across to a different bank of memory to access data and there is additional overhead involved. See for example this article and the two preceding it:
https://blogs.mycugc.org/2019/04/30/a-tale-of-two-servers-part-3-the-influence-of-numa-cpus-and-sockets-cores-persocket-plus-other-vm-settings-on-apps-and-gpu-performance/

-=Tobias

@Gheppy

Thank you very much for the quick reply. I thought so indeed. Will this also be the case running Workstation using Linux?

@olivierlambert its looks like it was not from XCP but from the cloud provider managing it (Cloudstack). as soon as max restrictions were removed i could assign all 128 threads as vCPUs to a single VM.

Forget about static min.

This means: start with 1GiB (dynamic min is memory on boot basically). You can balloon up to 4GiB while running (by changing dynamic min to 4GiB), and go down again to 1GiB by set again dynamic min to 1GiB => dynamic is your "live" range.

You can change the dynamic max (while the VM is running) up to 5GiB.

Ballooning will change the amount dynamically (if needed) inside your 1GiB->4GiB range while live.

But frankly, stick with static memory in general (having dynamic min == dynamic max == static max), this will solve a lot of headaches.

Thank you all for this wonderful information.

@Andrew I’ve started to experiment with Duet + OpenCore chain loading but I'm getting the same results…

I’ve tested it now on two different Legacy Machines, so either I’m setting it up wrong or Duet is the issue all around the baord… why could Duet potentially not be happy?

It’s not my target system, it’s gotta be something to do with the file structure or the DuetPkg itself, I've also checked for Debug Logs from OpenCore, seems it ain't booting at all, so it's definitely an issue early on.

@tjkreidl BIOS

@olivierlambert

Unfortunately I've moved VMs around (this is all testing at the moment) however there were other VMs with static=dynamic=2Gb which I could move.

Could you please either explain or point me to the documentation where I can understand how your comment works, please?

Hmm I would also check for BIOS/firmware update

What's the purpose of this other VNC daemon on TCP/9500?

From the Xen Project wiki page on XCP:

The ports used for VNC console are 5901-5999, and the ports used for text console are 9501-9599.

AFAICT: TCP/5900 & TCP/9500 belong to Dom0...

# xenstore-ls /local/domain/0 control = "" feature-poweroff = "1" feature-reboot = "1" feature-suspend = "1" domid = "0" name = "Domain-0" vncterm-pid = "1276" console = "" vnc-port = "5900" tc-port = "9500" vm = "/vm/7a4488c6-d4ac-4aaf-83f0-fb6d3f557f48"

But I don't know how to connect to DomU #1's console because there is no entry in Xenstore for DomU #1:

# xenstore-ls /local/domain/1 xenstore-ls: xs_directory (/local/domain/1): No such file or directory

Ideas? Suggestions?

Eric P.
Reno, Nevada

You could maybe raise the memory a bit, but yeah, the software RAID thing might hurt more than anything else 🤔

Maybe switch to full soft RAID with mdadm, that might be better 🙂