@olivierlambert ah good to know 😄 thats fine I'll take a look at the metrics maybe I'll write a small rrd stats pusher script as an interim solution for my usecase 🙂 until XO6 is available.

Thanks for letting me know 🙂

@stormi thanks, will look forward to try this out when the update drops.

@MrXeon So, the actual root issue here I believe, is opnsense installs come with an IP and dhcp server already assigned and enabled on the lan interface (I believe it's 192.168.1.1, but don't quote me). If your existing home network already uses 192.168.1.x/24 and already has a dhcp server, booting an opnsense install with it's virtual lan nic set to your existing home lan, there will be a lot of conflicts. Virtual nic order can be whatever you'd like (you can change and move around assignments in opnsense), but if it's preconfigured lan interface gets set to your preexisting lan network, there will be conflicts 🙂

@gnpcx What did you end up doing for this? The docs dont seem to have any guidance.

Its not solving it, but you can run

echo 3 > /proc/sys/vm/drop_caches

to release some of the cache again, without interfering with running processes.

[root@host2 ~]# free -m
total used free shared buff/cache available
Mem: 15958 3308 158 8 12491 2355
Swap: 1023 177 846
[root@host2 ~]# echo 3 > /proc/sys/vm/drop_caches
[root@host2 ~]# free -m
total used free shared buff/cache available
Mem: 15958 3308 2598 10 10051 2751
Swap: 1023 177 846

Hello @exime,

Here we have the EPT violation (write access to the r-x page) at 0x3fff2046. This address is tagged as an MMIO address, so very probably belongs to the device you're trying to passthrough.

Normally this has nothing to do with the 0x46xxx range (where MSI-X caps are pointing)? But the fact that there's some hacking in there make me think that Google engs also did some hacking all the place around.

Can you please while starting a native guest (or in dom0 before the passthrough) give a PCI dump for your device

lspci -vvv -s $YOUR_DEV_BDF

YOUR_DEV_BDF is your device PCI id (ex: 00:1:0)

Agreed, but the downside is to update the template from time to time. We'll probably create a new Packer plugin to work with XO API, so it will be easier to use and rely on 🙂 (Packer goal is to prep' a template from scratch and up to date for you, that you can use for solution 2. then)

@epretorious I would add that you have to be careful about overprovisioning when NUMA/vNUMA kicks in, that is when you allocate more VCPUs to exceed the number of physical CPUs of a bank of them as well as the associated physical memory (assume, for the sake of argument, you have two banks of physical CPUs and each has directly accessible to it one of two banks of memory) then things get inefficient because a CPU may need to go across to a different bank of memory to access data and there is additional overhead involved. See for example this article and the two preceding it:
https://blogs.mycugc.org/2019/04/30/a-tale-of-two-servers-part-3-the-influence-of-numa-cpus-and-sockets-cores-persocket-plus-other-vm-settings-on-apps-and-gpu-performance/

-=Tobias

@Gheppy

Thank you very much for the quick reply. I thought so indeed. Will this also be the case running Workstation using Linux?

@olivierlambert its looks like it was not from XCP but from the cloud provider managing it (Cloudstack). as soon as max restrictions were removed i could assign all 128 threads as vCPUs to a single VM.

Forget about static min.

This means: start with 1GiB (dynamic min is memory on boot basically). You can balloon up to 4GiB while running (by changing dynamic min to 4GiB), and go down again to 1GiB by set again dynamic min to 1GiB => dynamic is your "live" range.

You can change the dynamic max (while the VM is running) up to 5GiB.

Ballooning will change the amount dynamically (if needed) inside your 1GiB->4GiB range while live.

But frankly, stick with static memory in general (having dynamic min == dynamic max == static max), this will solve a lot of headaches.

Thank you all for this wonderful information.

@Andrew I’ve started to experiment with Duet + OpenCore chain loading but I'm getting the same results…

I’ve tested it now on two different Legacy Machines, so either I’m setting it up wrong or Duet is the issue all around the baord… why could Duet potentially not be happy?

It’s not my target system, it’s gotta be something to do with the file structure or the DuetPkg itself, I've also checked for Debug Logs from OpenCore, seems it ain't booting at all, so it's definitely an issue early on.

@tjkreidl BIOS

@olivierlambert

Unfortunately I've moved VMs around (this is all testing at the moment) however there were other VMs with static=dynamic=2Gb which I could move.

Could you please either explain or point me to the documentation where I can understand how your comment works, please?

Hmm I would also check for BIOS/firmware update