@abudef Note that even with this update, nested virtualization is still not really supported in XCP-ng 8.3.
It's there, you can enable it at your own risk. It broke due to some change in XAPI (even though Xen hypervisor had "support" for it).
It never actually got removed from Xen hypervisor (it was marked experimental in Xen 4.13 used in XCP-ng 8.2, it is also the case for Xen 4.17), although nothing really changed, it still has the same issues and limitations as said.

The current state of nested virtualization in Xen is quite clumsy and future plans are to remake properly it from ground without taking shortcuts and have proper tests to back it.

Aside that, after some experiments, it seems that mostly nested EPT is incomplete/buggy, so your L1 hypervisor should not rely on it. You should add hap=0 to nested XCP-ng Xen cmdline. Beware that it will imply a pretty large performance hit, but I had more consistent results with this.
I am quite suprised that Windows works while Linux don't, maybe it is somewhat related to PV drivers ?

ping @michael.manley

@mohammadm said in XCP-ng 8.3 & AMD Firepro S7150x2:

Too bad, so at this point we have 4/5 GPU's that are pretty much useless. Any other alternatives for GPU?

When using dGPU, PCI-e passthrough to the VM itself, it does work but after a while the whole screen turn black. They have to disconnect and connect to the RDP session again to see the screen again.

Looks like a screen suspend issue, have you tried to disable it ?

Hello,

You can try to convert the offending VM to HVM mode to avoid having to use PV boot (which is not really supported anymore in XCP-ng 8.3).

Hello @reiichi001

Not sure about USB passthrough, but regarding PCI passthrough, I am not aware of a way to specifically disable PCI passthrough, but there is a way to disable the use of IOMMU (which in fact disable the ability to do PCI Passthrough but may not be possible in combination with future features like PVH Dom0, Host Secure Boot, ...) by adding iommu=no to Xen cmdline in the bootloader.

It could be interesting to report all the information the guest can provide us, but there is no xenstore schema for that; and no support from xcp-rrdd either.

Aside that, I don't think xe-guest-utilities consider cache memory as used; but there may be corner cases.
What gives cat /proc/meminfo

The guest agent is built like any Rust program (so basically getting the toolchain with rustup, then cargo build --release in the guest agent directory).
Currently, there is some dependency on Xen-specific libraries unless https://gitlab.com/xen-project/xen-guest-agent/-/merge_requests/93 is merged (poke @yann).

Alternatively take the Linux x86 64bit executable in https://gitlab.com/xen-project/xen-guest-agent/-/releases (which I assume works on Slackware).

It's something on XO side IIRC.
XO needs to have icons for Slackware as it doesn't have it yet.

For those who have AMD EPYC 7003 (Zen 3 EPYCs), you may find in Processor settings in firmware

• Enhanced REP MOVSB/STOSB (ERMS)
• Fast Short REP MOVSB (FSRM)

Which is apparently disabled by default.
It could be interesting to enable them and see if it changes anything performance-wise. I am not sure if it's just for showing a flag, or if it changes anything in the CPU behavior though.

You can also try REP-MOV/STOS Streaming to see it changes anything too.

@Forza the default one seems cubic which in my testing causes chaotic (either good or bad) network performance on XCP-ng (even on non-EPYC platforms) where BBR is more consistent (and also better on AMD EPYC).

I also wonder if different queuing disciplines (tc qdisc) can help. For example mqprio that spreads packes across the available NIC HW queues?

Regarding PV network, I don't think queue management will change anything as netfront/netback is single-queue. It's multi-queue so maybe it changes something.

Is there a difference when running iperf3 with -C bbr flag on the client side ?
In my testing with AMD EPYC and some other CPUs, results are more consistent overall with BBR, and better on the AMD EPYC side (but no miracle, it's still far from perfect).

AMD EPYC 7262
vm to vm, 4 threads, iperf3
Without BBR : 4.5-6 Gbps (sometimes more; varies a lot)
With BBR : 7-8 Gbps

@Mefosheez So it is not concerned with (potential) PML5e issues.

Have you tried rebooting the host ?

@Mefosheez Looks like a OVMF bug to me (issue with PML5e ?).

What CPU are you using ?

@plaidypus I don't know a lot about NUMA on Xen, but we have a part in the docs regarding that
https://docs.xcp-ng.org/compute/#numa-affinity

And also other documentation on the subject
https://xapi-project.github.io/new-docs/toolstack/features/NUMA/index.html
there was a design session regarding NUMA in latest Xen Summit : https://youtu.be/KoNwEYMlhyU?list=PLQMQQsKgvLnvjRgDnb-5T51e1kGHgs1SO

@Buckle8711 Hello,

I took a look but unfortunately, I haven't found a workaround for this.
I suspect the Illumos/OmniOS Xen PV drivers to be too old to work in current Xen, but we currently don't have a way to prevent those drivers from loading (e.g HVM-only (non-PVHVM) domains).

@Andrew said in OmniOS / Illumos / Solaris as a guest - not working:

@olivierlambert XCP/Xen/Qemu expert question: Why does qemu start with only two interfaces when more are configured in XO?
qemu-dm-7 -machine pc-i440fx-2.10,accel=xen ... -device e1000,netdev=tapnet1,mac=5a:7f:c3:ba:04:d6,addr=5,rombar=0 -netdev tap,id=tapnet1,fd=7 -device e1000,netdev=tapnet0,mac=8e:68:02:55:da:c7,addr=4,rombar=0 -netdev tap,id=tapnet0,fd=8 ...
Where's the third and fourth? I see other VMs with more. Is it a template or vm-param issue?

These are the same "interfaces" just plugged differently to the guest (e.g tapnet0 exposed as e1000 and "tap" and tapnet1 exposed as e1000 and "tap" too).

You have 4 interfaces configured ?

PCIe AER needs proper PCIe, which in practice needs Q35 chipset in the guest (or some other guest type/PCI passthrough way).

Q35 support is currently work in progress

@brodiecyber said in Question on CPU masking with qemu and xen:

Hello all

I have a question but dot know enough about Xen hypervisor to answer it.
So it simple why cant XCP-ng mask the type of CPU that the vms running on.
Take for example the KVM on proxmox you can set host pass-through to send all CPU functions to the vm or mask it to an intel Xeon scalable or AMD Epyc.
I see that host pass-through is what XCP-ng does but was wondering why.

It doesn't exactly work this way.
With XCP-ng, it works at pool level, it basically takes the lowest CPU feature level of your hosts of the pool, and sets all VM to use it. That way you should always be able to migrate to another machine of the same pool, while not having to manually set the feature levels to improve performance.

Doesnt masking the cpu make the vm more portable as the profile can be set to for example QEMU-xen or something making it possible to say migrate a vm live from AMD to intel n different pools or a heterogeneous cluster.

In practice, you can't reliably live migrate between Intel and AMD architecture (even with QEMU) due to various subtle architecture differences. It may somewhat work with luck if the guest is using a ancient processing model (like 32-bits) but that's not really something we support.

And one last thing if it is possible is there no way to emulate other architecture supported by qemu such as arm64 or does xen hypervisor still need more features built out on arm before that i possible. Asking because my use case is i cant afford a raspberry pi but have alot of compute on x86 why not emulate arm 64 on the hypervisor rather than in a vm

Emulating a completely different CPU architecture while technically possible, is a completely different story.

@tuxen said (https://xcp-ng.org/forum/topic/3652/no-free-virtual-function-found-vgpu-s7150/4?_=1731502751059)

After some digging, could be the case of a GPU firmware being incompatible with UEFI. Do you have any spare server for testing XCP-ng boot in legacy/BIOS with this GPU?

Perhaps it is the issue ?

@ohajek

Nov 13 11:30:21 xen03 kernel: [10188.720655] AMD IOMMUv2 driver by Joerg Roedel jroedel@suse.de
Nov 13 11:30:21 xen03 kernel: [10188.720656] AMD IOMMUv2 functionality not available on this system

This is expected, Dom0 Kernel (Linux) is not supposed to access the IOMMU when it is already used by Xen. To check if AMD-Vi is working, you need to check xl dmesg instead.

I took a quick look at kern_gim_compiled.txt, and it look likes it timed-out somewhere

Oct 23 20:49:32 xen03 kernel: [   80.657394]        gim error:(wait_cmd_complete:2387)  wait_cmd_complete -- time out after 0.003004460 sec
Oct 23 20:49:32 xen03 kernel: [   80.657408]        gim error:(wait_cmd_complete:2390)   Cmd = 0x17, Status = 0x0, cmd_Complete=0

3ms looks like a short timeout for me, but aside that, it looks like a driver(gim) or hardware issue