@Forza said in Epyc VM to VM networking slow:

olivierlambert said in Epyc VM to VM networking slow:

If we become partners officially, we'll be able to have more advanced accesses with their teams. I still have hope, it's just that the pace isn't on me.

Hi, is there anything new to report on this? We have very powerful machines, but unfortunately limited by this stubborn issue.

Can you test https://xcp-ng.org/forum/topic/10862/early-testable-pvh-support ?

We observe very significant improvements on AMD EPYC with PVH.

We're still pin-pointing the issue with HVM, the current hypothesis is a issue regarding memory typing (grant-table accessed as uncacheable(UC) which is very slow) related to grant-table positionning in HVM.

Hello !

Xen supports 3 virtualization modes, PV (deprecated), HVM (used in XCP-ng) and PVH.
While HVM is supported in XCP-ng (and used), PVH hasn't been integrated yet, but today in XCP-ng 8.3 we have some early support for it.

The PVH mode has been officially introduced in Xen 4.10 as leaner, simpler variant of HVM (it was initially named HVM-lite) with little to no emulation, only PV devices, and less overall complexity.
It aims to be a great and simpler alternative to traditional HVM for modern guests.

A quick comparison of all modes
PV mode :

• needs specific guest support
• only PV devices (no legacy hardware)
• relies on PV MMU (less efficient than VT-x EPT/AMD-V NPT overall, but works without virtualization technologies)
• unsafe against Spectre-style attacks
• supports: direct kernel boot, pygrub
• deprecated

HVM mode :

• emulate a real-behaving machine (using QEMU)
  • including legacy platform hardware (IOAPIC, HPET, PIT, PIC, ...)
  • including (maybe legacy) I/O hardware (network card, storage ...)
  • some can be disabled by the guest (PVHVM), but they exist at the start of the guest
• relies on VT-x/AMD-V
• traditional PC boot flow (BIOS/UEFI)
• optional PV devices (opt-in by guest; PVHVM)
• performs better than PV mode on most machines
• compatible with pretty much all guests (including Windows and legacy OS)

PVH mode :

• relies on VT-x/AMD-V (regarding that, on the Xen side, it's using the same code as HVM)
• minimal emulation (e.g no QEMU), way simpler overall, lower overhead
• only PV devices
• support : direct kernel boot (like PV), PVH-GRUB, or UEFI boot (PVH-OVMF)
• needs guest support (but much less intrusive than PV)
• works with most Linux distros and most BSD; doesn't work with Windows (yet)

Installation

Keep in mind that this is very experimental and not officially supported.

PVH vncterm patches (optional)

While XCP-ng 8.3 actually has support for PVH, due to a XAPI bug, you will not be able to access the guest console. I provide a patched XAPI with a patched console.

# Download repo file for XCP-ng 8.3
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

# You may need to update to testing repositories.
yum update --enablerepo=xcp-ng-testing

# Installing the patched XAPI packages (you should see `.pvh` XAPI packages)
yum update --enablerepo=xcp-ng-tae2

This is optional, but you probably want that to see what's going on in your guest without having to rely on SSH or xl console.

Making/converting into a PVH guest

You can convert any guest into a PVH guest by modifying its domain-type parameter.

xe vm-param-set uuid={UUID} domain-type=pvh

And revert this change by changing it back to HVM

xe vm-param-set uuid={UUID} domain-type=hvm

PVH OVMF (boot using UEFI)

You also need a PVH-specific OVMF build that can be used to boot the guest in UEFI mode.

Currently, there is no package available for getting it, but I provide a custom-built OVMF with PVH support
https://nextcloud.vates.tech/index.php/s/L8a4meCLp8aZnGZ

You need to place this file in the host as /var/lib/xcp/guest/pvh-ovmf.elf (create all missing parents).
Then sets it as PV-kernel

xe vm-param-set uuid={UUID} PV-kernel=/var/lib/xcp/guest/pvh-ovmf.elf

Once done, you can boot your guest as usual.

Tested guests

On many Linux distros, you need to add console=hvc0 in the cmdline, otherwise, you may not have access to a PV console.

• Alpine Linux
• Debian

Known limitations

• Some stats shows "no stats" (XAPI bug ?)
• No support for booting from ISO, you can workaround this by importing your iso as a disk and using it as read-only disk
• No live migration support (or at least, don't expect it to work properly)
• No PCI passthrough support
• No actual display (only PV console)

Hello,
Unfortunately, the current approach of ballooning (dynamic memory) cannot work with PCI passthrough. I don't think it is possible to workaround that limitation (at least not in XCP-ng 8.3).

If I adjust dynamic to be 48 GiB/48 GiB the machine will then boot. Once booted, I can then once again apply the desired dynamic config of 16 GiB/48 GiB.

Am I misunderstanding the configuration options and this is just not supported, or have I stumbled across a bug?

What's probably happening is that the dynamic configuration you set is not effective yet, and only applies when you reboot, that's why you got PCI passthrough work because you actually used static memory allocation.

@olivierlambert

iperf (-P8) for VM to VM on Xeon Gold 6138
Before: 25-35 Gbps
After: 25-39 Gbps

Seems slightly higher at best, but hard to measure a difference as the performance tend to vary a lot between runs.

@vkeven
XCP-ng 8.1 release note says

VSS and quiesced snapshots support is removed, because it never worked correctly and caused more harm than good. Note that Windows guest tools version 9 (the default for recent versions of Windows if you install Citrix drivers) already removed VSS support, even for older versions of CH / XCP-ng

I am not sure if this VSS feature is bound to the PV drivers, or if it also needs hypervisor support. Though it is not recommended to stay on a old version of the guest agent.

Hello !

I am looking to get some feedback and evaluation on a performance-related patch for Xen (XCP-ng 8.3 only).
This patch changes the memcpy implementation of Xen to use the "ERMS variant" (aka REP MOVSB) instead of the current REP MOVSQ+B implementation.
This is expected to perform better on the vast majority of Intel CPUs and modern AMD ones (Zen3+), but may perform worse on some older AMD CPUs.

This change may impact the performance of PV drivers (especially network).

You can find more details regarding this proposed change in : https://github.com/xcp-ng-rpms/xen/pull/54
This change may be reworked in the future to take more in account the specificities of each CPUs (e.g check presence of ERMS flag).

Keep in mind that this patched version is experimental and not officially supported.

Installation :

# Download repo file for XCP-ng 8.3
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

# Installing the patched Xen packages (you should see `.erms` packages)
yum update --enablerepo=xcp-ng-tae1

You can revert the changes by downgrading the Xen package with the ones in the default repos.

yum downgrade --disablerepo=xcp-ng-tae1 "xen-*"

@nvs said in XCP-NG server crashes/reboots unexpectedly:

Thanks. Unfortunately my machine doesnt have IPMI. So can I just connect a serial cable between this machine and another machine

Yes though you would still need to boot using the "XCP-ng (Serial)" grub entry.
(you can also add some serial console bits adding them to xen cmdline)

@nvs said in XCP-NG server crashes/reboots unexpectedly:

@nvs Machine crashed/restarted itself again this morning. I didn't even have all of the usual VMs running this time. Nothing was logged in kern.log when it crashed again. Before it crashed I checked a few times in the hours before xl dmesg but nothing obvious to me (same log as I posted above). Any suggestions highly welcome as I'm sure how to proceed with troubleshooting this. My next step would be replacing the PSU and see if anything changes, but its a long shot.

Ok so it doesn't seem caused by a driver bug causing corruption somewhere in Xen/Linux.
So something is causing Xen to crash, and it's not very easy to know without using e.g the serial console (so you can get the actual Xen crash message).
You need for that something connected and monitoring the machine's serial console (or using IPMI) and boot XCP-ng in "(serial)" mode.

@nvs you need to reboot, and it should stick accros reboots

Side question, any chance to tail -f xl dmesg to see real time output? That would allow me to see any last messages before it crashes potentially.

xl dmesg -c allow you to clear the Xen console while displaying it

Can you try adding iommu=strict to Xen command-line ?

/opt/xensource/libexec/xen-cmdline --set-xen "dom0-iommu=strict"

And regularly check if there is something showing up in xl dmesg.

@JeffBerntsen said in Status of PVH:

@TeddyAstie Do you know if these can be set in XO or are they only via command line at this point? I'm curious and also curious as to how well supported because I'm looking at a possible migration from XS 7.1.2 to XCP-ng 8.3 where there are several 32-bit PV machines.

It can be only be set from command-line for now. Though Xen Orchestra should be updated to convert to PV-in-PVH instead of between HVM and PV (this one being unsupported and not bootable anymore).

@sborrill ouch that not practical
The command you provide doesn't work for pv-in-pvh.
A equivalent for the PV-shim would be

xe vm-param-add uuid={UUID} param-name=platform pvinpvh-xen-cmdline="pv-shim console=xen pv-linear-pt=true"

We don't have a usable support for PVH guest at this time, though, for booting a PV guest on XCP-ng 8.3, you can enable PV-shim, e.g

xe vm-param-set uuid={UUID} domain-type=pv-in-pvh

@abudef Note that even with this update, nested virtualization is still not really supported in XCP-ng 8.3.
It's there, you can enable it at your own risk. It broke due to some change in XAPI (even though Xen hypervisor had "support" for it).
It never actually got removed from Xen hypervisor (it was marked experimental in Xen 4.13 used in XCP-ng 8.2, it is also the case for Xen 4.17), although nothing really changed, it still has the same issues and limitations as said.

The current state of nested virtualization in Xen is quite clumsy and there are future plans to remake it properly from ground without taking shortcuts and have proper tests to back it.

Aside that, after some experiments, it seems that mostly nested EPT is incomplete/buggy, so your L1 hypervisor should not rely on it. You should add hap=0 to nested XCP-ng Xen cmdline. Beware that it will imply a pretty large performance hit, but I had more consistent results with this.
I am quite suprised that Windows works while Linux don't, maybe it is somewhat related to PV drivers ?

ping @michael.manley

@mohammadm said in XCP-ng 8.3 & AMD Firepro S7150x2:

Too bad, so at this point we have 4/5 GPU's that are pretty much useless. Any other alternatives for GPU?

When using dGPU, PCI-e passthrough to the VM itself, it does work but after a while the whole screen turn black. They have to disconnect and connect to the RDP session again to see the screen again.

Looks like a screen suspend issue, have you tried to disable it ?

Hello,

You can try to convert the offending VM to HVM mode to avoid having to use PV boot (which is not really supported anymore in XCP-ng 8.3).

Hello @reiichi001

Not sure about USB passthrough, but regarding PCI passthrough, I am not aware of a way to specifically disable PCI passthrough, but there is a way to disable the use of IOMMU (which in fact disable the ability to do PCI Passthrough but may not be possible in combination with future features like PVH Dom0, Host Secure Boot, ...) by adding iommu=no to Xen cmdline in the bootloader.

It could be interesting to report all the information the guest can provide us, but there is no xenstore schema for that; and no support from xcp-rrdd either.

Aside that, I don't think xe-guest-utilities consider cache memory as used; but there may be corner cases.
What gives cat /proc/meminfo

The guest agent is built like any Rust program (so basically getting the toolchain with rustup, then cargo build --release in the guest agent directory).
Currently, there is some dependency on Xen-specific libraries unless https://gitlab.com/xen-project/xen-guest-agent/-/merge_requests/93 is merged (poke @yann).

Alternatively take the Linux x86 64bit executable in https://gitlab.com/xen-project/xen-guest-agent/-/releases (which I assume works on Slackware).