@olivierlambert

iperf (-P8) for VM to VM on Xeon Gold 6138
Before: 25-35 Gbps
After: 25-39 Gbps

Seems slightly higher at best, but hard to measure a difference as the performance tend to vary a lot between runs.

@vkeven
XCP-ng 8.1 release note says

VSS and quiesced snapshots support is removed, because it never worked correctly and caused more harm than good. Note that Windows guest tools version 9 (the default for recent versions of Windows if you install Citrix drivers) already removed VSS support, even for older versions of CH / XCP-ng

I am not sure if this VSS feature is bound to the PV drivers, or if it also needs hypervisor support. Though it is not recommended to stay on a old version of the guest agent.

Hello !

I am looking to get some feedback and evaluation on a performance-related patch for Xen (XCP-ng 8.3 only).
This patch changes the memcpy implementation of Xen to use the "ERMS variant" (aka REP MOVSB) instead of the current REP MOVSQ+B implementation.
This is expected to perform better on the vast majority of Intel CPUs and modern AMD ones (Zen3+), but may perform worse on some older AMD CPUs.

This change may impact the performance of PV drivers (especially network).

You can find more details regarding this proposed change in : https://github.com/xcp-ng-rpms/xen/pull/54
This change may be reworked in the future to take more in account the specificities of each CPUs (e.g check presence of ERMS flag).

Keep in mind that this patched version is experimental and not officially supported.

Installation :

# Download repo file for XCP-ng 8.3
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

# Installing the patched Xen packages (you should see `.erms` packages)
yum update --enablerepo=xcp-ng-tae1

You can revert the changes by downgrading the Xen package with the ones in the default repos.

yum downgrade --disablerepo=xcp-ng-tae1 "xen-*"

TSnake41 opened this pull request in xcp-ng-rpms/xen

draft Use ERMS variant for memcpy #54

@nvs said in XCP-NG server crashes/reboots unexpectedly:

Thanks. Unfortunately my machine doesnt have IPMI. So can I just connect a serial cable between this machine and another machine

Yes though you would still need to boot using the "XCP-ng (Serial)" grub entry.
(you can also add some serial console bits adding them to xen cmdline)

@nvs said in XCP-NG server crashes/reboots unexpectedly:

@nvs Machine crashed/restarted itself again this morning. I didn't even have all of the usual VMs running this time. Nothing was logged in kern.log when it crashed again. Before it crashed I checked a few times in the hours before xl dmesg but nothing obvious to me (same log as I posted above). Any suggestions highly welcome as I'm sure how to proceed with troubleshooting this. My next step would be replacing the PSU and see if anything changes, but its a long shot.

Ok so it doesn't seem caused by a driver bug causing corruption somewhere in Xen/Linux.
So something is causing Xen to crash, and it's not very easy to know without using e.g the serial console (so you can get the actual Xen crash message).
You need for that something connected and monitoring the machine's serial console (or using IPMI) and boot XCP-ng in "(serial)" mode.

@nvs you need to reboot, and it should stick accros reboots

Side question, any chance to tail -f xl dmesg to see real time output? That would allow me to see any last messages before it crashes potentially.

xl dmesg -c allow you to clear the Xen console while displaying it

Can you try adding iommu=strict to Xen command-line ?

/opt/xensource/libexec/xen-cmdline --set-xen "dom0-iommu=strict"

And regularly check if there is something showing up in xl dmesg.

@JeffBerntsen said in Status of PVH:

@TeddyAstie Do you know if these can be set in XO or are they only via command line at this point? I'm curious and also curious as to how well supported because I'm looking at a possible migration from XS 7.1.2 to XCP-ng 8.3 where there are several 32-bit PV machines.

It can be only be set from command-line for now. Though Xen Orchestra should be updated to convert to PV-in-PVH instead of between HVM and PV (this one being unsupported and not bootable anymore).

@sborrill ouch that not practical
The command you provide doesn't work for pv-in-pvh.
A equivalent for the PV-shim would be

xe vm-param-add uuid={UUID} param-name=platform pvinpvh-xen-cmdline="pv-shim console=xen pv-linear-pt=true"

We don't have a usable support for PVH guest at this time, though, for booting a PV guest on XCP-ng 8.3, you can enable PV-shim, e.g

xe vm-param-set uuid={UUID} domain-type=pv-in-pvh

@abudef Note that even with this update, nested virtualization is still not really supported in XCP-ng 8.3.
It's there, you can enable it at your own risk. It broke due to some change in XAPI (even though Xen hypervisor had "support" for it).
It never actually got removed from Xen hypervisor (it was marked experimental in Xen 4.13 used in XCP-ng 8.2, it is also the case for Xen 4.17), although nothing really changed, it still has the same issues and limitations as said.

The current state of nested virtualization in Xen is quite clumsy and there are future plans to remake it properly from ground without taking shortcuts and have proper tests to back it.

Aside that, after some experiments, it seems that mostly nested EPT is incomplete/buggy, so your L1 hypervisor should not rely on it. You should add hap=0 to nested XCP-ng Xen cmdline. Beware that it will imply a pretty large performance hit, but I had more consistent results with this.
I am quite suprised that Windows works while Linux don't, maybe it is somewhat related to PV drivers ?

ping @michael.manley

@mohammadm said in XCP-ng 8.3 & AMD Firepro S7150x2:

Too bad, so at this point we have 4/5 GPU's that are pretty much useless. Any other alternatives for GPU?

When using dGPU, PCI-e passthrough to the VM itself, it does work but after a while the whole screen turn black. They have to disconnect and connect to the RDP session again to see the screen again.

Looks like a screen suspend issue, have you tried to disable it ?

Hello,

You can try to convert the offending VM to HVM mode to avoid having to use PV boot (which is not really supported anymore in XCP-ng 8.3).

Hello @reiichi001

Not sure about USB passthrough, but regarding PCI passthrough, I am not aware of a way to specifically disable PCI passthrough, but there is a way to disable the use of IOMMU (which in fact disable the ability to do PCI Passthrough but may not be possible in combination with future features like PVH Dom0, Host Secure Boot, ...) by adding iommu=no to Xen cmdline in the bootloader.

It could be interesting to report all the information the guest can provide us, but there is no xenstore schema for that; and no support from xcp-rrdd either.

Aside that, I don't think xe-guest-utilities consider cache memory as used; but there may be corner cases.
What gives cat /proc/meminfo

The guest agent is built like any Rust program (so basically getting the toolchain with rustup, then cargo build --release in the guest agent directory).
Currently, there is some dependency on Xen-specific libraries unless https://gitlab.com/xen-project/xen-guest-agent/-/merge_requests/93 is merged (poke @yann).

Alternatively take the Linux x86 64bit executable in https://gitlab.com/xen-project/xen-guest-agent/-/releases (which I assume works on Slackware).

It's something on XO side IIRC.
XO needs to have icons for Slackware as it doesn't have it yet.

For those who have AMD EPYC 7003 (Zen 3 EPYCs), you may find in Processor settings in firmware

• Enhanced REP MOVSB/STOSB (ERMS)
• Fast Short REP MOVSB (FSRM)

Which is apparently disabled by default.
It could be interesting to enable them and see if it changes anything performance-wise. I am not sure if it's just for showing a flag, or if it changes anything in the CPU behavior though.

You can also try REP-MOV/STOS Streaming to see it changes anything too.

@Forza the default one seems cubic which in my testing causes chaotic (either good or bad) network performance on XCP-ng (even on non-EPYC platforms) where BBR is more consistent (and also better on AMD EPYC).

I also wonder if different queuing disciplines (tc qdisc) can help. For example mqprio that spreads packes across the available NIC HW queues?

Regarding PV network, I don't think queue management will change anything as netfront/netback is single-queue. It's multi-queue so maybe it changes something.