16 EIB is pretty close (1 byte close) to 18446744073709551615 bytes, which is the maximum representable 64-bits number.

Unikraft would a a good fit for ram-constained devices.
Being able to have useful VMs with 32-64 MB each.

@deefdragon said in VM UUID via dmidecode does not match VM ID in xen-orchestra:

Out of curiosity, I dumped the DMI into a bin and opened it up in a hex editor.

I am seeing ASCII of the ID, but also a variant encoded in binary. In both cases, its formatted as 0b08f477-491a-a982-23c4-d224723624ea.

I believe the ASCII version is the one that gets populated into the serial number as it comes after ASCII encoded versions of the 3 lines above it in the decode.

In SMBIOS 2.8, the UUID is supposed to be encoded in little endian (i.e Microsoft GUID). Yet it is put as big endian instead. So when Linux generates the UUID string from the SMBIOS table, it is considered as little endian which causes this mismatch.

SMBIOS 2.4 is supposed to be used (which appears to be using big endian UUIDs), but for some reason, something in XCP-ng UEFI supports forces it to be SMBIOS 2.8.

So the binary UUID is the same, just that it is interpreted with a different endianness due to accidental format change.

@lovvel from a software standpoint, this is a 16 cores CPUs and AFAICT, Xen doesn't account for these slight differences between cores.

As to be fair, it's not really easy to know in practice if a 3D-VCache core will be faster than a non-3D-VCache one for a specific case.

@deefdragon can you provide us the output of dmidecode in the guest ?

deef@k31-w-3bfbbe:~$ sudo cat /sys/devices/virtual/dmi/id/product_serial
0b08f477-491a-a982-23c4-d224723624ea
deef@k31-w-3bfbbe:~$ sudo cat /sys/devices/virtual/dmi/id/product_uuid
77f4080b-1a49-82a9-23c4-d224723624ea
deef@k31-w-3bfbbe:~$ sudo cat /sys/hypervisor/uuid
0b08f477-491a-a982-23c4-d224723624ea

It looks like a endianness issue (0b08f477 vs 77f4080b).

@hoh said in Early testable PVH support:

Then I tried to get rid of the (fake) UEFI magic.

Well, it is actually a full standard UEFI implementation, but that works in PVH instead of HVM.

I thought it should work to just change the PV-bootloader to pygrub. Calling pygrub on the disk image works fine and is able to extract the images and args

# pygrub -l alpine.img
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
title: Alpine, with Linux virt
  root: None
  kernel: /boot/vmlinuz-virt
  args: root=UUID=4c6dcb06-20ff-4bcf-be4d-cb399244c4c6 ro  rootfstype=ext4 console=hvc0
  initrd: /boot/initramfs-virt

But starting the VM fails. It looks like it starts but then immediately something calls force shutdown, I'll dive deeper into the logs later.

But setting everything manually actually works. If extract the kernel and initrd to dom-0 and configure

PV-kernel=/var/lib/xcp/guest/kernel
PV-ramdisk=/var/lib/xcp/guest/ramdisk
PV-args="root=/dev/xvda1 ro rootfstype=ext4 console=hvc0"

it boots and I looks pretty much the same as with the pvh-ovmf magic. So perhaps the idea to use pygrub is wrong.

I don't know how good is supported pygrub nowadays, especially since PV support got deprecated in XCP-ng 8.2 then completely dropped in XCP-ng 8.3; with the pv-shim (pv-in-pvh) being the only remaining (but not endorsed) way of booting some PV guests today.

In my tests, pygrub was very clunky and rarely work as I expect. In practice (what upstream Xen Project mostly uses), it got replaced with pvgrub/pvhgrub and pvh-ovmf (OvmfXen) which are more reliable and less problematic security-wise (runs in the guest rather than in the dom0).
(for using pvhgrub, you need to set a pvhgrub binary (grub-i386-xen_pvh.bin which is packaged by some distros like Alpine Linux's grub-xenhost) as kernel like done with pvh-ovmf)

@hoh said in Early testable PVH support:

@TeddyAstie said in Early testable PVH support:

... PV-kernel=/var/lib/xcp/guest/pvh-ovmf.elf

Works fine. But IIUC, direct kernel boot should work as well. I tried setting pygrub, the VM loads the kernel and starts but then immediately stops. Any idea what's wrong?

What are you trying to boot ?

cc @andrew

It looks like an issue with https://github.com/xcp-ng-rpms/r8125-module, though I am not completely sure what is going on, and why the pagetable suddently gets invalid.

@gb.123 said in XCP-ng 8.3 updates announcements and testing:

Here is the summary:

If USB Keyboard & Mouse is passed-through along-with GPU:
The GPU gets stuck in D3 state (on Shutdown/Restart of VM) (Classic GPU reset problem)

If no vUSB is passed but GPU is passed through:
The GPU works correctly and resets correctly (on Shutdown/Restart of VM)

I have no clue what vUSB may change regarding GPU passthrough.

When I run :

$> lspci
Extract of Output (Partial):

07:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8

However, this controller does not show up when I run :
xe pci-list

Is it a bug that lspci & xe pci-list have different number of devices ?

How can I pass this controller since xe pci-list does not show it so I can't get the UUID ?
Will kernel parameters (like XCP-ng 8.2) work in this case ?

Question for @Team-XAPI-Network regarding the filtering on PCI IDs.
I don't think XAPI allows using arbitrary BDF, but I may be wrong.

Is it safe to run on XCP-ng host ?

 echo 1 > /sys/bus/pci/rescan

(I'm trying to find a way where the PCI card is reset by the host without complete reboot, though I am aware that the above command will not reset it.)

Probably. But it's not going to change anything as the device doesn't completely leave the Dom0 when passed-through.
FYI a function-level-reset is systematically performed by Xen when doing PCI passthrough, thus your device should be reset before entering another guest (aside reset bugs like you may have).

Also is it advisable to use :

xl pci-assignable-add 07:00.0

in XCP-ng 8.3 ? or is this method deprecated ?

I don't think XAPI supports this PCI passthrough approach.
This is a command which allows dynamically to remove a device from Dom0 and put it into "quarantine domain", so that it will be ready to passthrough it.

Current XAPI uses the approach of having a set of "passthrough-able" devices at boot time by modifying the xen-pciback.hide kernel parameter, which does the same but at boot time.

Not a Xen issue.
This seems to be either a configuration issue (knowing /opt/xensource/libexec/xen-cmdline --get-dom0 may help) causing a issue in XAPI (@Team-XAPI-Network).

Maybe crashing in xapi/pciops.ml#L71-L80 or xapi/xapi_pci_helpers.ml#L179-L207.

@Fionn with NIC passthrough, the network card is fully controlled by the guest, so the host cannot do anything with it anymore.

If you need to setup something for this network card (e.g MAC spoofing), it has to be done from within the guest.

@Forza said in Epyc VM to VM networking slow:

olivierlambert said in Epyc VM to VM networking slow:

If we become partners officially, we'll be able to have more advanced accesses with their teams. I still have hope, it's just that the pace isn't on me.

Hi, is there anything new to report on this? We have very powerful machines, but unfortunately limited by this stubborn issue.

Can you test https://xcp-ng.org/forum/topic/10862/early-testable-pvh-support ?

We observe very significant improvements on AMD EPYC with PVH.

We're still pin-pointing the issue with HVM, the current hypothesis is a issue regarding memory typing (grant-table accessed as uncacheable(UC) which is very slow) related to grant-table positionning in HVM.

Hello !

Xen supports 3 virtualization modes, PV (deprecated), HVM (used in XCP-ng) and PVH.
While HVM is supported in XCP-ng (and used), PVH hasn't been integrated yet, but today in XCP-ng 8.3 we have some early support for it.

The PVH mode has been officially introduced in Xen 4.10 as leaner, simpler variant of HVM (it was initially named HVM-lite) with little to no emulation, only PV devices, and less overall complexity.
It aims to be a great and simpler alternative to traditional HVM for modern guests.

A quick comparison of all modes
PV mode :

• needs specific guest support
• only PV devices (no legacy hardware)
• relies on PV MMU (less efficient than VT-x EPT/AMD-V NPT overall, but works without virtualization technologies)
• unsafe against Spectre-style attacks
• supports: direct kernel boot, pygrub
• deprecated

HVM mode :

• emulate a real-behaving machine (using QEMU)
  • including legacy platform hardware (IOAPIC, HPET, PIT, PIC, ...)
  • including (maybe legacy) I/O hardware (network card, storage ...)
  • some can be disabled by the guest (PVHVM), but they exist at the start of the guest
• relies on VT-x/AMD-V
• traditional PC boot flow (BIOS/UEFI)
• optional PV devices (opt-in by guest; PVHVM)
• performs better than PV mode on most machines
• compatible with pretty much all guests (including Windows and legacy OS)

PVH mode :

• relies on VT-x/AMD-V (regarding that, on the Xen side, it's using the same code as HVM)
• minimal emulation (e.g no QEMU), way simpler overall, lower overhead
• only PV devices
• support : direct kernel boot (like PV), PVH-GRUB, or UEFI boot (PVH-OVMF)
• needs guest support (but much less intrusive than PV)
• works with most Linux distros and most BSD; doesn't work with Windows (yet)

Installation

Keep in mind that this is very experimental and not officially supported.

PVH vncterm patches (optional)

While XCP-ng 8.3 actually has support for PVH, due to a XAPI bug, you will not be able to access the guest console. I provide a patched XAPI with a patched console.

# Download repo file for XCP-ng 8.3
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

# You may need to update to testing repositories.
yum update --enablerepo=xcp-ng-testing

# Installing the patched XAPI packages (you should see `.pvh` XAPI packages)
yum update --enablerepo=xcp-ng-tae2

This is optional, but you probably want that to see what's going on in your guest without having to rely on SSH or xl console.

Making/converting into a PVH guest

You can convert any guest into a PVH guest by modifying its domain-type parameter.

xe vm-param-set uuid={UUID} domain-type=pvh

And revert this change by changing it back to HVM

xe vm-param-set uuid={UUID} domain-type=hvm

PVH OVMF (boot using UEFI)

You also need a PVH-specific OVMF build that can be used to boot the guest in UEFI mode.

Currently, there is no package available for getting it, but I provide a custom-built OVMF with PVH support
https://nextcloud.vates.tech/index.php/s/L8a4meCLp8aZnGZ

You need to place this file in the host as /var/lib/xcp/guest/pvh-ovmf.elf (create all missing parents).
Then sets it as PV-kernel

xe vm-param-set uuid={UUID} PV-kernel=/var/lib/xcp/guest/pvh-ovmf.elf

Once done, you can boot your guest as usual.

Tested guests

On many Linux distros, you need to add console=hvc0 in the cmdline, otherwise, you may not have access to a PV console.

• Alpine Linux
• Debian

Known limitations

• Some stats shows "no stats" (XAPI bug ?)
• No support for booting from ISO, you can workaround this by importing your iso as a disk and using it as read-only disk
• No live migration support (or at least, don't expect it to work properly)
• No PCI passthrough support
• No actual display (only PV console)

Hello,
Unfortunately, the current approach of ballooning (dynamic memory) cannot work with PCI passthrough. I don't think it is possible to workaround that limitation (at least not in XCP-ng 8.3).

If I adjust dynamic to be 48 GiB/48 GiB the machine will then boot. Once booted, I can then once again apply the desired dynamic config of 16 GiB/48 GiB.

Am I misunderstanding the configuration options and this is just not supported, or have I stumbled across a bug?

What's probably happening is that the dynamic configuration you set is not effective yet, and only applies when you reboot, that's why you got PCI passthrough work because you actually used static memory allocation.

@olivierlambert

iperf (-P8) for VM to VM on Xeon Gold 6138
Before: 25-35 Gbps
After: 25-39 Gbps

Seems slightly higher at best, but hard to measure a difference as the performance tend to vary a lot between runs.

@vkeven
XCP-ng 8.1 release note says

VSS and quiesced snapshots support is removed, because it never worked correctly and caused more harm than good. Note that Windows guest tools version 9 (the default for recent versions of Windows if you install Citrix drivers) already removed VSS support, even for older versions of CH / XCP-ng

I am not sure if this VSS feature is bound to the PV drivers, or if it also needs hypervisor support. Though it is not recommended to stay on a old version of the guest agent.

Hello !

I am looking to get some feedback and evaluation on a performance-related patch for Xen (XCP-ng 8.3 only).
This patch changes the memcpy implementation of Xen to use the "ERMS variant" (aka REP MOVSB) instead of the current REP MOVSQ+B implementation.
This is expected to perform better on the vast majority of Intel CPUs and modern AMD ones (Zen3+), but may perform worse on some older AMD CPUs.

This change may impact the performance of PV drivers (especially network).

You can find more details regarding this proposed change in : https://github.com/xcp-ng-rpms/xen/pull/54
This change may be reworked in the future to take more in account the specificities of each CPUs (e.g check presence of ERMS flag).

Keep in mind that this patched version is experimental and not officially supported.

Installation :

# Download repo file for XCP-ng 8.3
wget https://koji.xcp-ng.org/repos/user/8/8.3/xcpng-users.repo -O /etc/yum.repos.d/xcpng-users.repo

# Installing the patched Xen packages (you should see `.erms` packages)
yum update --enablerepo=xcp-ng-tae1

You can revert the changes by downgrading the Xen package with the ones in the default repos.

yum downgrade --disablerepo=xcp-ng-tae1 "xen-*"

@nvs said in XCP-NG server crashes/reboots unexpectedly:

Thanks. Unfortunately my machine doesnt have IPMI. So can I just connect a serial cable between this machine and another machine

Yes though you would still need to boot using the "XCP-ng (Serial)" grub entry.
(you can also add some serial console bits adding them to xen cmdline)

@nvs said in XCP-NG server crashes/reboots unexpectedly:

@nvs Machine crashed/restarted itself again this morning. I didn't even have all of the usual VMs running this time. Nothing was logged in kern.log when it crashed again. Before it crashed I checked a few times in the hours before xl dmesg but nothing obvious to me (same log as I posted above). Any suggestions highly welcome as I'm sure how to proceed with troubleshooting this. My next step would be replacing the PSU and see if anything changes, but its a long shot.

Ok so it doesn't seem caused by a driver bug causing corruption somewhere in Xen/Linux.
So something is causing Xen to crash, and it's not very easy to know without using e.g the serial console (so you can get the actual Xen crash message).
You need for that something connected and monitoring the machine's serial console (or using IPMI) and boot XCP-ng in "(serial)" mode.