Best posts made by ronan-a

ronan-a

Hi everyone!

Two years ago I worked on many features in the Xen Orchestra project like delta backup algorithm, load balancing, ...
After all this time, I'm back to contribute to XCP-ng.

Currently, I'm working on performance improvements (VM migration, storage...) and on new SMAPIv3 plugins.
And maybe in the future on other cool stuff.

Ronan

ronan-a

qemu-dp: context and parameters

Here are some new charts. Make sure you understand the global QCOW2 image structure. (See: https://events.static.linuxfound.org/sites/events/files/slides/kvm-forum-2017-slides.pdf)

More explicit labels :

ext4-ng: qemu-dp with default parameters (O_DIRECT and no-flush)
ext4-ng (VHD): tapdisk with VHD (no O_DIRECT + timeout)
ext4-ng (Buffer/Flush): no O_DIRECT + flush allowed
Cache A: L2-Cache=3MiB
Cache B: L2-Cache=6.25MiB
Cache C: Entry-Size=8KiB
Cache D: Entry-Size=64KiB + no O_DIRECT + flush allowed
Cache E: L2-Cache=8MiB + Entry-size=8KiB + no O_DIRECT + flush allowed
Cache F: L2-Cache=8MiB + Entry-size=8KiB + no O_DIRECT
Cache G: L2-Cache=8MiB + Entry-size=8KiB
Cache H: L2-Cache=8MiB + Entry-size=16KiB + no O_DIRECT
Cache I: L2-Cache=16MiB + Entry-size=8KiB + no O_DIRECT

These results where obtained with an optane (nvme). We can see a better random write performance with the F configuration instead of using the default qemu-dp parameters and the ioping is not so bad. But it's not sufficient compared to tapdisk.

So like said in the previous message, it's the moment to find the bottleneck in the qemu-dp process.

ronan-a

qemu-dp/tapdisk and CPU Usage per function call

Thank you to flamegraph. (See: http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html)
Analysis and improvements in a future message!

qemu

tapdisk

ronan-a

SMAPIv3: results and analyze

After some investigation, it was discovered that the SMAPIv3 is not THE perfect storage interface. Here are some charts to analyze:

Yeah, there are many storage types:

lvm, ext (well known)
ext4 (storage type added on SMAPIv1)
ext4-ng (a new storage type added on SMAPIv3 for this benchmark and surely available in the future)
xfs-ng (same idea but for XFS)

You can notice the usage of RAID0 with ext4-ng, but it's not important for the moment.

Let's focus on the performance of ext4-ng/xfs-ng! How can we explain these poor results?! By default the SMAPIv3 plugins like gfs2/filebased added by Citrix use qemu-dp. It is a fork of qemu, it's also a substitute of the tapdisk/VHD environment used to improve performance and remove some limitations like the maximum size supported by the VHD format (2TB). QEMU supports QCow images to break this limitation.

So, the performance problem of the SMAPIv3 seems related to qemu-dp. And yes... You can see the results of the ext4-ng VHD and ext4-ng VHDv1 plugins, they are very close to the SMAPIv1 measurements:

The ext4-ng VHDv1 plugin uses the O_DIRECT flag + a timeout like the SMAPIv1 implementation.
The ext4-ng VHD plugin does not use the O_DIRECT flag.

Next, to validate a potential bottleneck in the qemu-dp process, two RAID0 have been set up (one with 2 disks and an other with 4), and it seems interesting to see a good usage of the physical disk! There is one qemu process for each disk in our VM, and the disk usage is similar of the performance observed in the Dom0.

For the future

The SMAPIv3/qemu-dp tuple is not totally a problem:

A good scale is visible with the RAID0 benchmark.
It's easy to add a new storage type in the SMAPIv3. (Two plugin types: Volume and Datapath automatically detected when added in this system. See: https://xapi-project.github.io/xapi-storage/#learn-architecture)
The QCow2 format is a good alternative to break the size limitation of the VHD images.
A RAID0 on the SMAPIv1 does not improve the I/O performance contrary to qemu-dp.

Next steps:

Understand how qemu-dp is called (context, parameters, ...).
Find the bottleneck in the qemu-dp.
Find a solution to improve the performance.

ronan-a

@peder Fixed! This fix will be available (as soon as possible) in a future xcp-emu-manager package.

ronan-a

@berish-lohith Just FYI I created a card in our backlog, I don't see too many blocking points to implement it correctly.

ronan-a

@jmccoy555 said in RunX: tech preview:

this uuid is of the SR created by the step in the first post?

Right!

Regarding all xe vm-param-set/xe vm-disk-remove/xe template-param-set commands, you must use the VM UUID returned by the xe vm-install command. I edited the post regarding that.

ronan-a

@bc-23 You don't have the patched RPMs because there is a new hotfix in the 8.2 and 8.2.1 versions on the main branch. So the actual xenopsd package version is greater than runx... So we must build a new version of the runx packages on our side to correct this issue. We will fix that.

ronan-a

@swivvle You can create a ext4 SR like this:

xe sr-create type=ext4-ng name-label=sr-test device-config:device=/dev/sda3

For a basic filebased SR:

xe sr-create type=filebased name-label=sr-test2 device-config:file-uri=/root/sr-folder

Don't hesitate if you have other questions.