Hi everyone!
Two years ago I worked on many features in the Xen Orchestra project like delta backup algorithm, load balancing, ...
After all this time, I'm back to contribute to XCP-ng. 
Currently, I'm working on performance improvements (VM migration, storage...) and on new SMAPIv3 plugins.
And maybe in the future on other cool stuff. 
Ronan
Here are some new charts. Make sure you understand the global QCOW2 image structure. (See: https://events.static.linuxfound.org/sites/events/files/slides/kvm-forum-2017-slides.pdf)
More explicit labels :
These results where obtained with an optane (nvme). We can see a better random write performance with the F configuration instead of using the default qemu-dp parameters and the ioping is not so bad. But it's not sufficient compared to tapdisk.
So like said in the previous message, it's the moment to find the bottleneck in the qemu-dp process.
Thank you to flamegraph. (See: http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html)
Analysis and improvements in a future message!
After some investigation, it was discovered that the SMAPIv3 is not THE perfect storage interface. Here are some charts to analyze:
Yeah, there are many storage types:
You can notice the usage of RAID0 with ext4-ng, but it's not important for the moment.
Let's focus on the performance of ext4-ng/xfs-ng! How can we explain these poor results?! By default the SMAPIv3 plugins like gfs2/filebased added by Citrix use qemu-dp. It is a fork of qemu, it's also a substitute of the tapdisk/VHD environment used to improve performance and remove some limitations like the maximum size supported by the VHD format (2TB). QEMU supports QCow images to break this limitation.
So, the performance problem of the SMAPIv3 seems related to qemu-dp. And yes... You can see the results of the ext4-ng VHD and ext4-ng VHDv1 plugins, they are very close to the SMAPIv1 measurements:
Next, to validate a potential bottleneck in the qemu-dp process, two RAID0 have been set up (one with 2 disks and an other with 4), and it seems interesting to see a good usage of the physical disk! There is one qemu process for each disk in our VM, and the disk usage is similar of the performance observed in the Dom0.
The SMAPIv3/qemu-dp tuple is not totally a problem:
Next steps:
@peder Fixed! This fix will be available (as soon as possible) in a future xcp-emu-manager package.
@swivvle You can create a ext4 SR like this:
xe sr-create type=ext4-ng name-label=sr-test device-config:device=/dev/sda3
For a basic filebased SR:
xe sr-create type=filebased name-label=sr-test2 device-config:file-uri=/root/sr-folder
Don't hesitate if you have other questions.
@theaeon said in RunX: tech preview:
Oh now that's interesting. Turns out the containers (both archlinux and the one i just created) are exiting w/ error 143. They're getting sigterm'ed from somewhere.
It's related to how we terminate the VM process: it's a wrapper and not the real process that manages the VM. But we shouldn't show this code to users, it's not the real code, I will create an issue on our side, thanks for the feedback.
@theaeon said in RunX: tech preview:
For what its worth, the podman logs archlinux command from above is w/o debug. I didn't quite realize it vanishing immediately was intended behavior though, tells you how versed I am in containers.
I'll try setting up the matrixdotorg/mjolnir thing again now that I have a command I know is working on runc.
Yeah by default the archlinux image executes the bash command and when the container is started, bash is launched and died just after that. Finally the VM is stopped. This behavior is the same on docker with this image. However using the interactive mode, it's not the case, but we must implement it for a next runx version.
@theaeon Like I said we must change how arguments are parsed in the runx script, so avoid additional params like --log-devel.
@theaeon Could you verify the UUIDs of your config in /etc/runx.conf?
In a shell:
xe sr-list uuid=<SR_UUID>
xe template-list uuid=<TEMPLATE_UUID>
You must have a valid output for each command.
Also ensure the template is a PV template.
Edit: You can also try this basic command: podman start archlinux without other arguments. The parsing of command line arguments must be improved.
It seems to be supported by the XAPI:
https://github.com/xapi-project/xen-api/blob/master/ocaml/xapi/cli_operations.ml#L833-L837
https://xapi-project.github.io/xen-api/classes/pool.html#enable_ha
SR ref set heartbeat_srs: Set of SRs to use for storage heartbeating
Did you tried this command?
@swivvle What's your exact command line? Have you restarted the toolstack?
@swivvle You can create a ext4 SR like this:
xe sr-create type=ext4-ng name-label=sr-test device-config:device=/dev/sda3
For a basic filebased SR:
xe sr-create type=filebased name-label=sr-test2 device-config:file-uri=/root/sr-folder
Don't hesitate if you have other questions.
@peder Fixed! This fix will be available (as soon as possible) in a future xcp-emu-manager package.
@Kalloritis I'm not sure about that. You see a stack of "unknown" functions because the tapdisk binary was not compiled with all debug symbols, contrary to qemu-dp.
And I'm waiting for the next XenServer release before continuing this diary.
Thank you to flamegraph. (See: http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html)
Analysis and improvements in a future message!