Offline
Hi everyone!
Two years ago I worked on many features in the Xen Orchestra project like delta backup algorithm, load balancing, ...
After all this time, I'm back to contribute to XCP-ng. 
Currently, I'm working on performance improvements (VM migration, storage...) and on new SMAPIv3 plugins.
And maybe in the future on other cool stuff. 
Ronan
Here are some new charts. Make sure you understand the global QCOW2 image structure. (See: https://events.static.linuxfound.org/sites/events/files/slides/kvm-forum-2017-slides.pdf)
More explicit labels :
These results where obtained with an optane (nvme). We can see a better random write performance with the F configuration instead of using the default qemu-dp parameters and the ioping is not so bad. But it's not sufficient compared to tapdisk.
So like said in the previous message, it's the moment to find the bottleneck in the qemu-dp process.
Thank you to flamegraph. (See: http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html)
Analysis and improvements in a future message!
After some investigation, it was discovered that the SMAPIv3 is not THE perfect storage interface. Here are some charts to analyze:
Yeah, there are many storage types:
You can notice the usage of RAID0 with ext4-ng, but it's not important for the moment.
Let's focus on the performance of ext4-ng/xfs-ng! How can we explain these poor results?! By default the SMAPIv3 plugins like gfs2/filebased added by Citrix use qemu-dp. It is a fork of qemu, it's also a substitute of the tapdisk/VHD environment used to improve performance and remove some limitations like the maximum size supported by the VHD format (2TB). QEMU supports QCow images to break this limitation.
So, the performance problem of the SMAPIv3 seems related to qemu-dp. And yes... You can see the results of the ext4-ng VHD and ext4-ng VHDv1 plugins, they are very close to the SMAPIv1 measurements:
Next, to validate a potential bottleneck in the qemu-dp process, two RAID0 have been set up (one with 2 disks and an other with 4), and it seems interesting to see a good usage of the physical disk! There is one qemu process for each disk in our VM, and the disk usage is similar of the performance observed in the Dom0.
The SMAPIv3/qemu-dp tuple is not totally a problem:
Next steps:
UPDATE AND IMPORTANT INFO
I am updating the LINSTOR packages on our repositories.
This update fixes many issues, especially regarding the HA.
However, this update is not compatible with the LINSTOR SRs already configured, so it is necessary to DELETE the existing SRs before installing this update.
We exceptionally allow ourselves to force a reinstallation during this beta, as long as we haven't officially released a production version.
In theory, this should not happen again.
To resume:
1 - Uninstall any existing LINSTOR SR.
2 - Install the new sm package: "sm-2.30.7-1.3.0.linstor.3.xcpng8.2.x86_64" on all used hosts.
3 - Reinstall the LINSTOR SR.
Thank you !
@dwhitehead42 You can install the plug-late-sr RPM from our repo.
Then follow this documentation: https://github.com/xcp-ng-rpms/plug-late-sr/tree/master/SOURCES
@peter_webbird We've already had feedback on CBT and LINSTOR/DRBD, we don't necessarily recommend enabling it. We have a blocking dev card regarding a bug with LVM lvchange command that may fail on CBT volumes used by a XOSTOR SR. We also have other issues related to migration with CBT.
@fatek Just for information, the current version 8.3 is not usable without major problems. Hower, I rebased recently all the LINSTOR sm changes from XCP-ng 8.2 to 8.3 in a new package: sm-3.2.3-1.7.xcpng8.3.x86_64.rpm, we passed the driver tests without too many problems. This RPM should be available during the month of October. Even after its release, we consider that it is not stable enough for production use until we have enough user feedback (but of course this new RPM is synchronized on all fixes and improvements of version 8.2).
EDIT: Released on October 25, we originally planned to wait a bit.
@Maelstrom96 Well there is no simple helper to do that using the CLI.
So you can create a new node using:
linstor node create --node-type Combined <NAME> <IP>
Then you must evacuate the old node to preserve the replication count:
linstor node evacuate <OLD_NAME>
Next, you can change your hostname an restart the services on each host:
systemctl stop linstor-controller
systemctl restart linstor-satellites
Finally you can delete the node:
linstor node delete <OLD_NAME>
After that you must recreate the diskless resources if necessary. Exec linstor advise r to see the commands to execute.
@BHellman The first post has a FAQ that I update each time I meet users with a common/recurring problem.
@henri9813 Can you share your logs (/var/log/SMlog + xensource.log) somewhere? I wonder if this error is triggered by a sm attach call.
@henri9813 Hi! Just to confirm, can you share the exception in the SMlog file of the master host? We are aware of an exception in the same style, and it is a side effect of another exception caused in the _snapshot function of LinstorSR. We have already planned to correct it in a new version of the sm, it's been a while that it's there but not critical. Thanks!
@manilx No, nothing more to do, once installed this package remains present as long as you do not upgrade to a new major version of XCP-ng. But no issues regarding updates.
@Razor_648 Do you have any exceptions in your /var/log/SMlog files?
@dwhitehead42 You can install the plug-late-sr RPM from our repo.
Then follow this documentation: https://github.com/xcp-ng-rpms/plug-late-sr/tree/master/SOURCES
@peter_webbird We've already had feedback on CBT and LINSTOR/DRBD, we don't necessarily recommend enabling it. We have a blocking dev card regarding a bug with LVM lvchange command that may fail on CBT volumes used by a XOSTOR SR. We also have other issues related to migration with CBT.
@nodje said in Create a VM with an existing iSCSI disk:
Is there anyway to achieve an iSCSI mount without automatic LVM VG creation?
You can try with the "RawISCSISR" driver but it's not really tested on our side, I can't confirm that it will work. It doesn't support creation of VDIs but you can use it to access existing LUNs on a target.
@Dark199 In practice you should have more info via dmesg or kern.log. I have never seen this error until now, since it impacts VMs, I am afraid it is something quite serious. Are your disks ok? Do you have enough RAM on the Dom-0?
@dslauter You can test the new RPMs using the testing repository, FYI: sm-3.2.3-1.14.xcpng8.3 and http-nbd-transfer-1.5.0-1.xcpng8.3.
@dslauter Just for your information, I will update the http-nbd-transfer + sm in a few weeks. I fixed many issues regarding HA activation in 8.3 due to bad migration of specific python code from version 2 to version 3.