Hi Everyone,

As you know, XCP-NG uses a highly modified kernel 4.4. A lot of device drivers are already present in this kernel tree, however there are edge cases where one wants to add a new driver.

We have added a simple repo which you can fork, add your driver source code and let XCP-NG build system do the rest of work for you

The output is located in RPMS folder, which you can ship to your XCP-NG host and install.

Please have a look https://github.com/xcp-ng-rpms/kernel-driver-template and share your feedback on being able to build and test your first driver.

@olivierlambert said in XenServer 8.0 - Major update due Q1 2019:

Our ambition is to become the first turnkey and Open Source virtualization platform.

And the community involved here is progressing to achieve this collectively.

@stormi happy to have you. Hope to see some great action!

So taking inspiration and doing small modifications to existing NFS driver, patch allows you to use both NFS and Ceph FS same time.

Note : The patch is applied against XCP-NG 7.6.0

# cd /
# wget "https://gist.githubusercontent.com/rushikeshjadhav/af53bb5747365875f0ab21bd3a64c6fe/raw/59ef7a4b54574e4163da1ac39acd640554bd0d24/ceph.patch"
# patch -p0 < ceph.patch

Apart from the patch, ceph.ko & ceph-common would be needed.

To install ceph-common on XCP-NG
# yum install centos-release-ceph-luminous --enablerepo=extras
# yum install ceph-common

To install and load ceph.ko
# wget -O ceph-4.4.52-4.0.12.x86_64.rpm "https://github.com/rushikeshjadhav/ceph-4.4.52/blob/master/RPMS/x86_64/ceph-4.4.52-4.0.12.x86_64.rpm?raw=true"
# yum install ceph-4.4.52-4.0.12.x86_64.rpm
# modprobe -v ceph (Optional)

User experience:

In Share Name, you can put multiple addresses from your CEPH mon.
In Advanced Options, you need to add your ceph authx user details, such as name and its secret file location. These will be used by mount.ceph.

Note: Keep secret file in /etc/ceph/ with permissions 600.

Current SR detection patch replies on presence of word "ceph" in advance options to discriminate between NFS and CEPH FS.

Log Location : /var/log/SMlog

Edit: Created RPM installer for ceph.ko

As a side experiment - I could extract and test CEPH FS kernel module for XCP-NG 7.4+ and mount a multi-mon ceph cluster (luminous).

Once ceph.ko module is in place, XCP-NG can mount the ceph fs mount point similar to NFS.

e.g. In NFSSR we use #mount.nfs4 addr:remotepath localpath

while for CEPHSR we can use #mount.ceph addr1,addr2,addr3,addr4:remotepath localpath

I'm currently looking at NFSSR.py to create CEPHFSSR.py - and would share once ready, meanwhile if anyone wants to help me in this by testing ceph.ko, developing CEPHFSSR.py, kindly ping here. Like EXT4 and XFS, this too will have some cleanup issues..

If this works as expected, I'd request @olivierlambert and @borzel to look at possibilities of integrating CEPH FS option in XO/XC near NFS SR.

Note: This is completely different than RBDSR which is worked by @rposudnevskiy and uses rbd image protocol of ceph.

Bonus : If someone knows how nfs-ganesha fits in this - buzzz... e.g. Each CEPH FS node can have a NFS server while each XCP-NG host mounts all of them using NFS 4.1 (pNFS). Bypassing need of ceph.ko. We can also rebuild NFS module as CONFIG_NFSD_PNFS is not set by default.

Caveats : CEPH FS caching is not fully explored.

State: Ready. See post below for instructions on how to use.

@jmccoy555 I did not continue this. Will definitely rejuvenate it once 8.1 is released. Hopefully next week.

@olivierlambert said in Windows 10 VM reboots when starting software:

I don't know if @r1 can build it for us, but we'll have to wait next week for @stormi to put the package in testing anyway

The patch needed a rework for fitment. I'd create a PR for modified patch and then it can go in testing repo.

Yes - its in xcp-ng-testing repo which is not enabled by default.

Link https://github.com/xcp-ng/xcp/wiki/Alternate-kernel is now updated with installation instructions.

@dave-opc To know more on next incident, you may add noreboot options to multiboot2 /boot/xen.gz in /etc/grub.cfg file against default kernel boot entry.

@dave-opc whats the XCP-ng version?

Did you see -

https://github.com/xcp-ng/xcp/wiki/Troubleshooting

https://github.com/xcp-ng/xcp/wiki/Logfiles

@chrispage1 said in Auto coalescing of disks:

Apr 7 16:07:09 node-01 SMGC: [20480] text = session.xenapi.host.call_plugin(slave, "on-slave", "multi", args)

Seems the issue is here. Are all your slaves in pool healthy?

@chrispage1 said in Auto coalescing of disks:

Question 2 - can I force coalesce by rescanning the disk? If I rescan the disk, is there any chance of this causing an outage?

This automatically kicks GC apart from regular schedule. Rescanning won't cause outage. Do check /var/log/SMlog for any obvious error causing GC to not work.

@borzel Thanks. That rules out the my suspicion on those patches. We are still working on reproducing this issue without success. We would really appreciate if you or someone from community can arrange a test hosts which shows this problem. Reason for test host is because we will have to replace the kernel multiple times to observe change.

Another test users can do is to remove iscsi from equation. Run some workloads on local disks (with backups) and verify control domain memory usage.

@borzel Also, I somehow need to be able to reproduce the issue at lab. If you can give more details about how do you do backup, may be I can simulate something.

@borzel I wish comparing kernel-alt and base kernel was easy to catch this... I'm sure that the tapdisk IO code is same in kernel and kernel-alt.

Also the 2 patches mentioned earlier are also present in base kernel of xcp-ng 8.2 as well as kernel-alt 4.19.142. They are also present for xcp-ng 8.1 base kernel, however they are not present in xcp-ng 8.1 kernel-alt.

Can you confirm your kernel-alt version?

@borzel Between 4.19.19-6.0.9 to 4.19.19-6.0.10, following two patches were added.

0001-block-cleanup-__blkdev_issue_discard.patch
0001-block-fix-32-bit-overflow-in-__blkdev_issue_discard.patch

Both are well vetted and seems stable without any further changes in them. Was there anything else updated along with kernel?

@borzel How frequently do you restart VMs? And what's the last dom-id? # xl list

@trent234 I think system should run headless and Xen should be able to hide that device from dom0, can you reassure that BIOS is set to not use any display device?

@appollonius You can set nomodeset option at the end of line which starts with module2 /boot/vmlinuz-4.19-xen.

With other user it did no make any difference.

@delaf said in Alert: Control Domain Memory Usage:

one server (268) with 4.19.19-6.0.10.1.xcpng8.1: no more problem!

Yeah, we need to be sure that this is a stable kernel and somewhere after this, the memory leak seems to have introduced.