Reworking the 'xe-guest-utilities' Repo

stormi

I must also add that there are many individuals inside Citrix' XenServer team who do care about open source.

stormi

@JustinTimperio don't hesitate to "release early and often" that work of yours so that you can get early feedback and make reviewing easier.

Note: providing makefiles for more package managers is not a priority in my eyes. We prefer to push for xe-guest-utilities packages in each distribution so that users of those don't even need the binaries and packages we provide. A few distros that already have such packages in their repos are listed at https://xcp-ng.org/docs/guests.html#install-from-the-distro-s-online-repositories (some are missing : I know there exist AURs for example).

One of my priorities, however, is to build our own binaries instead of using those provided by Citrix. If you want to help on that front, the ideal situation would be to be able to build them in Koji. This would imply that we manage to build all packages (deb, tgz, etc.), both 32 and 64 bit versions, and put all this into an ISO file, all that from the %build and %install sections of the spec file of our xcp-ng-pv-tools RPM. I think this is doable with Go's GOARCH environment variable, but I haven't tried yet and there may be obstacles to this solution.

Said differently, in case it's not very clear: one single source RPM => a RPM that contains an ISO image that itself contains all the built packages. That's not how packaging works usually, but that's the need

stormi

Oh, as a package maintainer, there's also a question I need to ask: if we were to follow your impulsion and include all the changes that you are currently working on or planning to, will you stay around for the next few years to continue maintaining that code, fix bugs, help review PRs, etc?

JustinTimperio

don't hesitate to "release early and often" that work of yours so that you can get early feedback and make reviewing easier.

Total agreement here. Unfortunately, I am basically rebasing the entire repo so none of my changes are in a fully functional state and not in a place where I could create a PR. If you want to track my progress check out this repo.

providing makefiles for more package managers is not a priority in my eyes.

For me, this one of my main priorities. I am of an 'open-source first' mentality and placing the burden on each distributions package maintainers is a bit backward in my eyes. Citrix already supports enterprise clients, and basically only updates rpm and deb packages. My goal is to bring all opensource, free distributions up to the same level of compatibility and support as enterprise. In addition, by placing the burden on each distributions team, updates have been largely ignored. Freebsd, Fedora, Ubuntu, CentOS, etc all have xe-guest-utilities that are in some cases, years out of date.

One of my priorities, however, is to build our own binaries instead of using those provided by Citrix

I'm a little confused about this statement. Since I am patching security issues in the go code itself, we will have to compile these into our own binaries, and therefore a separate package(not called xe-guest-utilities for obvious versioning conflict issues). Makefiles will be separated in a similar-ish way to how they are in the Citrix Repo. The core makefile creates a secure temp go build environment, then generates the target binaries. From there, each distribution has it own makefile (in Citrix repo mk/Makefile.deb and mk/Makefile.rpm), which then take the compiled binaries and collects the other various files needed to package these into a xe-guest-utilities.deb or xe-guest-utilities.deb.

If you want to help on that front, the ideal situation would be to be able to build them in Koji.

I can't speak to Koji, I've never used it. Honestly, building the binaries is like the easiest part of this process(you just need to setup some gopaths). I was just gonna use the tool wrote to natively build a package for each distro in its target distro(IE build fedora packages inside fedora), then return the compiled packages to an endpoint.

will you stay around for the next few years to continue maintaining that code, fix bugs, help review PRs, etc?

Just depends if the work I've done gets adopted or not. I usually take on projects when they have a direct impact on my life and also helps the community. I use a lot of distros and I'm really sick of having second rate drivers on non deb and rpm based systems. It solves a problem for me so I will maintain them for at least that reason. My hope obviously, is that this work will be helpful for people using none deb and rpm systems as well, I just try not to get too attached to how many people are using an opensource project I write. It can be really discouraging if you put hundreds of hours into a project for the community that then is never used.

JustinTimperio

O, also do you know anyone willing to help with RHEL and SLES testing? I don't have access to enterprise distros I can test or build on. Idk what is even in their /etc/os-release at this point? (I threw away Citrix distro identification script and rewrote it from scratch. It's 1/3 as long and supports way more distros

stormi

@JustinTimperio said in Reworking the 'xe-guest-utilities' Repo:

providing makefiles for more package managers is not a priority in my eyes.

For me, this one of my main priorities. I am of an 'open-source first' mentality and placing the burden on each distributions package maintainers is a bit backward in my eyes. Citrix already supports enterprise clients, and basically only updates rpm and deb packages. My goal is to bring all opensource, free distributions up to the same level of compatibility and support as enterprise. In addition, by placing the burden on each distributions team, updates have been largely ignored. Freebsd, Fedora, Ubuntu, CentOS, etc all have xe-guest-utilities that are in some cases, years out of date.

Well, I've been packager for a distro for long and I've almost never encountered spec files from software projects that were of any use. Spec files written by developers usually miss a lot of the distros' packaging rules, and little is reusable. Unless you're lucky and the developer knows your packaging rules very well (you can often find spec files targetd at RedHat/Fedora or Debian that are "good enough" to be useful, but you often find that aren't).

Also, lack of spec files or equivalents is never what delays a package update. Usually it's just that the maintainer for the distro is less available, or left. So to really help them, one would have to join them and maintain the package for them.

So my rule of thumb is: produce package sources only if you intend to produce packages, else don't. What a packager needs is a source tarball as well and build and installation instructions (that can be provided as a spec file or debian rules, but you don't need them to be for your distro specifically), the rest is in their hands. I think you do want to produce such packages, so why not, as long as maintaining them in the long term does not increase the maintenance burden.

However I still consider that the best solution is to maintain the package for all those distros, in each distro's repository. Ideally, they should even be installed by default with the system when XCP-ng is detected, like many distro do regarding VMWare or Virtualbox guest additions. We need both:

xe-guest-utilites-latest in every distro
The ISO image shipped with XCP-ng and containing the xe-guest-utilities packages for the most used distros + a generic tarball

One of my priorities, however, is to build our own binaries instead of using those provided by Citrix

I'm a little confused about this statement. Since I am patching security issues in the go code itself, we will have to compile these into our own binaries, and therefore a separate package(not called xe-guest-utilities for obvious versioning conflict issues). Makefiles will be separated in a similar-ish way to how they are in the Citrix Repo. The core makefile creates a secure temp go build environment, then generates the target binaries. From there, each distribution has it own makefile (in Citrix repo mk/Makefile.deb and mk/Makefile.rpm), which then take the compiled binaries and collects the other various files needed to package these into a xe-guest-utilities.deb or xe-guest-utilities.deb.

What is confusing for most people is that at XCP-ng level we provide an ISO image that people mount on they guest systems and install the tools from there. That ISO image itself is delivered through XCP-ng's RPM packages. My goal is to provide that package, that contains the ISO, the ISO itself containing the various variations of the xe-guest-utilities packages and the install.sh script. This my need as XCP-ng packager.

will you stay around for the next few years to continue maintaining that code, fix bugs, help review PRs, etc?

Just depends if the work I've done gets adopted or not. I usually take on projects when they have a direct impact on my life and also helps the community. I use a lot of distros and I'm really sick of having second rate drivers on non deb and rpm based systems. It solves a problem for me so I will maintain them for at least that reason. My hope obviously, is that this work will be helpful for people using none deb and rpm systems as well, I just try not to get too attached to how many people are using an opensource project I write. It can be really discouraging if you put hundreds of hours into a project for the community that then is never used.

Another maintenance point is that the more distros you support, the more new releases of those distros you'll have to test when they go out.

stormi

@JustinTimperio I don't know, but there are at least test cases: https://github.com/xenserver/xe-guest-utilities/tree/master/mk/testcases

JustinTimperio

@stormi You bring up some interesting points. I want to make it clear, my intention is not to abandon the creation of a generic tarball. I just also want to include a full package build process for each distro that is officially maintained. My plan was to message the current maintainers of xe-guest-utilites for each distro and strike up a convo.

My goal is to provide that package, that contains the ISO, the ISO itself containing the various variations of the xe-guest-utilities packages and the install.sh script. This my need as XCP-ng packager.

Ahhhh I see what you mean now. Sure that should be pretty easy. I also wanted to create a 'net-install.sh' so users won't even need to mount an iso, just run something like curl net-install-url | sudo bash.

Another maintenance point is that the more distros you support, the more new releases of those distros you'll have to test when they go out.

Yeah, I have def thought about that. I'm still not totally sure how I want to address it, but I currently have custom templates in my cluster for each of these distros. By using the xoapi and PRT I'm pretty confident that I can fully automate the build, and testing process. Another alternative is a custom travis-ci cluster but I'll cross that bridge when I get there.

I don't know, but there are at least test cases: https://github.com/xenserver/xe-guest-utilities/tree/master/mk/testcases

I checked those out but that dir contains /etc/rhel-release rather than /etc/os-release which I generally prefer. We can always figure that out later but currently, this new script provides more os info in way fewer lines. (Sample output here)

ravenet

@JustinTimperio I can test SLES for you, I'm a SUSE partner so have access. Here's output from a sles11.4,12.4, 12.5, 15 and 15.1. Sorry don't have a sles15.2 spun up (only leap15.2) but easy enough to extrapolate.
Best way to get stuff into their official distro is to have it build on OBS

NAME="SLES"
VERSION="11.4"
VERSION_ID="11.4"
PRETTY_NAME="SUSE Linux Enterprise Server 11 SP4"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:11:4"

NAME="SLES"
VERSION="12-SP4"
VERSION_ID="12.4"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP4"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp4"

NAME="SLES"
VERSION="12-SP5"
VERSION_ID="12.5"
PRETTY_NAME="SUSE Linux Enterprise Server 12 SP5"
ID="sles"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:12:sp5"

NAME="SLES"
VERSION="15"
VERSION_ID="15"
PRETTY_NAME="SUSE Linux Enterprise Server 15"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15"

NAME="SLES"
VERSION="15-SP1"
VERSION_ID="15.1"
PRETTY_NAME="SUSE Linux Enterprise Server 15 SP1"
ID="sles"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sles:15:sp1"

JustinTimperio

@ravenet Thanks so much! I'll add these examples to our new test case folder I will definitely be in touch when we are ready to test on SLES!

An Update on Progress:

I have a new collaborator working with me on reworking these tools which should help with development. He uses XCP-ng as well and is pretty familiar with all the OS's I'm currently planning on officially supporting.

Actual progress has been a little sporadic as we pull apart these tools. The complete lack of documentation has made auditing the Go tedious, as I personally am not familiar with the xenbus standard and/or the data-structure Citrix is using pass info to dom0. The more I pull apart this repo, the more I am questioning the sanity of the Citrix developers. My gut feeling is a lot of this is legacy code that no one wanted to fix or touch for that matter. At some point when we get closer to a proper release, I'll create a list of all the bizarre and poorly designed code we found.

We do have a working makefile and the tools do compile correctly into a generic package.tgz, but there are still a number of things that need to be fixed before it would function on a guest os. I'm also currently in the process of auditing the use of unsafe.Pointer( ) in the existing wrapper for 'timerfd_settime' calls. I'm not really sure why this was done and using the unsafe package is typically bad practice in any production code. It explicitly breaks Go compatibility guidelines and type safety which is kind of the point of Go IMO

JustinTimperio

For those interested, since we have put a ton of work into beefing up os detection, I have forked our work into a separate repo. This was something I needed for a variety of my opensource projects install scripts and I hope it will also be helpful for others.

You can check it out here. https://github.com/JustinTimperio/nixinfo

For example, on my desktop, this single command returns the following:

❯ curl -s https://raw.githubusercontent.com/JustinTimperio/nixinfo/master/nixinfo.sh | sh

Distribution='ARCH'
Full_Name='Arch Linux'
Package_Manager='pacman'
Kernel='5.8.14'
Major='rolling'
Minor='rolling'
Patch='rolling'

Or my Nginx host returns:

❯ curl -s https://raw.githubusercontent.com/JustinTimperio/nixinfo/master/nixinfo.sh | sh

Distribution='FREEBSD'
Full_Name='FreeBSD 12.1-RELEASE'
Package_Manager='pkg'
Kernel='1201000'
Major='12'
Minor='1'
Patch='n/a'

JustinTimperio

@olivierlambert @stormi

So I have been working on this repo for about a month now and it is in a far worse state than I ever imagined. I showed the codebase to some golang pro's and they pretty much roasted every aspect of the original source code. It uses multiple depreciated packages, breaks multiple best practices that golang considers to be fundamental to portability and future-proofing, has a complete lack of any documentation on a variety of esoteric calls, and the source code does not compile against GOOS=freebsd.

I was hoping I could just make some quick changes to clean up this repo for the XCP-NG community but this is a huge endeavor to get in a good place for the future. I would very much like to continue work on this but it has become clear to me that a small team of developers will be required to finish this in a timely fashion. I would suggest creating a new repo called XCP-guest-tools as a properly upgraded version of this repo would rewrite 40%-60% of the original source code to bring it up to a reasonable standard. I also think at that point it would be extremely unlikely that Citrix would ever merge the changes so splitting off to a newly named package would go a long way to avoid confusion about versions and source.