Reworking the 'xe-guest-utilities' Repo
-
@JustinTimperio said in Reworking the 'xe-guest-utilities' Repo:
- So when you say you don't have the build tools yourself, what does that mean exactly? If you have a working go dev environment, there is a makefile that will generate source. It won't make the individual packages (.deb, .rpm, .pkg) but that is pretty trivial in the scope of this. Does this mean you are not building these tools from source for each release?
Yes, we have an open issue for that because I want that everything we provide be built by us, but currently the packages we have in our guest tools ISO are extracted from XenServer's RPM (that contains the ISO).
Our situation is different from that of any linux distribution that wants to provide the tools (which we encourage any distro to do), since what we provide is a RPM that contains an ISO that itself contains DEBs, RPMs, and others. It's not something we can easily build in our build system, koji (though there may be a way to manage it). So we've get to automate that build process.
In addition to that, I know the packages produced by Citrix are made so that they still work on older distributions, and I haven't had the time to define which distro we should build the binaries on for maximum compatibility. Building them in the same build root as we build XCP-ng may not be appropriate.
- Can you clarify the versioning for me? (Let's use ubuntu as a test case to reduce complexity) The official Ubuntu Repos contain version
xe-guest-utilities_7.10.0-0_amd64.debas the most recent version. The XCP-ng tools iso on the other hand contains versionxe-guest-utilities_7.16.0-2_amd64.deb. Is this because Ubuntu has yet to accept new versions built by Citrix and provided through their iso?
It's most likely because they haven't synced the sources recently. I doubt they provide binaries built by Citrix. Official versioning is, as far as I can tell, that of the tags in https://github.com/xenserver/xe-guest-utilities/tags
- What is the meaning of the management agent version in XO? This has been adding to my confusion about what is actually getting versioned here.
As Olivier said, this is the version that the tools themselves report to the XAPI. It does not seem to be related to the tags discussed above, and I don't know the logic behind those numbers.
-
I think there's absolutely no logic, sadly…
-
Okay, I think I am starting to get the picture here.
XO is reporting what's XAPI is seeing.
@olivierlambert Not sure it's pulling from that command. The only thing that the XAPI returns that is relevant is
PV-drivers-detected ( RO): true. If I runxe-param-list someuuid | grep 8.0I get nothing. (Could totally be wrong here)Official versioning is, as far as I can tell, that of the tags in
@stormi Okay weird, so they don't use releases but they will bump versions using tags if they edit a file....
It's most likely because they haven't synced the sources recently. I doubt they provide binaries built by Citrix.
@stormi That's disconcerting...
I think there's absolutely no logic, sadly…
Yeah, that's why I wanted to fork this into something that actually follows open-source standards lol. Most of my desire to move this into a separate repo is to avoid all these issues, which IMO, makes the Xen project look kind of poorly. When I was a noob, I spent so many hours trying to understand what guest tools were and why none of the versions were the same. Its been over a year now and sadly it has nothing to do with skill level. The tools are poorly maintained and versioned. Since Citrix is doing such a poor job, I think becoming the new upstream developer and supplier of these tools is a far more achievable option than trying to get Citrix to pull it together. XCP-ng as an open-source project can work directly with distro developers to unify and standardize these tools and make sure all distros are up to date.
Furthermore, by actually developing these tools and adding new features that Citrix is neglecting, XCP can 'corner' the market so to speak. I would way rather see you all grow and succeed over Citrix or VMware.
-
@stormi said in Reworking the 'xe-guest-utilities' Repo:
Our situation is different from that of any linux distribution that wants to provide the tools (which we encourage any distro to do), since what we provide is a RPM that contains an ISO that itself contains DEBs, RPMs, and others. It's not something we can easily build in our build system, koji (though there may be a way to manage it). So we've get to automate that build process.
Yeah, that is the bain of every multi-distro project. I am actually tooling something right now that can hopefully do that. It won't be the most advanced thing on the face of the earth but it will at least allow for testing on multiple distro's at once. Will prob take me a few days to finish.
-
Be sure to use:
xe vm-param-get param-name=PV-drivers-version uuid=<VM UUID>Output example:
# xe vm-param-get param-name=PV-drivers-version uuid=1c970fff-a74c-2f9d-21fe-9a9259e333e1 major: 8; minor: 0; micro: 50; build: 1 -
Okay yeah, that works. Wow, what the hell then... So basically the management engine is a totally pointless output as far as determining the xe-guest-utilities version. I would have thought that the management engine was == to utilities version.
- FreeBSD 12.1 is
xe-guest-utilities-6.2.0_3.txzandManagement agent 6.2 detected - Alpine Linux 3.11 is
xe-guest-utilities-7.17.0-r1.apkandManagement agent 6.6 detected - Ubuntu 20.04 is
xe-guest-utilities_7.16.0-2_amd64.debandManagement agent 8.0 detected
- FreeBSD 12.1 is
-
-
Having tests will be a great argument to become the new upstream
Our situation is different from that of any linux distribution that wants to provide the tools (which we encourage any distro to do), since what we provide is a RPM that contains an ISO that itself contains DEBs, RPMs, and others. It's not something we can easily build in our build system, koji (though there may be a way to manage it). So we've get to automate that build process.
This weekend I was able to finish the first tool I needed to help streamline the process of testing and building these tools. Its something I'm just calling PRT(Parallel Remote Terminal) for now. I currently have a number of testbeds in my cluster but testing manually on each is a huge pain. PRT is a tool that allows a user to define a group of remote hosts using a yaml file, then send the same command in parallel to each host. PRT returns the connection status, output, and errors on each host directly back to your machine. I hope you won't judge too hard on the quality of my code and docs because this was built in space of 48 hours lol.
You can check it out here:
https://github.com/JustinTimperio/prtI'll clean this up and improve it over the coming weeks but obviously, this is just tooling to help facilitate the actual work of improving these tools.
-
Hey @stormi could you send me an original Citrix iso with its tagged version? I'm trying to figure out if they are making changes before shipping by comparing checksums.
-
You can download the latest built tools from https://www.citrix.com/downloads/citrix-hypervisor/, select CH 8.2 express edition and then the Citrix VM Tools for linux.
-
@stormi I've tried that but all the downloads are behind a login wall.
-
I could upload it but I think it would take me the same amount of time as it would take you to create a Citrix account. If you really can't, I'll try to remember to do it a bit later.
-
I was trying to avoid making a Citrix account lol. (I have a strong distaste for them, for reasons I will cover below)
After digging through the Citrix repo, I have found somethings that frankly infuriate me. Citrix has really no desire to support the opensource community as seen in this comment and their total lack of responsiveness in fixing issues. For instance, this issue, opened and fixed by a member of the fedora team, has sat for over a year. They also have zero documentation anywhere in the code base, and from issues like this they have no intention of adding it. Citrix has abandoned the opensource model to try and compete with VMWare.
While I can understand your desire to maintain some relation with Citrix, I personally have no desire to help them. They have burned their goodwill with the opensource community and this is why, I assume, you started XCP-ng. Moving forward, my desire is to turn this repo into a peer-reviewed, well documented, and actively maintained repository. If Citrix would like to use the improvements XCP-ng makes that's fine, but I don't want my work to sit for years in a pull request that will never be merged by Citrix.
A quick update on progress:
- Added support in install.sh for OpenSuse
- Rebuilt the security audit process of go binaries from python2 --> python3.
- Improved the audit process pretty dramatically.
- Started patching 23 security issues found by the new audit process.
- Started adding documentation across the entire repo.
Up next:
- New package makefiles for the following package managers.
- apt/dpkg
- yum/dnf
- zypper/yast
- pacman
- pkg
- apk - New test process for xe-linux-distribution
-
I agree on various point, Citrix never really was an Open Source friendly company… It's not inside their culture. However, I still have hope we can help them to go into a better direction.
Collaboration is always more powerful than competition, that's why we love Open Source after all
-
I must also add that there are many individuals inside Citrix' XenServer team who do care about open source.
-
@JustinTimperio don't hesitate to "release early and often" that work of yours so that you can get early feedback and make reviewing easier.
Note: providing makefiles for more package managers is not a priority in my eyes. We prefer to push for
xe-guest-utilitiespackages in each distribution so that users of those don't even need the binaries and packages we provide. A few distros that already have such packages in their repos are listed at https://xcp-ng.org/docs/guests.html#install-from-the-distro-s-online-repositories (some are missing : I know there exist AURs for example).One of my priorities, however, is to build our own binaries instead of using those provided by Citrix. If you want to help on that front, the ideal situation would be to be able to build them in Koji. This would imply that we manage to build all packages (deb, tgz, etc.), both 32 and 64 bit versions, and put all this into an ISO file, all that from the
%buildand%installsections of the spec file of ourxcp-ng-pv-toolsRPM. I think this is doable with Go's GOARCH environment variable, but I haven't tried yet and there may be obstacles to this solution.Said differently, in case it's not very clear: one single source RPM => a RPM that contains an ISO image that itself contains all the built packages. That's not how packaging works usually, but that's the need
-
Oh, as a package maintainer, there's also a question I need to ask: if we were to follow your impulsion and include all the changes that you are currently working on or planning to, will you stay around for the next few years to continue maintaining that code, fix bugs, help review PRs, etc?
-
don't hesitate to "release early and often" that work of yours so that you can get early feedback and make reviewing easier.
Total agreement here. Unfortunately, I am basically rebasing the entire repo so none of my changes are in a fully functional state and not in a place where I could create a PR. If you want to track my progress check out this repo.
providing makefiles for more package managers is not a priority in my eyes.
For me, this one of my main priorities. I am of an 'open-source first' mentality and placing the burden on each distributions package maintainers is a bit backward in my eyes. Citrix already supports enterprise clients, and basically only updates rpm and deb packages. My goal is to bring all opensource, free distributions up to the same level of compatibility and support as enterprise. In addition, by placing the burden on each distributions team, updates have been largely ignored. Freebsd, Fedora, Ubuntu, CentOS, etc all have
xe-guest-utilitiesthat are in some cases, years out of date.One of my priorities, however, is to build our own binaries instead of using those provided by Citrix
I'm a little confused about this statement. Since I am patching security issues in the go code itself, we will have to compile these into our own binaries, and therefore a separate package(not called xe-guest-utilities for obvious versioning conflict issues). Makefiles will be separated in a similar-ish way to how they are in the Citrix Repo. The core makefile creates a secure temp go build environment, then generates the target binaries. From there, each distribution has it own makefile (in Citrix repo
mk/Makefile.debandmk/Makefile.rpm), which then take the compiled binaries and collects the other various files needed to package these into axe-guest-utilities.deborxe-guest-utilities.deb.If you want to help on that front, the ideal situation would be to be able to build them in Koji.
I can't speak to Koji, I've never used it. Honestly, building the binaries is like the easiest part of this process(you just need to setup some gopaths). I was just gonna use the tool wrote to natively build a package for each distro in its target distro(IE build fedora packages inside fedora), then return the compiled packages to an endpoint.
will you stay around for the next few years to continue maintaining that code, fix bugs, help review PRs, etc?
Just depends if the work I've done gets adopted or not. I usually take on projects when they have a direct impact on my life and also helps the community. I use a lot of distros and I'm really sick of having second rate drivers on non deb and rpm based systems. It solves a problem for me so I will maintain them for at least that reason. My hope obviously, is that this work will be helpful for people using none deb and rpm systems as well, I just try not to get too attached to how many people are using an opensource project I write. It can be really discouraging if you put hundreds of hours into a project for the community that then is never used.
-
O, also do you know anyone willing to help with RHEL and SLES testing? I don't have access to enterprise distros I can test or build on. Idk what is even in their
/etc/os-releaseat this point? (I threw away Citrix distro identification script and rewrote it from scratch. It's 1/3 as long and supports way more distros
-
@JustinTimperio said in Reworking the 'xe-guest-utilities' Repo:
providing makefiles for more package managers is not a priority in my eyes.
For me, this one of my main priorities. I am of an 'open-source first' mentality and placing the burden on each distributions package maintainers is a bit backward in my eyes. Citrix already supports enterprise clients, and basically only updates rpm and deb packages. My goal is to bring all opensource, free distributions up to the same level of compatibility and support as enterprise. In addition, by placing the burden on each distributions team, updates have been largely ignored. Freebsd, Fedora, Ubuntu, CentOS, etc all have
xe-guest-utilitiesthat are in some cases, years out of date.Well, I've been packager for a distro for long and I've almost never encountered spec files from software projects that were of any use. Spec files written by developers usually miss a lot of the distros' packaging rules, and little is reusable. Unless you're lucky and the developer knows your packaging rules very well (you can often find spec files targetd at RedHat/Fedora or Debian that are "good enough" to be useful, but you often find that aren't).
Also, lack of spec files or equivalents is never what delays a package update. Usually it's just that the maintainer for the distro is less available, or left. So to really help them, one would have to join them and maintain the package for them.
So my rule of thumb is: produce package sources only if you intend to produce packages, else don't. What a packager needs is a source tarball as well and build and installation instructions (that can be provided as a spec file or debian rules, but you don't need them to be for your distro specifically), the rest is in their hands. I think you do want to produce such packages, so why not, as long as maintaining them in the long term does not increase the maintenance burden.
However I still consider that the best solution is to maintain the package for all those distros, in each distro's repository. Ideally, they should even be installed by default with the system when XCP-ng is detected, like many distro do regarding VMWare or Virtualbox guest additions. We need both:
xe-guest-utilites-latestin every distro- The ISO image shipped with XCP-ng and containing the
xe-guest-utilitiespackages for the most used distros + a generic tarball
One of my priorities, however, is to build our own binaries instead of using those provided by Citrix
I'm a little confused about this statement. Since I am patching security issues in the go code itself, we will have to compile these into our own binaries, and therefore a separate package(not called xe-guest-utilities for obvious versioning conflict issues). Makefiles will be separated in a similar-ish way to how they are in the Citrix Repo. The core makefile creates a secure temp go build environment, then generates the target binaries. From there, each distribution has it own makefile (in Citrix repo
mk/Makefile.debandmk/Makefile.rpm), which then take the compiled binaries and collects the other various files needed to package these into axe-guest-utilities.deborxe-guest-utilities.deb.What is confusing for most people is that at XCP-ng level we provide an ISO image that people mount on they guest systems and install the tools from there. That ISO image itself is delivered through XCP-ng's RPM packages. My goal is to provide that package, that contains the ISO, the ISO itself containing the various variations of the
xe-guest-utilitiespackages and theinstall.shscript. This my need as XCP-ng packager.will you stay around for the next few years to continue maintaining that code, fix bugs, help review PRs, etc?
Just depends if the work I've done gets adopted or not. I usually take on projects when they have a direct impact on my life and also helps the community. I use a lot of distros and I'm really sick of having second rate drivers on non deb and rpm based systems. It solves a problem for me so I will maintain them for at least that reason. My hope obviously, is that this work will be helpful for people using none deb and rpm systems as well, I just try not to get too attached to how many people are using an opensource project I write. It can be really discouraging if you put hundreds of hours into a project for the community that then is never used.
Another maintenance point is that the more distros you support, the more new releases of those distros you'll have to test when they go out.
