git repository structure and conventions for RPMs

stormi

Hello everyone.

I had a go at writing some conventions and rules for the git repositories we are about to create for the RPMs in XCP-ng.

Here is the draft (work in progress).

Comments and suggestions welcome!

RPM git repository conventions

Most of those conventions come from the experience of RPM-based linux distributions such as Fedora or Mageia. They are very important to keep the builds automatable and ease maintenance and development in the long term.

Location

Name

The repository name must match that of the source package. The source package name is that of the source RPM. For example the binary xen-hypervisor package, among other packages, comes from the xen source package, so the repository name is simply xen.

Tree structure

SOURCES/
SPECS/name_of_package.spec
.gitattribute
.gitignore
README.md

The directory structure is the same as in source RPMS (except dotfiles related to git + README.md).
The SOURCES/ directory contains unmodified upstream source archives whose name and provenance match the SourceX (where X is 0, 1, 2...) tags in the spec file, unless impossible. If the XCP-ng community is the upstream for this package, the archive must still come from a tagged release of that software, e.g. https://github.com/xcp-ng/xcp-featured/releases/tag/v1.1.0
It can also contain patches.

Naming sources

For sources that come from the XCP-ng community itself, standard convention applies here: {nameofproject}-{version}.{extension}. Example: xcp-featured-1.1.0.tar.gz.
If the upstream project follows a different convention, we'll usually adapt to theirs in order not to alter the upstream release archive.

Storing sources : `git-lfs`

git is not good at storing large binary files, so we use git-lfs for that. See https://git-lfs.github.com/

Make sure ot install git-lfs on your system before cloning our RPM source repositories with git. It should automatically retrieve the binary files and handle their update through git-lfs.

Modifying existing patches made by others

Do not modify existing patches from XenServer, CentOS or others. Actually, we'll usually not even modify our own patches, unless it really makes no sense to keep two separate patches. Modifying existing patches would make future updates difficult to manage if the patches change, and hides our own changes. It's not manageable in the long run.

Instead, apply them to the sources then create a separate patch on top of that.

Naming patches

TODO: to be improved by looking at how others do it + Add examples.

A naming convention for patches is important because it can give a lot of useful information:

provenance of the patch
version the patch was made for
what the patch does

For easier maintenance, we want to differentiate patches that come from XenServer source packages from those added by the XCP-ng community.

So here is a naming convention proposal.

Patches included in the original XenServer packages: leave their name as is.

For patches we created:

{name-of-package}-{version}-{what-it-does-in-a-few-words}.XCP-ng.patch

For patches that backport changes from the upstream project:

{name-of-package}-{version}-{what-it-does-in-a-few-words or upstream commit or merge request message (with dashes instead of spaces)}.backport.patch

For patches that backport fixes from CentOS that the XenServer project did not apply

{name-of-package}-{version}-{name-of-the-patch-in-centos}.centos.patch

For patches from anywhere else:

{name-of-package}-{version}-{description or name of commit or ...}.{provenance}.patch

{version} is the version the patch was introduced for. In case it was needed to rediff it, don't change the version. If the changes to the patch to make it still apply are significant enough, though, change it to current version but keep information about the initial version the patch was introduced for, in the patch header.

Special rules may apply to some packages, especially packages with a lot of patches (e.g. kernel).

Patch creation

Several ways:

git diff from the root of a git repository
git format-patch
use the patcher tool:
- extract the sources to SOURCES/{packagename}-{version} (tar xf {nameoftarball} in the SOURCES/ directory is usually sufficient for that)
- apply already existing patches if they modify files that you are about to modify yourself
- ptr start
- make changes
- from the SOURCES/ directory: ptr diff > {name-of-patch}

Patches will usually be applied with the -p1 switch. man patch for what it means.

Patch header

The naming convention gives some information, but documented patches are better. People who'll look at them must understand at first glance what they are about. So use the header for that.

TODO: write header writing rules.

Patches in the spec file

The proposal is to add a newline after existing patches, then # XCP-ng patches, then our patches numbered starting at 1000 (or higher if that order or magnitude is already used by XenServer, CentOS or EPEL in that spec file).

Advantages:

find our patches at a glance
less merge conflicts when we'll reapply our changes on top of the spec file from XS (and some have lots of patches, so we would have conflicts if we put ours too close to the rest).
our patches will be applied last

Example:

Patch88: use_existing_io_context.patch
Patch89: use_libaio_by_default.patch
Patch90: Use_the_legacy_grant_copy_ioctl

# XCP-ng patches
Patch1000: qemu-dp-2.10.2-add-rbd-support.XCP-ng.patch

Upstreaming patches

TODO write rules about how to submit our patches upstream and keep track of the upstream bug report or other form of submission.

Git branches

The source RPM repositories would usually have the following branches and tags.

The master branch is for the next development version of XCP-ng.

Once a new x.y.z version of XCP-ng is released:

create the x.y branch from master if that's the first release for that branch.
create the x.y.z tag

This will be automated at some point.

For packages that initially come from XenServer:

tag XS-x.y.z (eg. XS-7.5.0) : unmodified sources and spec file from XenServer. Exception: non-free files and trademarked files (such as a Citrix logo) must be removed before pushing anything to the git repositories. Example: https://github.com/xcp-ng-rpms/xen/tree/XS-7.5.0. Note that the Citrix_Logo_Black.png file was removed and replaced by this: https://github.com/xcp-ng-rpms/xen/blob/XS-7.5.0/SOURCES/Citrix_Logo_Black.png.deleted-by-XCP-ng.txt.
branch XS-x.y: starts with the state of XS-major.minor.patch then its contents evolves if the XenServer project issues patch updates to that x.y branch, or hotfixes.

Handling experimental changes

Experimental changes to existing packages

At first, most changes that we will bring to existing packages will be considered experimental, and will go the the extras_testing repository then to the extras repository.

This means that some packages will exist in two versions. One in base (or updates if an update is issued after the release), and one in extras_testing / extras.

We will handle this through one single spec file. See the blktap example: https://github.com/xcp-ng-rpms/blktap/blob/209205c4cddeadd80b0c10228fb6045dec782935/SPECS/blktap.spec

What triggers the build of the extras version is when we add a --define 'xcp_ng_section extras' parameter to rpmbuild.

A suffix for the release is created if the value of %xcp_ng_section is defined and its value is extras (on top of spec file)

# XCP-ng: release suffix for 'extras' section
%if "%{?xcp_ng_section}" == "extras"
%define rel_suffix .extras
%endif

The suffix is added to the Release tag if present.

Release: 1.17%{?rel_suffix}

A specific patch is applied only if %xcp_ng_section is defined and its value is extras

#XCP-ng patches
%if "%{?xcp_ng_section}" == "extras"
Patch1000: blktap-3.5.0-make-O_DIRECT-optional.XCP-ng.patch
%endif

So depending on the rpmbuild parameters this spec file will either produce blktap-3.5.0-1.17.x86_64.rpm (currently in base: https://updates.xcp-ng.org/7/dev/base/x86_64/Packages/) or blktap-3.5.0-1.17.extras.x86_64.rpm (currently in extras_testing: https://updates.xcp-ng.org/7/dev/extras_testing/x86_64/Packages/).

Experimental new packages

Those won't need all the above as long as we don't plan to push them to both base/updates and extras/extras_testing. Just write spec files the standard way.

As a convention maybe we can just add a header comment in the spec file:

# Target repository: extras

stormi

Draft updated: added section about git-lfs and a part about git branches.

stormi

First example: https://github.com/xcp-ng-rpms/qemu-dp

No changes yet in the XCP-ng-7.5-community branch, but I still created the branch already. No XCP-ng-7.5 branch because we don't plan to release a modified qemu-dp in the main repositories.

tzirechnoy

@stormi said in git repository structure and conventions for RPMs:

The repository name must match that of the source package.

A you really shure you want a repo for every source package?

It could be hundreds of repos! And you would need to make snapshots/synchronizations for making xcp-ng releases among all of them.

And it could lead to situation, when you need to reimport some new history for some abandoned and resurrected package.

While having one more level of indirection is not a fun either, not having it seems a bit restrictive.

stormi

A separate git repository per package is what Fedora does with success for tens of thousands of packages: https://src.fedoraproject.org/

At some point, a single repository for everything wouldn't scale.

One repo per package is totally manageable with some automation, and is a prerequisite if we want to use an advanced building system such as https://pagure.io/koji.

And if we need to resurrect a dropped package, well we can do it, but I don't understand why having a separate git repository for it would make it harder.

stormi

Document updated: new, simpler, branch structure for the repositories + section added to detail how we handle experimental changes to existing packages.