Structure of the RPM repositories

stormi

[Updated: renamed community to extras and main to base]

After many thoughts, using my experience as a packager for the Mageia linux distribution and looking at how CentOS and Fedora handle repositories, I've come with the following proposal. Since it's a proposal, it means that it's open to discussion of course

But first, some goals we want to reach through this structure:

Allow users to update from XCP-ng 7.4 or later to the latest 7.x release with a simple yum update.
Allow users to test future updates before they are pushed to everyone, to help validate them.
Have a place where additionnal community-made packages can be made available.
Release fixes to current stable release fast.

This would lead to the following structure.

At top level, just a number:

.
├── 7
└── 8

7 for current major version number. 8 for the next, someday (we won't create it yet, actually, so we'll just have 7).

Then, a few subdirectories:

.
├── 7
│   ├── 7.4
│   ├── 7.5
│   ├── 7.6
│   ├── 7.7
│   ├── dev
│   └── latest
└── 8

In the above, let's consider that we are in the future and that the current stable branch is 7.7.

latest is a symbolic link that will always point at the latest stable 7.x branch. Our current maintenance plan is to have people always update to the latest 7.x branch, so counting from XCP-ng 7.5, the yum repositories configured on XCP-ng instances would point at that latest repository. This means that because of our limited resources, at first we don't plan to keep patching previous 7.x releases for security issues or bugs.

dev is the future 7.8 (in our example). Can be unstable. That's for pioneers and developers. Once 7.8 is ready, dev will be copied to a 7.8 repository, and the latest symlink will be made to point to 7.8.

Now let's go one level deeper.

.
├── 7
│   ├── 7.4
│   ├── 7.5
│   │   ├── builddeps
│   │   ├── extras
│   │   ├── extras_testing
│   │   ├── base
│   │   ├── updates
│   │   └── updates_testing
│   ├── 7.6
│   ├── 7.7
│   ├── dev
│   └── latest
└── 8

base is the state of the packages in the installation ISO for 7.5. Activated by default.
updates contains released updates to the 7.5 branch during the period between 7.5 and 7.6. Activated by default.
updates_testing contained (past tense: remember current release is 7.7 in our example) update candidates for the 7.5 branch before they were pushed to everyone. Disabled by default.
extras contains additional RPMs from the community. Use at your own risks, but they can bring useful additions! We'll try to guarantee the upgrade path even with these installed, but that's on a best-effort basis. Disabled by default.
extras_testing is for those additional community-provided RPMs while still at an early experimental state. Disabled by default, of course.
builddeps contains extra RPMs needed for building other packages. Disabled by default. Will probably get merged into base and updates in due time, but we're not ready for that yet.

Let's unfold the last levels: standard repository structure from CentOS 7.

.
├── 7
│   ├── 7.4
│   ├── 7.5
│   │   ├── builddeps
│   │   ├── extras
│   │   ├── extras_testing
│   │   ├── base
│   │   ├── updates
│   │   │   ├── Source
│   │   │   │   ├── repodata
│   │   │   │   └── SPackages
│   │   │   │       └── source-RPMs-are-here.src.rpm
│   │   │   └── x86_64
│   │   │       ├── Packages
│   │   │       │   └── RPMs-are-here.rpm
│   │   │       └── repodata
│   │   └── updates_testing
│   ├── 7.6
│   ├── 7.7
│   ├── dev
│   └── latest
└── 8

So, how does it look?

r1

stormi said in Structure of the RPM repositories:

community_testing is for those community-provided RPMs while still at an experimental state. Disabled by default, of course.

Im no expert of this but looks ok and usable to me except I feel community and community_testing are same because both are going to be supported by community only and pretty much always be in an experimental stage.

Another suggestion, instead of community, you may want to mention "experimental" as it warns user. The project itself is a community-driven project and may not need another subdirectory showing "community" as some external set of members.

My 2c.

stormi

r1 said in Structure of the RPM repositories:

Im no expert of this but looks ok and usable to me except I feel community and community_testing are same because both are going to be supported by community only and pretty much always be in an experimental stage.

This may be true. I considered that community would mean "unsupported, but people used it with success" whereas community_testing would mean "untested, except by me". If you've got some driver RPM already in use by several in community and want to test a risky change, having community_testing available for that could be useful.

But we could also start without and add it later if need be.

Another suggestion, instead of community, you may want to mention "experimental" as it warns user. The project itself is a community-driven project and may not need another subdirectory showing "community" as some external set of members.

You're right. I thought of it and changed the name several times while writing the document but I finally came back to community, though it's probably not the best choice. My concern with "experimental" was that there would probably be some officially unsupported packages in it that would actually get good support from those who provide them. On the plus side, its name conveys an unambiguous meaning of "unsupported". On the minus side, it's hard to understand the difference between experimental and updates_testing without an explanation I think.

Other options:

extras
lab
anything else?

r1

extras is fine by me. A readme file in that directory with sufficient warning to make user alert but not scare away & a file with list of package => user xcp-ng profile link will help.

stormi

Let call it extras then. Ok for the readme and credits files.

stormi

New repository structure is online, without any stable branch yet (since 7.4.1 used a different structure).

https://updates.xcp-ng.org/7/dev/

Not usable yet for testing, but I invite you to start using it for RPM building and to report issues.

[xcp-ng-main]
name=Main repository for XCP-ng
baseurl=https://updates.xcp-ng.org/7/dev/main/x86_64/
gpgcheck=0
priority=1

[xcp-ng-builddeps]
name=Build dependencies for XCP-ng
baseurl=https://updates.xcp-ng.org/7/dev/builddeps/x86_64/
gpgcheck=0
priority=2

Note that if you install the yum-plugin-priorities package, you'll probably meet some errors when trying to install -devel packages from CentOS repositories. But that's a good thing in my opinion because that encourages us to rebuild the source RPMs corresponding to the exact version of packages in xcp-ng-main, to avoid build errors related to API changes (such as that of libdrm that breaks the build of qemu-dp).

And if someone very brave manages to make an ordered list of all the source RPMs by build order, in order to be able to rebuild them all without CentOS repositories at all, I promise to rebuild them and push them to the builddeps repo!
Actually, I think we might have to import some source RPMs from CentOS to boostrap the process, but afterwards we would have a solid build environment for 7.5

r1

stormi said in Structure of the RPM repositories:

And if someone very brave manages to make an ordered list of all the source RPMs by build order, in order to be able to rebuild them all without CentOS repositories at all, I promise to rebuild them and push them to the builddeps repo!

Only Citrix knows it, see if you can get this order from @johnelse.

stormi

We can infer most of it from the BuildRequires of the source RPMs actually.

r1

I tried it once but it gave me cyclic dependencies. Im new to the process. YMMV.

stormi

Yes, you usually need a phase called "bootstrapping" in the distros world where you start with some prebuilt binaries of A in order to be able to build B which will in turn be used to rebuild A

stormi

Updated: main renamed to base.