Add build reverse dependencies research #212
Conversation
majamassarini
force-pushed
the
reverse-dependencies
branch
6 times, most recently
from
1d3004e
to
34fa362
Compare
| --- | ||
| title: Build reverse dependencies in Packit | ||
| authors: mmassari | ||
| --- |
There was a problem hiding this comment.
General comment, what are the project structures that would support such reverse dependencies. Firstly w.r.t. where the builds will live
- copr project with multiple packages
- Easy to pick up the latest changes
- Need to manually clean the project build, otherwise deleted packages will still be used (e.g. a package is already downstream and the old copr package is no longer needed)
- Bootstrap packages would be weird to handle
- multiple copr projects linked via
additional_repos- Maximum control of the packages
- Messy to navigate
- koji based builds
- Good control of packages to be used
- Not universal for upstream projects
There was a problem hiding this comment.
This is not decided yet. However I think it probably will be a new copr project created by Packit (if we will go in the way of using COPR) or a Koji side-tag and in this case, you are right, it could not be for all upstream projects.
There was a problem hiding this comment.
Re: side-tag, that's for the case of making the reverse dependency build a production workflow right? Can scratch builds also have side-tag or equivalent?
There was a problem hiding this comment.
Can scratch builds also have side-tag or equivalent?
You can assign a side tag to a scratch build.
If for production workflow you mean you need to have your commit released in dist-git. This is already supported in Koshei. We are looking for a way to have the workflow triggered before a new release in dist-git.
There was a problem hiding this comment.
I'm thinking w.r.t. the end result. Will it eventually be used in a bodhi update, or will it be used only for testing dependencies. The former might be needed only sporadically, like for SOVERSION bump, etc. but the latter could be used more freely. Also the former could be covered by the dist-git label workflow
| --- | ||
| title: Build reverse dependencies in Packit | ||
| authors: mmassari | ||
| --- |
There was a problem hiding this comment.
Second part of the general comment about the supported project structures. Where are the packages to be rebuilt
- Monorepo
- Easy to control what triggers build chains
- Probably already the discussion of a different research/issue
- Monorepo with external references, i.e. point a package to a git repo and use the
packit.yamlinside there- Good control of what packages to rebuild
- Requires dependent project to have packit integration
- Completely external reference
- Source and build environment can be anything, dist-git, copr project or packit project
- Can dynamically query all dependencies using
fedrq(only in case of dist-git source)
There was a problem hiding this comment.
I don't think there is a need for changing the project structure but I am not sure I am following you.
The reverse-dependencies should be dynamically rebuilt (and also discovered if we don't go in the way of statically defining them).
This is done more easily when we work with Koji and the Fedora ecosystem where all the dependencies should already be satisfied, https://koschei.fedoraproject.org/ already does it and you don't need to specify anything.
If we decide to use COPR and enable every upstream repo we should probably find a way to describe in the new Job the reverse dependencies (I would say statically) and a way to build them in COPR (there could be many approaches for this also the action can be used).
For me this is a "last step", an enhancement in the new Job but if I understand it correctly you would benefit from it because not all your packages are released in Fedora?
There was a problem hiding this comment.
but if I understand it correctly you would benefit from it because not all your packages are released in Fedora?
Yes, this is for pre-release/pre-packaging. Referencing a git-repo under packit would be sufficient for that.
If we decide to use COPR and enable every upstream repo
The main supported upstream model here sould be to build in copr with dist-git source (probably from fedrq).
I don't think there is a need for changing the project structure
I was thinking of the structure of monorepo, just because it would be more natural to navigate a dependency tree and it would share much of the scheduling logic that is needed for monorepo dependencies.
Indeed an alternative would be to have it in Jobs, but I'm afraid complex dependencies would become harder to manage.
These would be another upstream model for when some dependencies are not yet packaged.
| 1. a way to resolve reverse dependencies | ||
|
|
||
| - we can follow the [Koschei way](https:/fedora-infra/koschei/blob/master/koschei/backend/depsolve.py) which uses `hawkey` from `libdnf` | ||
| - or we can follow the [Mass prebuild way](https://gitlab.com/fedora/packager-tools/mass-prebuild/-/blob/main/mass_prebuild/utils/whatrequires.py?ref_type=heads): given a package or a set of packages, namely "main packages", the mass pre-builder will calculate the list of its direct reverse dependencies: packages that explicitly mark one of the main packages in their "BuildRequires" field. As far as I can understand it, this is a **less deep** and thus **less complete** approach compared with the Koschei solution. |
There was a problem hiding this comment.
Related to option 2 here. I think fedrq would be a more manageable alternative. Not sure about managing copr or non Fedora dependencies though, but maybe the tool can evolve there.
There was a problem hiding this comment.
Thanks, I will give it a look.
| - or we can follow the [Mass prebuild way](https://gitlab.com/fedora/packager-tools/mass-prebuild/-/blob/main/mass_prebuild/utils/whatrequires.py?ref_type=heads): given a package or a set of packages, namely "main packages", the mass pre-builder will calculate the list of its direct reverse dependencies: packages that explicitly mark one of the main packages in their "BuildRequires" field. As far as I can understand it, this is a **less deep** and thus **less complete** approach compared with the Koschei solution. | ||
| - we can **avoid calculating dependencies** and let the user specify them in a statistical way. | ||
|
|
||
| 2. a way to schedule builds in Koji or COPR (or both). |
There was a problem hiding this comment.
This point 2 is needed more generally, e.g. for monorepo
| --- | ||
| title: Build reverse dependencies in Packit | ||
| authors: mmassari | ||
| --- |
There was a problem hiding this comment.
(self-promotion) I was starting to implement this as a tmt-plugin (tmt-copr and also tmt-koji, but need to finish 1 first :D)
- Still requires testing-farm integration to pull in additional tmt plugins
- Less to manage on packit side
- More control on how to display results, logs
- Needs the secret management of testing-farm to get the api token to interact with services. But an alternative could incorporate packit cli and packit service to get things working faster
This can solve some of the issues of "Build our own solution", especially w.r.t. viewing the results
| - [mini mass rebuild](https:/hroncok/mini-mass-rebuild) | ||
| - [copr-autorebuilder](https://pagure.io/copr-autorebuilder/tree/master) | ||
| - [copr-rebuild-tools](https:/fedora-copr/copr-rebuild-tools/tree/main) |
There was a problem hiding this comment.
I assume these would probably not be a good fit since they are not included, right?
There was a problem hiding this comment.
Yes, they are mass rebuild tools not focused on reverse dependencies.
We can give them a further look if we decide to implement our own solution and we want to investigate more on how to submit a large number of builds.
| Even if it is a CLI tool we can not easily integrate it in packit-service because of two main design choices: | ||
|
|
||
| - the tool interacts with a database | ||
| - the tool keeps a session and can interact with the user |
There was a problem hiding this comment.
so there is no way of disabling this interactive mode?
There was a problem hiding this comment.
yes, you can decide to submit one step at time, specifying which step you want, but still the code is highly coupled with the "entire session idea" and the database.
|
|
||
| - but we should be aware that **the mass prebuild tool is designed in a way that lets you interact with every single step of the long rebuild process**. I don't know exactly why, but if I got it correctly, this process is error prone and the user may like to be able to continue it from a point of failure or may like to resubmit a failed build. I think it is worth discussing with the mass pre build developer why he has chosen to go this way. | ||
|
|
||
| If we want the user to be able to interact with every step in this long process we should probably be capable of dealing with **job dependencies**. |
There was a problem hiding this comment.
I hope the more straightforward way, i.e. using batch builds/ chain builds without the need to solve the dependencies ourselves will be enough. We should probably get feedback from the potential users for this so that we can decide.
Update research/build-reverse-dependencies/index.md Co-authored-by: Laura Barcziová <[email protected]>
majamassarini
force-pushed
the
reverse-dependencies
branch
from
e4b964b
to
6f9c8bd
Compare
Fix (partially) packit/packit-service#2361