[May] gix repo exclude query` and full access to work trees

[Byron]

As much as the last month felt like there was incredible and divers progress, this month felt much more tame in comparison. Maybe we learn why that is as well, let's check.

The worktree checkout block

In the quest to resolve attributes for paths being checked out, I decided it should be easier to first learn how its done by implementing exclude file matching, allowing to answer questions like is_excluded(path) - simple enough, right? Little did I know… .

Finalization of exclude file matching in git-attributes

Exclude file handling is complex as it requires you to deal with

• immutable global excludes consisting of one or more files
• a mutable .gitignore file stack which changes depending on the path that is matched
• immutable global command-line overrides with additional patterns

To add even more complexity, I realized quite late in the game that matching patterns like target/ is actually special as it's implemented by matching against directories while the .gitignore stack is built. Then when matching a file, one actually shortcuts the search by seeing that the containing folder is actually ignored.

The above mechanism as implemented by git comes at a price though: one seems to be unable to use negative excludes on files contained in these folders, which never get a chance to match due to the matching algorithm. It's a shortcoming of the git implementation that can be fixed, and it was interesting for me to rationalize why gitoxide should actually not fix it: gitoxide prefers to also mirror shortcomings if doing so avoids surprises. Half-jokingly I was thinking that maybe one day there is a feature toggle to enable certain fixes per user request.

Cleanup path handling which moved from git-features to git-path

Path handling was always a bit tricky as gitoxide tries hard to not bind itself to any encoding as git seems to do. Over time it turned out that git does in fact assume UTF-8 compatible encodings on windows at least which can fail due to the infamous illformed UTF-8 problem.

From that realization all code that handles paths and needed conversions was directed to use git-features::path::*, which was like a catch-all for this kind of functionality in a crate that could cheaply be shared.

With the rise of the git-discover crate (detailed later), it became necessary (or at least seemed cleaner) to move all path related functionality into the new git-path crate with the added benefit of the up and coming implementation for path-specs (as sketched in the git_path::Spec type).

gix repo exclude query sub-command

As always, every relevant feature has to go through all the layers to become useful and to be able to test it outside of its test-sandboxes. This led to the creation of a subcommand very similar to git check-ignore, but with a few improvements.

It's probably best to show the output differences in a typical Rust project:

➜  target git:(main) git check-ignore foo/bar -nv
.gitignore:5:target/    foo/bar

➜  target git:(main) gix repo exclude query foo/bar
../.gitignore:5:target/ target/foo/bar

gix naturally shows paths to .gitignore files relative to the working directory as one would expect, something git doesn't do as it uses chdir to set itself to the root of the repository. The latter leads to somewhat confusing output of the path that it matches which looks like foo/bar, even though it effectively is target/foo/bar, which is exactly what gix will state here. The latter makes so much more sense when one looks at the pattern that matched it.

As a side-effect to this work, git-config now has a much more convenient API and its usage was revamped within git-repository which now has the notion of 'config-cache' with often-accessed configuration values that are looked up only once and validated during Repository initialization.

Community

Unblockingonefetch release with worktree support

After casually strolling through onefetch I came to the realization that previously it could open and analyze linked worktrees (those created with git worktree add …) whereas now it could not. gitoxide wasn't yet able to handle .git files which link to a worktree private git directory.

I really thought I could add this in a couple of days… at most, and ended up spending quite exactly 14 days on the matter.

The task is essentially two fold. First one has to understand gitdir files which come in two forms to find the locations related to a worktree git directory and detect it as such, and secondly one has to handle references a little differently.

The first matter was resolved relatively quickly, and it helped (but also was a necessity) to move git_repository::discover into its very own git-discover crate. That way, tests could run faster and could generally be more focussed than otherwise. Having repository discovery in a separate crate also opens up new opportunities for tests in plumbing crates, which often refrain from using git-repository in favor of being forced to more directly dog-food various plumbing crate APIs.

The second matter, ref handling, was more involved though and required various refactoring to one seemingly simple change: now there are optionally two roots for references. Those for the work tree itself, i.e. worktree-private references, and the shared ones in the common git repository. A major requirement to make this happen was the need to categorize references, that is looking at their name and deriving some properties for them. The most important one, you guessed it, is whether or not it's worktree-private.

Another big change was reference iteration, which now had to not only iterate packed refs and loose refs in lock-step, but also had to deal with two iteration bases for loose references.

Overall, the implementation didn't give me any trouble, but it took time to understand what needed to be done and write the right tests for all of the interactions I identified.

It might also be interesting that gitoxide implements ref-handling in work-trees as documented and not as implemented by git. It's also made so that all operations support handling those special main-worktree and worktrees/<name> ref prefixes, which git doesn't seem to do. Overall this should make git-ref the go-to implementation for handling references.

As a side-note, this work also clearly showed that implementation ref-tables is a feature that is far in the future and will certainly only ever be motivated by servers. It's entirely unclear how ref-tables would deal with layering worktree private refs on top of common refs, for example, and I have the feeling that it's not meant to do that at all. After all, the server doesn't deal with work-trees so it should be fine.

git-config with preliminary includeIf support

Even though we keep failing creating small PRs that can be merged more often, the same is not true for delivering features, now allowing includeIf to be evaluated. As a side-effect, git-path now allows to flexibly canonicalize paths similar to how git does it, a capability that will certainly replace many occurrences of canonicalize() in the codebase.

git-discover grows up

Despite being able to handle work-trees during discovery, thanks to contributions alone it will now respect ceiling directories and avoid crossing the file system boundary when searching upwards for git repositories.

Launch of new YouTube show named Gifting Gitoxide

Sidney and I decided that it's time to change the format to something that allows him to go more hands-on with gitoxide. Thus the new format is all about reviewing contributions. This I will do with him once a week, but will also record review sessions on other PRs as I see fit. These are probably helpful to those who contributed as well due to the additional insights and train of thought they will provide.

Vergen and additional MSRV learnings

Vergen was the reason for a quest to setting up an MSRV for gitoxide, and it's also the reason I chose to follow the MSRV of the windows crate which uses much more recent compiler versions.

vergen helped to get to that conclusion by not providing any insights (due to a lack of reply) into why they need a lower MSRV.

Now gitoxide will assure there is at lest one recent release before the MSRV is increased which will also cause git-repository to receive a 'breaking-change' indicating version bump.

Outlook

We are still looking to read attributes during checkout, and now that excludes can be matched and showed the way, implementing attribute matching and access will pose no difficulties (and clearly assuming that the attribute logic itself isn't unexpectedly tricky). From there filters can finally be run and I will feel we are finally making progress on properly checking out a repository. From that point of view, this month was full of digressions, but sometimes priorities change temporarily.

Furthermore I expect to have less time for gitoxide development due to two paid jobs coming up and taking priority. Of those one is more permanent so I would expect gitoxide to reduce its velocity by 25% at least, and probably more like 50% in June.

In retrospect, I am very happy to finally have tackled one of the big unknowns that were worktrees, allowing gitoxide to handle more repositories than ever (submodules notwithstanding).

Cheers,
Sebastian

PS: The latest timesheets can be found here.