Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor bootstrap to make use of go-git-providers #968

Merged
merged 11 commits into from
Apr 7, 2021

Conversation

hiddeco
Copy link
Member

@hiddeco hiddeco commented Feb 23, 2021

This pull request changes the implementation used to connect to Git
providers like GitHub from fluxcd/pkg/git to fluxcd/go-git-providers,
eliminating quite some bugs that were present in the GitLab integration.

In addition to this, the bootstrap logic has been factored out into an
(for now) internal package with two bootstrap "runners" PlainGitBootstrapper
and GitProviderBootstrapper. The GitProviderBootstrapper is a superset
of the PlainGitBootstrapper that besides Reconciler also implements the
RepositoryReconciler.

The Git actions rely on an interface, making it easier to support other
implementations than go-git at a later moment, to for example support
bootstrapping to Git servers that only support the v2 protocol.

By doing this, it was possible to introduce a new flux bootstrap git
command that bootstraps Flux to a plain Git repository and applies the
resources to the cluster, prompting the user to upsert the public key
by hand if a private key was generated.

As a finishing touch, almost all options are now configurable like they
already were for the flux <install|create> commands, making it possible
to have much finer grain configurations like e.g. the Git author that
makes the commit, the type of key that is generated, adding a [skip ci]
appendix to the commit message, and many more.

Future enhancement possibilities:

  • Assign different rights than maintainer to GitHub and GitLab teams
  • Provide another Git implementation so bootstrapping to e.g. Azure DevOps is possible
  • ...

Help testing this PR 🦸

  1. Clone the repository and checkout the branch:

    $ git clone https:/fluxcd/flux2.git
    $ cd flux2
    $ git checkout -b go-git-providers-bootstrap origin/go-git-providers-bootstrap
  2. Build the binary (requires Go 1.16 and Kustomize):

    $ make build
  3. Run one of the bootstrap commands:

    $ ./bin/flux bootstrap git --help
    $ ./bin/flux bootstrap gitlab --help
    $ ./bin/flux bootstrap github --help

Ref list

Fixes #799
Fixes #835
Fixes #916
Fixes #924
Fixes #1021
Fixes #1129
Fixes #1143

Addresses part of #43

@hiddeco hiddeco force-pushed the go-git-providers-bootstrap branch 16 times, most recently from daf35f3 to 9123fee Compare March 9, 2021 08:23
@hiddeco hiddeco force-pushed the go-git-providers-bootstrap branch 4 times, most recently from 09808bd to 655d8ab Compare March 15, 2021 10:36
@somtochiama
Copy link
Member

The --owner flag description should highlight that it only accepts the actual path / textual path for GitLab groups.

Great work @hiddeco 😁. It has been working okay so far.

@hiddeco
Copy link
Member Author

hiddeco commented Mar 30, 2021

From @somtochiama's testing:

$ flux bootstrap gitlab --owner=root --repository=fleet-infra --branch=master --path=./staging-cluster/flux-system --hostname=https://custom --personal

returns

✔ repository "https://https://example.cm/root/fleet-infra" reconciled
► cloning branch "master" from Git repository "https://https://example.com/root/fleet-infra.git"
✗ failed to clone repository: Get "https://https://example.com/root/fleet-infra.git/info/refs?service=git-upload-pack": dial tcp: lookup https on 192.168.0.1:53: no such host

but running without https in --hostname:

► connecting to example.com
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x23a10e8]
goroutine 1 [running]:
github.com/fluxcd/go-git-providers/gitprovider/cache.(*cacheRoundtripper).RoundTrip(0xc000428120, 0xc00094c200, 0xc000428120, 0x0, 0x0)
	../go/pkg/mod/github.com/fluxcd/[email protected]/gitprovider/cache/httpcache.go:71 +0xc8
net/http.send(0xc00094c200, 0x2c3d680, 0xc000428120, 0x0, 0x0, 0x0, 0xc000428af8, 0x203000, 0x1, 0x0)
	/usr/local/go/src/net/http/client.go:251 +0x454
net/http.(*Client).send(0xc0008b4030, 0xc00094c200, 0x0, 0x0, 0

Likely related to fluxcd/go-git-providers#55

@hiddeco hiddeco force-pushed the go-git-providers-bootstrap branch 3 times, most recently from 46e5fab to e71dc06 Compare April 6, 2021 10:28
@hiddeco hiddeco marked this pull request as ready for review April 6, 2021 11:13
@hiddeco hiddeco force-pushed the go-git-providers-bootstrap branch 3 times, most recently from eb9afc4 to dae8230 Compare April 6, 2021 15:03
This commit factors out the bootstrap logic into a new `bootstrap`
package, while also moving to `go-git-providers` to handle things
around Git providers (e.g. repository creation, deploy key
upsertions).

The `GitProviderBootstrapper` is a superset of the
`PlainGitBootstrapper` that besides `Reconciler` also implements the
`RepositoryReconciler`.

The Git actions rely on an interface, making it easier to support
other implementations than `go-git` at a later moment, to for example
support bootstrapping to Git servers that only support the v2 protocol.

Signed-off-by: Hidde Beydals <[email protected]>
To be used in a future version of Flux to better select Flux components
in a namespace, as the namespace value for the
`app.kubernetes.io/instance` could be used by non Flux related
workloads.

Signed-off-by: Hidde Beydals <[email protected]>
This includes making a lot of things configurable (e.g. SSH key
algorithm, RSA bit size, etc.) that used to be static.

Signed-off-by: Hidde Beydals <[email protected]>
This command makes it possible to bootstrap to a generic Git server
using the local SSH agent, or a given password or private key file.

If a private key is generated, the user is prompted to give the
generated key access to the repository.

Signed-off-by: Hidde Beydals <[email protected]>
Using the `--commit-message-appendix` flag a string can be added to the
commit messages made by the bootstrapper process to for example skip CI
actions from executing using e.g. `[skip ci]`.

Signed-off-by: Hidde Beydals <[email protected]>
With this commit comes a lot of evil.

Signed-off-by: Hidde Beydals <[email protected]>
We have observed that the code at times outperforms GitHub mechanics,
resulting in not found errors that are only true for a millisecond.
Retrying those actions once with a 2 second delay should be more
friendly to users.

Signed-off-by: Hidde Beydals <[email protected]>
@hiddeco hiddeco force-pushed the go-git-providers-bootstrap branch from dae8230 to 7481c6b Compare April 7, 2021 08:28
Copy link
Member

@stefanprodan stefanprodan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Awesome work @hiddeco 🥇 🎖️ 🏅

@hiddeco hiddeco merged commit 0d2f6bf into main Apr 7, 2021
@hiddeco hiddeco deleted the go-git-providers-bootstrap branch April 7, 2021 08:40
ybelleguic pushed a commit to ybelleguic/flux2 that referenced this pull request Jan 9, 2023
build: Add postbuild script for fuzzing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/bootstrap Bootstrap related issues and pull requests enhancement New feature or request
Projects
None yet
4 participants