Implement drift detection and correction for cluster state

[stefanprodan]

This PR implements Flux readiness checks and drift detection for the cluster state. The provider applies changes to the Flux components and GitRepository/Kustomization manifests on the cluster, thus enabling changes to Git URL and branch to be actuated. It also adds a check for verifying the kubeconfig during planning phase.

Description

Changes:

• Detect is Flux is running in the cluster and if it is ready during planning phase.
• Restore Flux on a cluster if readiness checks fails.
• Apply changes in configuration directly on the cluster on updates.
• Disallow changes to the bootstrap path field in the same way the CLI does it (breaking change).

Motivation and Context

Being able to update Flux in the cluster by detecting drift in the cluster state during planning and apply.

Fix: #500
Fix: #656
Fix: #653
Fix: #564
Fix: #176
Fix: #499

How has this been tested?

• Have you added an acceptance test for the functionality being added?
• Have you run the acceptance tests on this branch?

Manual testing for:

• change Git branch
• change Git URL
• cluster will Flux controllers deleted
• cluster with Flux GitRep object delete
• recreated cluster without Flux outside of TF

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Documentation

• I have updated the documentation (if required) with make docs

Checklist:

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I've read the CONTRIBUTION guide
• I have signed-off my commits with git commit -s

[stefanprodan]

@swade1987 it would be great if you could test this PR on your clusters, especially the kubeconfig validation during planning.

[swade1987]

@stefanprodan I'll take this for a spin next week and keep you posted.

[swade1987]

@swade1987 it would be great if you could test this PR on your clusters, especially the kubeconfig validation during planning.

Looking good @stefanprodan

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Get "https://xxxx/api/v1/namespaces/flux-system": dial tcp: lookup XXXX: no such host
│
│   with kubernetes_namespace.flux_system,
│   on main.tf line 67, in resource "kubernetes_namespace" "flux_system":
│   67: resource "kubernetes_namespace" "flux_system" {
│
╵

[swade1987]

Changing the git branch works as well (as long as the branch exists in GitHub) (see below)

provider "flux" {
  kubernetes = {
    host                   = kind_cluster.this.endpoint
    client_certificate     = kind_cluster.this.client_certificate
    client_key             = kind_cluster.this.client_key
    cluster_ca_certificate = kind_cluster.this.cluster_ca_certificate
  }
  git = {
    url = "ssh://[email protected]/${var.github_org}/${var.github_repository}.git"
    branch = "test-branch"
    ssh = {
      username    = "git"
      private_key = tls_private_key.flux.private_key_pem
    }
  }
}

Ran terraform apply then ...

flux export source git flux-system | grep branch
    branch: test-branch

[swade1987]

LGTM @stefanprodan, I ran a number of tests locally and things look good.