diff --git a/content/en/flux/cheatsheets/bootstrap.md b/content/en/flux/cheatsheets/bootstrap.md index 6702abd19..68d79b37a 100644 --- a/content/en/flux/cheatsheets/bootstrap.md +++ b/content/en/flux/cheatsheets/bootstrap.md @@ -2,7 +2,7 @@ title: "Bootstrap cheatsheet" linkTitle: "Bootstrap" description: "Showcase various configurations of Flux controllers at bootstrap time." -weight: 30 +weight: 29 --- ## How to customize Flux @@ -193,6 +193,11 @@ patches: name: "(kustomize-controller|helm-controller|source-controller)" ``` +{{% alert color="info" title="Horizontal scaling" %}} +When vertical scaling is not an option, you can use sharding to horizontally scale +the Flux controllers. For more details please see the [sharding guide](sharding.md). +{{% /alert %}} + ### Persistent storage for Flux internal artifacts Flux maintains a local cache of artifacts acquired from external sources. diff --git a/content/en/flux/cheatsheets/sharding.md b/content/en/flux/cheatsheets/sharding.md new file mode 100644 index 000000000..26b96cbc3 --- /dev/null +++ b/content/en/flux/cheatsheets/sharding.md @@ -0,0 +1,325 @@ +--- +title: "Controller sharding and horizontal scaling" +linkTitle: "Sharding" +description: "Showcase how to configure sharding of Flux controllers at bootstrap time." +weight: 30 +--- + +When Flux is managing tens of thousands of applications, it is advised to adopt +a sharding strategy to spread the load between multiple instances of Flux controllers. +To enable horizontal scaling, each controller can be deployed multiple times +with a unique label selector which is used as the sharding key. + +What follows is a guide on how to bootstrap multiple controller instances and how to +shard the reconciliation of Flux resources using the `sharding.fluxcd.io/key` label. + +## Bootstrap with sharding + +At [bootstrap time](bootstrap.md), you can define the number of shards +and spin up a Flux controller instance for each shard. + +First you'll need to create a Git repository and clone it locally, then +create the file structure required by bootstrap with: + +```sh +mkdir -p clusters/my-cluster/flux-system +touch clusters/my-cluster/flux-system/gotk-components.yaml \ + clusters/my-cluster/flux-system/gotk-sync.yaml \ + clusters/my-cluster/flux-system/kustomization.yaml +``` + +Then you'll create a dedicated directory inside `flux-system` for each shard: + +```sh +mkdir -p clusters/my-cluster/flux-system/shard1 +touch clusters/my-cluster/flux-system/shard1/kustomization.yaml +``` + +### Configure controller sharding + +In the `shard1` directory generate a set of controller deployments that +will reconcile the Flux resources labels with `sharding.fluxcd.io/key: shard1`. + +To spin up a dedicated source-controller, kustomize-controller and helm-controller instance, +use the following patches in `clusters/my-cluster/flux-system/shard1/kustomization.yaml`: + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +namespace: flux-system +resources: + - ../gotk-components.yaml +nameSuffix: "-shard1" +commonAnnotations: + sharding.fluxcd.io/role: "shard" +patches: + - target: + kind: (Namespace|CustomResourceDefinition|ClusterRole|ClusterRoleBinding|ServiceAccount|NetworkPolicy) + labelSelector: "app.kubernetes.io/part-of=flux" + patch: | + apiVersion: v1 + kind: all + metadata: + name: all + $patch: delete + - target: + labelSelector: "app.kubernetes.io/component=notification-controller" + patch: | + apiVersion: v1 + kind: all + metadata: + name: all + $patch: delete + - target: + kind: Deployment + name: (image-reflector-controller|image-automation-controller) + patch: | + apiVersion: v1 + kind: Deployment + metadata: + name: all + $patch: delete + - target: + kind: Service + name: source-controller + patch: | + - op: replace + path: /spec/selector/app + value: source-controller-shard1 + - target: + kind: Deployment + name: source-controller + patch: | + - op: replace + path: /spec/selector/matchLabels/app + value: source-controller-shard1 + - op: replace + path: /spec/template/metadata/labels/app + value: source-controller-shard1 + - op: replace + path: /spec/template/spec/containers/0/args/6 + value: --storage-adv-addr=source-controller-shard1.$(RUNTIME_NAMESPACE).svc.cluster.local. + - target: + kind: Deployment + name: kustomize-controller + patch: | + - op: replace + path: /spec/selector/matchLabels/app + value: kustomize-controller-shard1 + - op: replace + path: /spec/template/metadata/labels/app + value: kustomize-controller-shard1 + - target: + kind: Deployment + name: helm-controller + patch: | + - op: replace + path: /spec/selector/matchLabels/app + value: helm-controller-shard1 + - op: replace + path: /spec/template/metadata/labels/app + value: helm-controller-shard1 + - target: + kind: Deployment + name: (source-controller|kustomize-controller|helm-controller) + patch: | + - op: add + path: /spec/template/spec/containers/0/args/- + value: --watch-label-selector=sharding.fluxcd.io/key=shard1 +``` + +The above configuration will generate three deployments `source-controller-shard1`, +`kustomize-controller-shard1` and `helm-controller-shard1` all configured +with `--watch-label-selector=sharding.fluxcd.io/key=shard1`. + +To enable these deployments at bootstrap, add the `shard1` directory to +the `clusters/my-cluster/flux-system/kustomization.yaml` resources: + +```yaml +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization +resources: +- gotk-components.yaml +- gotk-sync.yaml +- shard1 +patches: + - target: + kind: Deployment + name: "(source-controller|kustomize-controller|helm-controller)" + annotationSelector: "sharding.fluxcd.io/role notin (shard)" + patch: | + - op: add + path: /spec/template/spec/containers/0/args/0 + value: --watch-label-selector=sharding.fluxcd.io/key notin (shard1, shard2) +``` + +Note how this configuration excludes the sharding keys from the main controllers watch with +`--watch-label-selector=sharding.fluxcd.io/key notin (shard1, shard2)`. This ensures +that the main controllers will not reconcile any Flux resources labels with the sharding keys. + +### Install and Upgrade shards + +Push the changes to main branch: + +```sh +git add -A && git commit -m "init flux" && git push +``` + +And run the bootstrap for `clusters/my-cluster`: + +```sh +flux bootstrap git \ + --url=ssh://git@// \ + --branch=main \ + --path=clusters/my-cluster +``` + +Verify that the main controllers and the ones assigned to shard1 are running: + +```console +$ kubectl -n flux-system get deployments + +NAME READY UP-TO-DATE AVAILABLE AGE +helm-controller 1/1 1 1 1m +helm-controller-shard1 1/1 1 1 1m +kustomize-controller 1/1 1 1 1m +kustomize-controller-shard1 1/1 1 1 1m +notification-controller 1/1 1 1 1m +source-controller 1/1 1 1 1m +source-controller-shard1 1/1 1 1 1m +``` + +When upgrading Flux, either by rerunning bootstrap with a newer version or +by using the [Flux GitHub Actions](https://github.com/fluxcd/flux2/tree/main/action), +the sharded controllers will be automatically upgraded along with the main ones. + +## Assign resources to shards + +To assign a group of Flux resources to a particular shard, label +them with `sharding.fluxcd.io/key`. + +For example, assuming you want to assign the reconciliation of an application +to the `shard1` controllers, label both the Flux source and its Kustomization +with `sharding.fluxcd.io/key: shard1`: + +```yaml +--- +apiVersion: source.toolkit.fluxcd.io/v1 +kind: GitRepository +metadata: + name: podinfo + namespace: default + labels: + sharding.fluxcd.io/key: shard1 +spec: + interval: 10m + url: https://github.com/stefanprodan/podinfo + ref: + semver: 6.x +--- +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: podinfo + namespace: default + labels: + sharding.fluxcd.io/key: shard1 +spec: + interval: 10m + targetNamespace: default + sourceRef: + kind: GitRepository + name: podinfo + path: ./kustomize + prune: true +``` + +Note that Source object kinds which have a dependency on another kind +(i.e. `HelmChart` on a `HelmRepository`) need to have the same labels +applied to work as expected. + +For example, assuming you want to assign the reconciliation of a Helm release +to the `shard1` controllers, label the HelmRelease, its chart and its repository +with `sharding.fluxcd.io/key: shard1`: + +```yaml +apiVersion: source.toolkit.fluxcd.io/v1beta2 +kind: HelmRepository +metadata: + name: podinfo + namespace: default + labels: + sharding.fluxcd.io/shard: shard1 +spec: + interval: 10m + type: oci + url: oci://ghcr.io/stefanprodan/charts +--- +apiVersion: helm.toolkit.fluxcd.io/v2beta1 +kind: HelmRelease +metadata: + name: podinfo + namespace: default + labels: + sharding.fluxcd.io/key: shard1 +spec: + interval: 10m + chart: + metadata: + labels: + sharding.fluxcd.io/key: shard1 + spec: + chart: podinfo + sourceRef: + kind: HelmRepository + name: podinfo +``` + +Note that with `.spec.chart.metadata.labels` we set the sharding key on the +generated Flux `HelmChart` object, so that both the Helm repository and charts +are managed by source-controller-shard1 instance. + +### Bulk assign shards + +Instead of manually labeling each Flux resource with a shard key, use a top-level +Flux Kustomization and automatically label all resources. + +For example, assuming you want to assign a tenant to a particular shard, in the +root Flux Kustomization that reconcile the tenant's Flux sources, kustomizations and +Helm releases label these resources as follows: + +```yaml +apiVersion: kustomize.toolkit.fluxcd.io/v1 +kind: Kustomization +metadata: + name: tenant1 + namespace: tenant1 +spec: + interval: 10m0s + path: ./apps + prune: true + sourceRef: + kind: GitRepository + name: tenant1 + targetNamespace: tenant1 + commonMetadata: + labels: + sharding.fluxcd.io/key: shard1 + patches: + - target: + kind: HelmRelease + patch: | + apiVersion: helm.toolkit.fluxcd.io/v2beta1 + kind: HelmRelease + metadata: + name: all + spec: + chart: + metadata: + labels: + sharding.fluxcd.io/key: shard1 +``` + +With `.spec.commonMetadata.labels` we set the shading key on all +the Flux resources and with the `.spec.patches` we set the same shading key +for all generated `HelmCharts`.