KubeAccessHowTo

Accessing CGR's Kubernetes

CU Get Reg is currently migrating to Kubernetes for better developer experience and tooling.

If you need to work on CGR's Infrastructure (e.g. Service Routing, Container Deployment, and Disaster Management), then you might need to access CGR's K8S cluster.

This document will guide you through the process of gaining access to the cluster.

To access the cluster one must:

1. Be a member of thinc-org/cugetreg-infra GitHub Team.
2. Installed and setup kubelogin to authenticate kubectl command.

The following sections will guide you through this process.

Setting Up kubectl

1. Install kubelogin
2. Place this content to ~/.kube/config

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTmpJNE56WXpPRGd3SGhjTk1qSXdPVEV4TURZd05qSTRXaGNOTXpJd09UQTRNRFl3TmpJNApXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTmpJNE56WXpPRGd3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFSbnV1SUR1a2k1NWlWd2k3YVJkUC9MdjNVTGdZS3BxV1QxTzZuQW85WkkKNTJUdXNZVWorWE9TMFZIdDdNbkxQK01QTzBCVGlCcGtBMkVUa1QxWGMwbVRvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVUMzYlRrTFgxSW5qaW5YMGZiZ1pvCndjcXZCNjB3Q2dZSUtvWkl6ajBFQXdJRFJ3QXdSQUlnYU9VdzJ2UWlYK2xiMFY5N1A5VVhiYmlnWUNrRkRaOU0KRUxFTS9IaUk5b2tDSUdxQjdLaVlZV1VBMElXQUdwK3RWVTBWdHBITThiNkRzWUFjSlhJQ294NlgKLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo=
    server: https://kube.internal.cugetreg.com
  name: default
contexts:
- context:
    cluster: default
    user: oidc
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: oidc
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - oidc-login
      - get-token
      - --oidc-issuer-url=https://auth.internal.cugetreg.com/application/o/kube-login/
      - --oidc-client-id=3d5e451eed977426857e466a95e49c8ad67650f6
      - --oidc-extra-scope=profile
      command: kubectl
      env: null
      provideClusterInfo: false

This will setup a proper OIDC flow.

3. Done
4. Now, try kubectl get nodes. You should see login page.

Emergency Kubernetes Access

In case that GitHub is down, Authentik is down, or other authentication failure.

There exist a kubeconfig named k3s.yaml which contains X509 certificate capable of authenticating against API Server without going though OIDC flow.

You can also do sudo kubectl inside the CGR's Server is you have Root access.

Revoking Permission

To remove someone from group permission, first remove them from GitHub group, then remove them from Authentik's Directory.

Removing just either of them will not work!

Implementation Detail

CU Get Reg uses K3S hosted on Reg Chula's single VM to provides K8S functionalities.

The K8S API Server endpoint is exposed at https://kube.cugetreg.com. The API is set-up to accept OIDC Token provided by CGR's OIDC provider (Authentik) at https://auth.frontier.cugetreg.com running inside the cluster itself.

The authentication flow is as followed:

1. kube-login is triggered by kubectl command
2. It launch CGR's Authentik in your browser
3. CGR's Authentik return OIDC token containing groups membership
4. Token is presented to Kube API Server. Kube API look at groups membership and determine roles

the cluster contains ClusterRoleBinding for those in Authentik group cugetreg-infra to bind with role cluster-admin.

Todo

2. Remove Let's Encrypt Staging from CGR's Server Cert Store