Skip to content

observeinc/helm-charts

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Observe Helm Charts

This repository contains Helm charts for installing the telemetry agents required for Observe apps on Kubernetes.

Contents:

  • stack: Installs several agents required for the Kubernetes Observe app
    • logs (provided by fluent-bit)
    • metrics (provide by grafana-agent)
    • events (kubernetes state events)
  • traces: Installs trace collection for the OpenTelemetry Observe app

Quick Start

First, install and update the observe helm repository:

helm repo add observe https://observeinc.github.io/helm-charts
helm repo update

Namespace

We currently require that Observe charts are installed to an observe namespace. The namespace should further be annotated with a human-readable cluster name. Helm supports creating namespaces as a convenience feature, but does not support managing and configuring the namespace (that is, the --create-namespace option in helm install is equivalent to manually running kubectl create namespace). Thus it is recommended to manage the namespace externally to Helm. This can likely be done by following the same methodology you use to manage the creation of your Kubernetes clusters, and/or your other namespaces.

You can also manage the namespace manually using the following commands:

CLUSTER_NAME="My Cluster"
kubectl create namespace observe && \
	kubectl annotate namespace observe observeinc.com/cluster-name="$CLUSTER_NAME"

Installing Stack

The values for ${OBSERVE_COLLECTION_ENDPOINT} and ${OBSERVE_TOKEN} are provided when you install the Kubernetes app in Observe, and set up a new connection.

helm install --namespace=observe observe-stack observe/stack \
	--set global.observe.collectionEndpoint="${OBSERVE_COLLECTION_ENDPOINT}" \
	--set observe.token.value="${OBSERVE_TOKEN}"

# store values for further configuration and upgrades
helm -n observe get values observe-stack -o yaml > observe-stack-values.yaml

Installing Traces

The values for ${OBSERVE_COLLECTION_ENDPOINT} and ${OBSERVE_TOKEN} are provided when you install the OpenTelemetry app in Observe, and set up a new connection.

The same token should not be re-used for the stack (Kubernetes) and traces (OpenTelemetry) charts. Instead, create a new connection for the OpenTelemetry app, and provide the new token you are prompted to create.

helm install --namespace=observe observe-traces observe/traces \
	--set global.observe.collectionEndpoint="${OBSERVE_COLLECTION_ENDPOINT}" \
  --set observe.token.value="${OBSERVE_TOKEN}" \
  --create-namespace


# store values for further configuration and upgrades
helm -n observe get values observe-traces -o yaml > observe-traces-values.yaml

Configuration

Required Values

You must set global.observe.collectionEndpoint, which is provided when configuring a connection in Observe. To have Helm manage a Kubernetes secret containing your datastream token, you must also set observe.token.value. Otherwise, you must set observe.token.create to false, and manually create the secrets (see Managing Secrets Manually).

These values should be persisted in a custom values file:

global:
  observe:
    # A unique URL associated with your customer ID, provided in the new connection installation instructions.
    collectionEndpoint: "https://123456789012.collect.observeinc.com"

observe:
  token:
    # The Datastream token. This will typically be unique to each chart being installed, or each release of
    # of a chart.
    value: <datastream token>

Kubernetes Event Batch Limits

You can add -collector-listerwatcher-limit batch_size to the kube-state-events container args to adjust the batch size (default is 500). This can reduce initial memeory usage, which may allow you to run a smaller container.

Sizing

While the default configuration of the observe charts are intended to be appropriate for a variety of use cases, additional example configurations are also provided for other cases:

Resource allocations for each example configuration are as follows:

xs m l xl
events 20m
64Mi
50m
256Mi
200m
1Gi
400m
2Gi
logs 10m
64Mi
100m
128Mi
200m
192Mi
500m
256Mi
metrics 50m
256Mi
250m
2Gi
500m
4Gi
200m (ds)
1Gi
xs m deployment
traces 50m (ds)
128Mi
250m (ds)
256Mi
250m (x10)
256Mi

By default, logs and traces are daemonsets, while events and metrics are single-replica deployments. Some use cases may require scaling the metrics agent beyond a single replica; in this case we recommend using a daemonset.

Advanced Configuration

The subcharts managed by the parent stack and traces charts can be further configured according to the upstream charts. Refer to the documentation for these charts directly for advanced configuration:

Keep in mind that these upstream charts are managed by Observe's parent charts, which provide the default configuration necessary to publish the data to Observe. This means that changes to the underlying upstream charts must be specified in the appropriate YAML blocks.

Stack:

logs:
  fluent-bit:
    # advanced fluent-bit configuration overrides begin at this level of indentation

metrics:
  grafana-agent:
    # advanced grafana-agent configuration overrides begin at this level of indentation

Traces:

opentelemetry-collector:
  # advanced opentelemetry-collector configuration overrides begin at this level of indentation

Managing Secrets Manually

If you do not wish to have Helm manage your token as a Kubernetes secret (which implies that it will be stored as a chart value), you can manage it manually. To prevent the chart from managing the secret, first set observe.token.create to false.

Then, ensure a secret exists in the observe namespace with the correct name:

Stack

kubectl -n observe create secret generic credentials --from-literal='OBSERVE_TOKEN=<kubernetes datastream token>'

Extra configuration files for fluent-bit

You can also add extra fluent-bit configuration files like this:

logs:
  fluent-bit:
    config:
      extraFiles:
        systemd.conf: |
          [INPUT]
              Name                systemd
              Tag                 systemd
              Read_From_Tail      on

Every file in the extraFiles section will be mounted in the /fluent-bit/etc/custom-configs directory. These files will be included in the main fluent-bit.conf file by @INCLUDE directive which is configured as part of outputs default configuration value (for some reason fluent-bit subchart doesn't include these files by default).

If you're overriding the outputs default configuration, you can include the extra files in your custom configuration file like this:

logs:
  fluent-bit:
    config:
      outputs: |
        [OUTPUT]
            Name                http
            # other output configuration...

        {{- include "observe.includeExtraFiles" . }}

Traces

kubectl -n observe create secret generic otel-credentials --from-literal='OBSERVE_TOKEN=<opentelemetry datastream token>'

Uninstall Stack

helm -n observe uninstall observe-stack

Uninstall Traces

helm -n observe uninstall observe-traces

Build

Dependencies

Kind needs to be installed in order to build and test this repository:

https://kind.sigs.k8s.io/docs/user/quick-start/