Stac-19417: Document request tracing

SUMMARY.md

@@ -46,6 +46,7 @@
 ## 🕵️ Agent
 * [Network configuration](setup/k8s-network-configuration.md)
 * [Using a custom registry](setup/agent/k8s-custom-registry.md)
+* [Request tracing](setup/agent/k8sTs-agent-request-tracing.md)
 
 ## 🔐 Security
 

k8s-quick-start-guide.md

@@ -412,7 +412,9 @@ To set up a StackState Self-hosted integration you need to have:
 * An up-and-running Self-hosted Cluster.
 * A recent version of Helm 3.
 * A user with the permission to `create privileged pods`, `ClusterRoles` and `ClusterRoleBindings`:
- * ClusterRole and ClusterRoleBinding are needed to grant StackState Agents permissions to access the Kubernetes API.
+ * ClusterRole and ClusterRoleBinding are needed to:
+ * Grant StackState Agents permissions to access the Kubernetes API
+ * Generate a secret for the mutating validation webhook which is part of [request tracing](/setup/agent/k8sTs-agent-request-tracing.md)
  * StackState Agents need to run in a privileged pod to be able to gather information on network connections and host information.
 
 ### Set up a self-hosted integration

setup/agent/k8s-custom-registry.md

@@ -47,10 +47,15 @@ Login Succeeded
 
 The images listed below are used in the StackState Agent Helm Chart:
 
+- [quay.io/stackstate/container-tools](https://quay.io/stackstate/container-tools)
+- [quay.io/stackstate/generic-sidecar-injector](https://quay.io/stackstate/generic-sidecar-injector)
+- [quay.io/stackstate/http-header-injector-proxy-init](https://quay.io/stackstate/http-header-injector-proxy-init)
+- [quay.io/stackstate/http-header-injector-proxy](https://quay.io/stackstate/http-header-injector-proxy)
 - [quay.io/stackstate/stackstate-k8s-agent](https://quay.io/repository/stackstate/stackstate-k8s-agent)
 - [quay.io/stackstate/stackstate-k8s-process-agent](https://quay.io/repository/stackstate/stackstate-k8s-process-agent)
 - [quay.io/stackstate/stackstate-k8s-cluster-agent](https://quay.io/repository/stackstate/stackstate-k8s-cluster-agent)
 
+
 ## Configuring the Helm Chart to use a custom registry
 
 This section describes the values that need to be configured in the StackState Agent Helm Chart to use a custom registry.
@@ -65,8 +70,8 @@ For example:
 ```yaml
 global:
  imageRegistry: 57413481473.dkr.ecr.eu-west-1.amazonaws.com
-all:
- image:
- pullSecretUsername: johndoe
- pullSecretPassword: my_secret-p@ssw0rd
+ imagePullCredentials:
+  default:
+  username: johndoe
+  password: my_secret-p@ssw0rd
 ```

setup/agent/k8sTs-agent-request-tracing.md

@@ -0,0 +1,71 @@
+---
+description: StackState Kubernetes Troubleshooting
+---
+
+# Request tracing
+
+## Observability through load balancers, service meshes and between clusters
+
+StackState can observe connections between services and pods in different Clusters, or when the connections go through a Service Mesh or Load Balancer. Observing these connections is done through `request tracing`. Traced requests will result in connections in the [topology perspective](/use/views/k8s-topology-perspective.md), to give insight in the dependencies across an application and help with finding the root cause of an incident.
+
+## How does it work
+
+Request tracing is done by injecting a unique header (the `X-Request-ID` header) into all HTTP traffic. This unique header is observed at both client and server through an eBPF probe installed with the StackState Agent. These observations are sent to StackState, which uses the observations to understand which clients and server are connected.
+
+The `X-Request-Id` headers are [injected](#enabling-the-trace-header-injection-sidecar) by a sidecar proxy that can be automatically injected by the StackState Agent. The sidecar gets injected by a [mutating webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook), which injects the sidecar into every pod for which the `http-header-injector.stackstate.io/inject: enabled` annotation is defined. 
+
+It's also possible to add the `X-Request-Id` header if you application [already has a proxy or LoadBalancer](#add-the-trace-header-id-to-an-existing-proxy), or through [instrumenting your own code](#instrument-your-application). Advantage of this is that the extra sidecar proxy isn't needed.
+
+## Enabling the trace header injection sidecar
+
+Enabling trace header injection is a two-step process:
+
+ 1. Install the mutating webhook into the cluster by adding `--set http-header-injector-webhook.enabled=true` to the helm upgrade invocation when installing the StackState agent
+ 2. For every pod that has a endpoint which processes http(s) requests, place the annotation `http-header-injector.stackstate.io/inject: enabled` to have the sidecar injected.
+
+{% hint style="warning" %}
+**Enabling the mutating webhook will only take effect upon pod restart**
+
+If the annotation is placed before the webhook is installed. Installing the webhook has no effect until the pods get restarted.
+{% endhint %}
+
+### Disabling trace header injection
+
+Disabling the trace header injection can be done with the reverse process:
+
+1. Remove the `http-header-injector.stackstate.io/inject: enabled` annotation from all pods.
+2. Redeploy the StackState Agent without the `--set http-header-injector-webhook.enabled=true` setting. 
+
+{% hint style="warning" %}
+**Disabling the mutating webhook will only take effect upon pod restart**
+
+If step 1 is skipped and only the mutating webhook is disabled, all pods need a restart for the sidecar to be removed. 
+{% endhint %}
+
+### Overhead
+
+Request tracing adds a small, fixed amount of CPU overhead for each HTTP request header that gets injected and observed. The exact amount is dependent on the system that it's ran on, so it's advised to enable this feature first in an acceptance environment to observe the impact before moving to production. The sidecar proxy takes a minimum of 25Mb of memory per pod it's deployed with, up to a maximum of 40Mb. 
+
+## Add the trace header id to an existing proxy
+
+To add the `X-Request-Id` header from an existing proxy, two properties are important:
+
+1. Each request/response pair has to get a unique ID.
+2. The `X-Request-Id` header should be added to both request and response, to be observed on both client and server.
+
+### Add the trace header id in envoy
+
+In envoy, the `X-Request-Id` header can be enabled by setting `generate_request_id: true` and `always_set_request_id_in_response: true` for [http connections](https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto)
+
+## Instrument your application
+
+It's also possible to add the `X-Request-Id` header form either the client side to each request, or on the server side to each response. It's important to ensure each request/response gets a unique `X-Request-Id` value. Also, the `X-Request-Id` requires that if an ID is already present in a request, the response should contain that same ID.
+
+## Supported systems/technologies
+
+- HTTP/1.0 and HTTP/1.1 with keepAlive
+- Trace header injection and trace observation on unencrypted traffic 
+- Trace observation for OpenSSL Encrypted traffic
+- Trace header injection alongside LinkerD
+- Any LoadBalancer that forwards the `X-Request-Id` header in requests and responses
+- Any cross-cluster networking solution that forwards the `X-Request-Id` header in requests and responses