Handling with low traffic services in SLOs

[gagbo]

While trying to find information about reducing the NaN in the to-be-released autometrics grafana dashboard, I found how to make sloth generated SLIs ignore datapoints when the count of events do not meet a threshold I think autometrics-dev/autometrics-rs@1bd54d9

If instead of > 0 we use > count we should be able to force a threshold of “minimum events to trigger an alert” from my understanding. But I don’t know what value should be added there if not 0, because it might not be valid for most projects. Should we use a default value ourself? Should we just ask for a threshold on generation?

[gagbo]

One solution I like would be to use increase instead of rate operator for our queries, and make the generated sloth file find a way to threshold all SLIs so that a single failure doesn't consume the full error budget:

  - name: latency-99
    objective: 99
    description: Common SLO based on function latency
    sli:
      events:
        error_query: >
          sum by (objective_name, objective_percentile) (increase(function_calls_duration_count{objective_percentile="99"}[{{.window}}]))
          -
          (sum by (objective_name, objective_percentile) (
            label_join(increase(function_calls_duration_bucket{objective_percentile="99"}[{{.window}}]), "autometrics_check_label_equality", "", "objective_latency_threshold")
            and
            label_join(increase(function_calls_duration_bucket{objective_percentile="99"}[{{.window}}]), "autometrics_check_label_equality", "", "le")
          ))
        total_query: sum by (objective_name, objective_percentile) (increase(function_calls_duration_count{objective_percentile="99"}[{{.window}}])) > 100
    alerting:
      name: High Latency SLO - 99%
      labels:
        category: latency
      annotations:
        summary: "High latency on SLO: {{$labels.objective_name}}"
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket

In the total_query line, the 100 threshold is obtained as :

let sli_threshold = 100.0 / (100.0 - objective)

So the threshold for the current builtin values would be

90% -> 10
95% -> 20
99% -> 100
99.9% -> 1000

The > (rather than >=) should ensure that 1 failure is still below the threshold.

[IvanMerrill]

I'm not sure that increase is safe from counter resets which would be a problem. As per the docs it's recommended to use rate for recording rules rather than increase.

[gagbo]

rate has the same issue as increase when it comes to restart, I suppose the recording rule advice was just to get closer to the raw data without the extra sugar. But we could also compute the threshold using .window and rate (or irate) if sloth allow us to

[emschwartz]

From the docs on increase:

increase should only be used with counters and native histograms where the components behave like counters. It is syntactic sugar for rate(v) multiplied by the number of seconds under the specified time range window, and should be used primarily for human readability. Use rate in recording rules so that increases are tracked consistently on a per-second basis.

[IvanMerrill]

I think we can give a default value, that's pretty low, e.g 10 and allow it to be configured. We can't get it right for everyone, and for certain applications every request might be incredibly valuable, but saying that we should receive 10 requests overall before we want to be woken up in the night is a decent starting point.

[emschwartz]

When you say "allow it to be configured", do you mean via labels (such that you would change something in your code to affect this value) or by regenerating the alerting rules using our CLI and Sloth?

[emschwartz]

We ended up adding a minimum threshold of 1 request per minute to the alerting rules