Discovering function names before real metrics are produced

[emschwartz]

We have some use cases where it would be nice to see all the functions that have been "autometricized" (as @mies calls it), before you actually have traffic that would produce metrics. For example, autometrics-dev/vscode-autometrics#11

One suggestion was to try to start all of the counters at 0 (autometrics-dev/autometrics-rs#70), but most of the metrics libraries don't actually allow you to start counters at 0 and instead only appear when you increment them. (Open question: how would Prometheus itself handle if the counters start at zero? Would it care?)

The other options would involve having autometrics collect up all the function names and either writing them to a file or exposing them on some other endpoint. This is doable (with a kind of crazy trick) in Rust. I'm not sure how it would work in other languages.

Any ideas how we could make this work?

[gagbo]

One suggestion was to try to start all of the counters at 0 (autometrics-dev/autometrics-rs#70)

I don't think it's actually possible, as I noted in the Go issue:

we can't really initialise the metrics to 0, since we don't know in advance the sets of labels that are going to be relevant for the function_calls_count and function_calls_duration metrics.

So the initialization to 0 of the rust library probably only adds a point to the metric function_calls_count without any label, which doesn't help to discover the function names. To be verified though.

I think the only solution is to collect all the autometricized functions (as their annotations are being processed) into some project-global state that could be used by the init() function that's provided by autometrics to set proper (read "with proper labels") 0 values on the automatic metrics.

This is already what we are probably doing with the build_info metric — set a gauge to a value just after creating the metric — with the difference that we will need a registry of some sort to properly set 0 values for calls_count and duration metrics. For build_info it was easier to find the relevant information as there is a single, known in advance, location to check.

Being able to do it through metrics has the advantage of being able to rely on Prometheus to query the list of known functions (and since we query the data to Prometheus eventually, that means we'd keep a single source of truth)

[emschwartz]

Yeah, the global state you're describing is what this would entail. Collecting up the function names somewhere is one part of the task, but the other is actually producing the metrics because most of the libraries wouldn't allow us to start a count at zero, even if we knew the labels.

[gagbo]

It should be possible to increment a counter by 0 to add the labels without affecting data. Prometheus requires client libs to accept 0 and I don't see anything that requires increments to sums to be positive in OTel

We can ignore trying to add data to the duration histograms (unless somehow we could add a NaN value in durations)

[emschwartz]

Doh! You're right that that should probably work

[emschwartz]

I just had a thought on how this could work in dynamic languages like Typescript and Python.

In dynamic languages, I think the wrappers and decorator functions are evaluated right away, rather than when the wrapped function is actually called. This means that in the logic of the wrapper/decorator, we might be able to increment the counters by zero. This should theoretically mean that all the metrics will start at zero, so the UI would be able to use Prometheus data (rather than static analysis) to list all the functions.

One further detail on this: we need to think about whether we can make the set of labels perfectly match a set of labels that will actually be produced at runtime. If we can (for example, we can set result="ok"), then it seems fine that we'll initialize a time series in the Prometheus database that will subsequently be used.

If we can't make the labels match 100%, because there are some we just won't know at startup but would know later when it's called for real, we might want to make this behavior only trigger during development as opposed to in production. We don't really want to create additional time series in Prometheus that will never be used beyond the initial moment. In Rust, I would disable this behavior in release builds while in other languages you'd probably rely on environment variables or something like that to determine whether it's running in production or not.

[brettimus]

cool yeah this works in python, i just tested it out

[brettimus]

started PR here, obviously will need more discussion around configuring this behavior, etc

but this will serve us very well when developing ui for autometrics SLOs

[keturiosakys]

Confirmed this works in TS/JS for wrapper functions and opened a WIP PR: autometrics-dev/autometrics-ts#73

Still confirming if this can work with class decorators.

[emschwartz]

An advantage of setting the metrics to zero to start, instead of using static analysis to find all of the functions, has to do with discovering which functions are part of SLOs.

@mies and @brettimus raised this issue. Our current approach for static analysis relies on tree sitter, which I believe is only meant to operate on a single file. That's fine for finding plain autometrics annotations, but it wouldn't allow you to determine which functions were part of which SLO because the objective would likely be defined in another file.

If we rely on the libraries themselves to produce dummy metrics at startup, they would be able to access the objective details and could attach the right labels.

[P2P-Nathan]

For dev and debug implementations I think it's fine to observe an artificial 0 reading, as long as it's clearly documented.

In these situations I think the trade off of confirmation that instrumentation works is worth slight inaccuracies in the data.

Maybe even an environmental variable that could be switched once you move from functionality testing to performance testing.

[gagbo]

Why do we need to observe a value for the histograms? If we get all the function_calls labels, we can be sure that (eventually) the histogram will have the same labels. So there's no need to taint the histogram with false data. If we want to have something that disambiguate "no call yet" from "this timeseries just won't exist", we can do the work elsewhere in the front-end to query the label of the function in the function_calls metric, and see whether the function name exists there or not.

[emschwartz]

@gagbo the latency SLO labels are only ever present on the histogram metric. The labels on the counters alone would tell you the function names and we could show the success rate SLOs but not the latency ones. We could choose not to show SLOs at all, but it would be very odd if we showed only the success rate SLOs but not the latency ones.

[gagbo]

True, I forgot about the SLO situation. I thought about observing NaN to force something, but it wouldn't work I guess

• there's no spec about what should happen in client libraries when observing NaN in a histogram, so it's probably going to vary wildly across languages, and
• I guess the function_calls_duration_sum sub-metric will always be NaN if we observe NaN in the beginning, which would throw off computations.

Since we do know that Prometheus deals with histograms by having the _count and _sum submetrics though, maybe we can increment those; but I don't think client libraries expose handles to modify those metrics individually (for correctness purposes I'd assume)

[emschwartz]

Yeah I also wondered whether it would be possible to include function_calls_duration_seconds_count{...} 0, and how Prometheus would handle that.

I think it's probably not the way to go even if Prometheus would correctly handle it because client libraries definitely do not expose a way to set that so we'd probably need to resort to appending those lines to the exported metrics output string, which feels very very dirty.

[gagbo]

FTR, calling inc(0) at startup on a counter should not break the alerting and PromQL results from the queries.

The code that check for resets for counters when calling rate (or irate for that matter) is here and it seems that it's going to be fine if one of the counter value is 0 and not necessarily positive¹

Footnotes

1. it simply loops over the span on which you want the rate, and adds the last value before reset to the total count of events. In the end the resultFloat value contains the total count by which the counter was incremented in the timespan. ↩

[emschwartz]

Thanks @gagbo for checking!

[brettimus]

on my python branch that initializes calls at zero, i've been getting a lot of "NaN" results back from prometheus. could this be from dividing by zero in, e.g., the error query?

(if intuition says this might be happening, i can try to reproduce tomorrow)

[emschwartz]

If that's the issue, you might want to try adding a > 0 in the denominator of the query. That will filter out any time series where that value is zero, so you won't have anything to divide by and therefore shouldn't get the NaNs.

[emschwartz]

Implementation of this is being tracked in #59