No prometheus metrics

[fcamposdespegar]

With last changes we lost prometheus metrics. Problem may be related with metric names.

[puneetkhanduri]

Users were running into this issue with long metric names so we had to abbreviate and trim the names to avoid exceptions. The metrics are now being exported with abbreviated names as a stopgap solution which we don't like. Open to suggestions.

[suever]

@puneetkhanduri It seems as though the use of Prometheus labels could be beneficial to shorten the metric names but maintain legibility. Not only is the current approach difficult to read (and is not documented at all), it seems to combine values that should really be separated out (e.g. all GET requests get squashed into the same metric so you cannot filter based on path).

[puneetkhanduri]

@suever I can see the counter interface offers us the ability to provide tags in addition to metric names. Can you propose a naming structure until the long metric names issue is resolved. Here's what I currently have in mind:

name = <unmodified full original metric name>.take(63) // take first 63 characters and drop the rest
tags = "name", <unmodified full original metric name>

or

name = <unabbreviated endpoint>.<abbreviated field>
tags = "type" , ("raw" | "noise")

Open to anything better you might have in mind.

[puneetkhanduri]

@fcamposdespegar @suever : Please share your feedback if any so we may address and close this issue.

[suever]

Sorry for the delay, I had to actually install an old version of opendiffy to determine what the metrics were supposed to represent based on the names before they were truncated.

I would propose something like the following where the cardinality will still be high, but the number of metrics (and length of their names) is reasonable.

These examples may not be 100% accurate but provide an example of my thinking

diffy_requests_total{method="GET", path="/app"}
diffy_requests_differences_total{method="GET", path="/app", exclude=noise="true"}
diffy_requests_differences_total{method="GET", path="/app", exclude=noise="false"}
diffy_requests_differences_total{method="GET", path="/app", exclude_noise="true", key="headers"}
diffy_requests_differences_total{method="GET", path="/app", exclude_noise="true", key="body.field"}
diffy_requests_differences_total{method="GET", path="/app", exclude_noise="true", key="headers.accept-encoding"}

[puneetkhanduri]

Fixed in the latest release with the introduction of endpoint and field tags to corresponding metrics.