Add attribute filtering function to View metric data stream configuration

[MrAlias]

Fixes #3441

Currently the metric view allows users to specify a set of attribute keys that should be preserved. This limited functionality for attribute filtering with a view impacts the usefulness of views, as pointed out in #3441. Instead of specifying attribute filtering as just an allow-list of attribute keys, allow implementations to accept generalized attribute filtering functions.

These filtering functions can then be directly constructed by users to accommodate their needs and allows attribute filtering function utilities to be built if a SIG or external community wants to standardize on filtering methods other than an allow-list for attribute keys.

These changes remain backwards compatible in that they continue to allow implementations to just accept a list of attribute keys to act as an allow-list.

Prior art

The (non-stable) Go implementation of a view currently implements this filtering functionality with its stream configuration parameter.

[jack-berg]

Same question I posted applies here as well. What do we gain from allowing you to filter on key-value pairs as opposed to just the keys?

We could consider taking this a step further and re-adding the "AttributeProcessor" concept, which is defined as:

interface AttributesProcessor {
  Attributes process(Attributes, Context);
}

But if we did this, I'd say we should take it even further, and add a MeasurementProcessor:

interface Measurement {
  Instrument getInstrument();

  double getValue();

  double getAttributes();
}

interface MeasurementProcessor {
  @Nullable
  Measurement process(Measurement, Context);
}

This would allow users to transform attributes before aggregation, including enriching with attributes from context. It would also allow for filtering measurements whose values are outside some acceptable range. It would also allow for converting the units of values.

So more flexibility. I'm in favor of going all the way in terms of flexibility and adding MeasurementProcessor, rather than adding more constrained tools in a piecemeal fashion.

[MrAlias]

I'm in favor of going all the way in terms of flexibility and adding MeasurementProcessor, rather than adding more constrained tools in a piecemeal fashion.

That does look more flexible and useful to users. I am in favor of going that direction as well. I can close this and open something to shift there if others agree.

[MrAlias]

If we switch to the MeasurementProcessor, would it overlap with the exiting View attribute key filter? Could we make that part of the view stream configuration parameters optional to implement (assuming they already aren't 🤷 😉 #3524 (comment))?

[jsuereth]

SO I think the big difference here is where the config is located.

I want a MeasurementProcessor, but I also want more flexible attributes processing on views. View come with an implicit filter for where they apply. I'm not sure I want a measurement processor on views or as a global thing.

@jack-berg what's your thinking here for where MeasurementProcessor lives? can I make a global one, like Log/Span processsor?

[jmacd]

On this topic, I support the idea of a MeasurementProcessor. Lightstep has been pushing on an OTel-Collector pipeline recently and one of the features we wanted to inject via the OTel SDK is a way to extend the attribute set w/ context variables. The same functionality should be able to remove attribute-values, (see it in draft form),

Process(ctx context.Context, inAttrs []attribute.KeyValue) (outAttrs []attribute.KeyValue)

[jack-berg]

@jack-berg what's your thinking here for where MeasurementProcessor lives? can I make a global one, like Log/Span processsor?

I was imagining it was a view level configuration, allowing it to apply broadly to many instruments or narrowly to just one.

If we switch to the MeasurementProcessor, would it overlap with the exiting View attribute key filter?

If we make measurement processor a view level config option, we can think of the existing view attribute key filter as shorthand for a measurement processor which only retains certain attribute keys. Would need to decide what to do when a key filter and a measurement processor are configured, as we could either reject that configuration as invalid, or apply one after the other sequentially based on which was registered first.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[github-actions]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

[MrAlias]

A review by @dashpole in the Go SIG pointed out that having this added to a view will make it problematic to specify a metric signal configuration via a static yaml file.

Closing.