Howto remove datapoints from high cardinality meters or Why do DELTA temporality does not remove datapoints that had no updates

[RoadRunnr]

I have counter that over time can reach a cardinality of a few thousand datapoints. However, at any given time, only a small number of those datapoints will receive updates.

OpenTelemetry explicitly states as a design goal in https://opentelemetry.io/docs/specs/otel/metrics/data-model/#events--data-stream--timeseries:

Delta-to-Cumulative: Metrics that are input and output with Delta temporality unburden the client from keeping high-cardinality state. The use of deltas allows downstream services to bear the cost of conversion into cumulative timeseries, or to forego the cost and calculate rates directly.

That lead me to believe that meters with delta temporality would not keep datapoints between export cycles.
However, that turns out to be a wrong assumption. Even for meters with delta temporality all datapoints are kept forever. For meters with high-cardinality this can consume a significant amount of memory.

OpenTelemetry does allow gaps and resets in metric (https://opentelemetry.io/docs/specs/otel/metrics/data-model/#resets-and-gaps). So removing datapoints should be doable.

So, my questions is: How do I remove datapoints from high-cardinality meters?

Also, my test suggest that the SDK might not adhere to the specified behavior of MetricReader (https://opentelemetry.io/docs/specs/otel/metrics/sdk/#metricreader) for delta temporality. The specification says:

For synchronous instruments with Delta aggregation temporality, MetricReader.Collect MUST only receive data points with measurements recorded since the previous collection.

However, it appears as if datapoints are exported that had no change.

[albertored]

For synchronous instruments with Delta aggregation temporality, MetricReader.Collect MUST only receive data points with measurements recorded since the previous collection.

This looks like a bug that we should fix.

The first one instead (keeping measurements in memory forever also for delta readers) is more an implementation choice. From the specification point of view we are compliant but I agree that the implementation is sub-optimal. After a brief look at the code I think we can improve this by automatically dropping datapoints that have been already exported in delta readers. Let's wait for the opinion of @tsloughter, in the meanwhile I will make a couple of tests.