Add Histogram support aggregation

[alimousazy]

Required for all PRs:

• CHANGELOG.md updated
• Sign CLA (if not already signed)
• README.md updated (if adding a new plugin)

Adding Histogram (aggregation) support. User can list down metric for aggregation, supported methods
1- Percentile 90 80 ... 20 etc.
2- Count.
3- Mean.
4- Variance.

Method used to calculate these values has a very low memory print and specially designed for streaming data. More information can be found here http://jmlr.org/papers/volume11/ben-haim10a/ben-haim10a.pdf and here https://www.vividcortex.com/blog/2013/07/08/streaming-approximate-histograms/.

Note: This method of Streaming Approximate Histograms result quantile approximations there for it may not lead for exact values. User can optimize accuracy by increase bucket size.

I'm thinking about introducing middleware plugins which process data between inputs and output this will make the agent code clean and allow user to provide additional processing to the pipeline, If you agree on this I will move the code and put it inside a folder called middleware. Creating interface for middle ware to add and retrieve metrics using channels

[alimousazy]

I just moved the Histogram code to new type of plugins (which I called filter) this will make it easier to write test cases against it and will allow future add of these types of plugins which act like middleware between input and the output, some of use cases for middleware/filter plugins will be adding sampling, bandwidth control, control tag and fields included in metrics, shaping metrics like adding/renaming tags, fields and even metric names plus editing metric values . adding aggregation support , split metrics to multi metric based on fields, combine metrics to one metrics and a lot more.

[alimousazy]

I will add more error handling and Unit test once we agree one filter implementation.

[alimousazy]

@sparrc any idea on this ? what you think about the filter plugins ?

[sparrc]

@alimousazy you'll need to give me more time, I'm very swamped with other features at the moment.

[alimousazy]

sure. take your time.

[jadbox]

Will this also handle sum/aggregate count?

[alimousazy]

@jadbox it does handle
1- Percentile 90 80 ... 20 etc.
2- Count.
3- Mean.
4- Variance.
Sum can be added easily if you want I can do that.

Here is an example usage if you can merge pull and compile it from source

[[filter.histogram]]
  bucketsize = 20  
  flush_interval = "1m"
  [filter.histogram.metrics]
    (replace with your metric name) = [0.90]

You can add more percentile if you want like [0.90, 0.50]
Note: this might change after telegraf team review the pull request.

I'm planning to add more filter in the future (which intercept metrics between input and output plugins) some of these filters might be [[filter.rename]] ,[filter.rename.tags],[[filter.must_have]], [[filter.sample_metrics]], [[filter.only_pass_if]], [[filter.bandwidth]] which help shape metrics before emitting for final distention.

[jadbox]

@alimousazy "Sum can be added easily if you want I can do that." YES, please!

My data "firehose" looks like the following (abstractly):

userID | timestamp | doesActionA | doesActionB
joe, 1466550440, 50, 20
joe, 1466550440, 10, 15

and I want to aggregate in telegraph before sending to Influx:

# aggregate into 1s blocks, and send each block to Influx
joe, 1466550440, 60, 35

My pipeline does 100k/s in messages so micro-aggregates before going into Influx is needed.
I appreciate the guidance as I'm an Influx/Telegraf newbie.

[alimousazy]

if joe is used as name of metric then configuration should be like this

[[filter.histogram]]
  bucketsize = 20  
  flush_interval = "1m"
  [filter.histogram.metrics]
    joe = [0.90, 0.25]

*Note: you can tone aggregation interval by modifying flush_interval (I may change flush interval to aggregation interval) , If you don't need percentile just leave the array empty.

The output will be similar to this

{"fields":{ "doesActionA_count":3,"doesActionA_mean":1,"doesActionA_p90":1,"doesActionA_variance":0 "doesActionB_count":3,"doesActionB_mean":1,"doesActionB_p90":1,"doesActionB_variance":0 },"name":"joe","tags":{tags___},"timestamp":1466718240}

[jadbox]

@alimousazy Thanks, that's helpful. I guess what I'm really looking for is just aggregation... not histogram aggregation. I'll take this concern to #380

[alimousazy]

I'm planing to add support for all the metric types mentioned here http://metrics.dropwizard.io/3.1.0/getting-started/, once this get reviewed and we agree on first version

[sparrc]

@alimousazy the histogram filter should do all by default. Since Filters in general should have a plugin structure and should thus appear in the default config file. Can you explain how that is going to look and exactly how one would configure the filters for generically aggregating on fields, tags, measurement names, etc.?

[alimousazy]

@sparrc Filters are middleware which stand between inputs and outputs (Pipes) . here is how histogram filter configuration. (Note that filter are independent from input and output )

Filters will look like this in configuration files (Note the configuration is flexible and we can change it)

[[filter.histogram]]
  bucketsize = 20  
  flush_interval = "1m"
  [filter.histogram.metrics]
    (replace with your metric name) = [0.90]

here is the interface that filters should implemented

package telegraf

type Filter interface {
        // SampleConfig returns the default configuration of the Input
        SampleConfig() string

        // Description returns a one-sentence description on the Input
        Description() string

        //create pipe for filter
        Pipe(in chan Metric) (out chan Metric)

        // start the filter
        Start(shutdown chan struct{})
}

we may add other filters (middleware) in the future like adding sampling, bandwidth control, control tag and fields included in metrics, shaping metrics like adding/renaming tags, fields and even metric names plus editing metric values , split metrics to multi metric based on fields, combine metrics to one, alerting and anomaly detection.

Please check the code for more detail 8f7bbdb

[sparrc]

I'll need to do a fuller review, but like I said initially this is going to need to support matching metrics on more than just measurement name, and all name matching will need to support glob filters, much of which we already have implemented in internal.

[sparrc]

why is this in a goroutine?

[alimousazy]

Filter suppose to work in is own context like other plugins it may have timer inside and i don't want to block the agent code from collecting metrics or flushing them. consider filter as pipe with input and output (black box). Histogram filter for example has timer for flushing metrics on specific intervals

[alimousazy]

I have to add command line option to generate default filters (Disabled) when generating configuration file ) please let me know what command line option you prefer for such operation ? also I'm going to work on adding glob matching on metric value and tags value ( I think it is not possible to do matching on field value since it is numeric ), is that what you expect ?

[alimousazy]

Command line option added

 -filter-filter     filter the filter plugins to enable, separator is :
  -filter-list       print all the available filters

I will work on adding globing and unit test

[alimousazy]

@sparrc I added a new way to match metric, User can now specify Tags or Measurement to match against (Glob enabled) , rollup name, or function to apply on data set beside if they want to keep the old metric @jadbox.

User also can pass the original metric and only create rollup for aggregation like in second example

example
   rollup = [
      "(Name new) (Tag interface en*) (Functions mean 0.90)",
      "(Name cpu_value) (Measurements cpu) (Functions mean sum) (pass)",
    ]

Please tell me what you think

[sparrc]

I'm closing this in favor of #1777