-
Notifications
You must be signed in to change notification settings - Fork 495
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add series aggregation DSL function aggregate
#2294
Add series aggregation DSL function aggregate
#2294
Conversation
aggregate
Oh I really like this idea. These are operations I often feel that the TSDB should do, but often they don't or not with the order of operations you may want. Some things to think about:
Looking at how r or pandas does things can show some of the possibilities. If we just want to start with a simple function with limited behavior (for example, timestamps must line up, nan values or ignored) etc that is fine. But we might not want to take the aggregate func name for that. |
Another factor is aggregation and tags / dimensions, it will most like be very useful to be able to do a group by within a tagset. So for example, says you have a series that is Edit: ... Oh, maybe you already do this... |
Thanks for the feedback @kylebrandt!
|
NaN: Honestly I don't think we really ever thought that out well enough. If we are sticking with the simplest possible version, I would just drop them Lining up timestamps: Agreed, building interpolation is no small request. Not only do you have to downsample but also upsample. There are all sorts of different ways to do this Name: I wanted to reserve |
cmd/bosun/expr/funcs.go
Outdated
"aggregate": { | ||
Args: []models.FuncType{models.TypeSeriesSet, models.TypeString, models.TypeString}, | ||
Return: models.TypeSeriesSet, | ||
Tags: tagFirst, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be a function that extracts what the resulting tag keys will be, so it will come from groups
arg to Aggregate arg[1]
. The reason is that if you look in expr/parse, the nodes have a Tag method which makes sure at parse time that the resulting tags of function make sense in any sort of union operations.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, thanks, that makes sense now. I updated it, please have another look when you get a chance.
cmd/bosun/expr/funcs.go
Outdated
} | ||
res.Group = opentsdb.TagSet{} | ||
results.Results = append(results.Results, res) | ||
} else { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
drop the else, return return &results, nil
above in the if condition, and then the rest of the code does not need to be indented. Same idea under https:/golang/go/wiki/CodeReviewComments#indent-error-flow in terms of keeping minimal indentation.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done 👍
cmd/bosun/expr/funcs.go
Outdated
vals := []string{} | ||
for _, grp := range grps { | ||
if val, ok := result.Group[grp]; ok { | ||
vals = append(vals, val) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No need for else, can `continue in the if statement, and then have the error after the if statement. Same idea of minimal indentaiton
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done 👍
cmd/bosun/expr/funcs.go
Outdated
res := Result{} | ||
newSeries := make(Series) | ||
|
||
switch aggregator { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably useful to mimic what we do in http://bosun.org/expressions#windowquery-string-duration-string-period-string-num-scalar-funcname-string-seriesset :
In addition to supporting Bosun’s reduction functions that take on argument, percentile operations may be be done by setting funcName to p followed by number that is between 0 and 1 (inclusively). For example, "p.25" will be the 25th percentile, "p.999" will be the 99.9th percentile. "p0" and "p1" are min and max respectively (However, in these cases it is recommended to use "min" and "max" for the sake of clarity.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice, I wasn't aware that it's done that way in the window query. Will make it so 👍
For NaN, I added a test that just asserts that NaN-aligned values remain NaN - I'm not sure we should drop them here, since that's what the Left to do:
|
Just "aggr"? reasonable unlike a z prefix, and leaves agg and aggregate for future. |
I've never really been happy with dropna though, since it can result in an empty series which I think throws an error because we never really defined what to do about that. For now I would maybe:
I say the above because I'm trying to think of what will make the function least error prone (annoying) in the context of some of the stuff that needs to be better defined in Bosun in general. So in the bigger picture outside of your PR (I should read through existing tests, maybe some of this is covered):
|
Regarding the timeout I doubt that is you, probably just travis being slow - we use a free tier. |
Recent push to master will require an update, but should just be as simple as removing the |
Ok, thanks! I'll make those changes (probably tomorrow or early next week, getting a bit late over here). I'll let you know when I've made them. Just about the NaN values again: as it stands, nothing that the aggregation function does can result in a NaN, so the only way you can end up with it is if there were NaN values in the original series given as input. Maybe I'm missing the wider context, but wouldn't that be the expected behavior? If we drop NaN entries, I feel like we're doing things users might not expect from this function. |
@kylebrandt Ok, I have updated the code to:
NaN behavior is still to pass them along, to avoid the complexities and possibly unexpected behavior of removing them. What do you think? |
@@ -290,7 +290,7 @@ func (a *Results) Equal(b *Results) (bool, error) { | |||
if a.IgnoreOtherUnjoined != b.IgnoreOtherUnjoined { | |||
return false, fmt.Errorf("ignoreUnjoined flag does not match a: %v, b: %v", a.IgnoreOtherUnjoined, b.IgnoreOtherUnjoined) | |||
} | |||
if a.NaNValue != a.NaNValue { | |||
if a.NaNValue != b.NaNValue { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice find!
cmd/bosun/expr/funcs.go
Outdated
@@ -385,6 +411,198 @@ var builtins = map[string]parse.Func{ | |||
}, | |||
} | |||
|
|||
// Aggr combines multiple series matching the specified groups using an aggregator function. If group | |||
// is empty, all given series are combined, regardless of existing groups. | |||
// Available aggregator functions include: avg (average), p50 (median), min (minimum) and max (maximum). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update comment to reflect changes
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
would be good to have sum as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated the comment and added sum
. Also updated the documentation accordingly.
cmd/bosun/expr/funcs.go
Outdated
for _, result := range series { | ||
for t, v := range result.Value.Value().(Series) { | ||
newSeries[t] += v | ||
counts[t] += 1 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
counts[t]++
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That's it, no more Python for me 🐍
Few more comments but in general looks good to me - I'm content to go with your take on NaN. I think maybe the number of This will be quite useful :-) +1 |
Thanks, I removed two of the |
Ok, made some fixes to the documentation too. All done making changes now 👍 |
@hermanschaaf Looks good! In a new PR can you add a little more to the documentation in terms of behavior: for example, timestamps needing to line up, what happens with NaN etc. ? |
Thanks @kylebrandt - sure thing, I will open a new PR for that |
This PR adds an
aggregate
DSL function, which allows one to combine different series in a seriesSet using a specified aggregator (currentlymin
,max
,p50
,avg
).This is particularly useful when comparing data across different weeks (using the
over
) function. In our case, for anomaly detection, we want to compare the current day's data with an aggregated view of the same day in previous weeks. In particular, we want to compare each point in the last day to the median of each point in the corresponding day for the last 3 weeks, so that any anomalies that occurred in a previous week are ignored. This way we compare with a hypothetical "perfect" day.For example:
Which looks like this:
Or, if we wanted to combine series but maintain the
region
andcolor
groups`, that query would look like this:which would result in one merged series for each unique region/color combination.
I am very happy to take suggestions for changes / improvements. With regards to naming the function, I would have probably chosen "merge", but since that is already taken, I went with the OpenTSDB terminology and used "aggregate".