Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Aggregate metrics transform(s) #3668

Closed
jamtur01 opened this issue Sep 1, 2020 · 4 comments · Fixed by #20836
Closed

Aggregate metrics transform(s) #3668

jamtur01 opened this issue Sep 1, 2020 · 4 comments · Fixed by #20836
Labels
domain: metrics Anything related to Vector's metrics events domain: transforms Anything related to Vector's transform components

Comments

@jamtur01
Copy link
Contributor

jamtur01 commented Sep 1, 2020 •
edited by jszwedko
Loading

Aggregate Vector's metric types into coarser metrics across a configurable time interval.

Proposed 1.0 aggregations

  • count
  • diff
  • max
  • min
  • mean
  • sum
  • stdev

Ref:
@jamtur01 jamtur01 added domain: metrics Anything related to Vector's metrics events domain: transforms Anything related to Vector's transform components labels Sep 1, 2020
@binarylogic binarylogic changed the title feat: Aggregate metrics transform(s) Aggregate metrics transform(s) Sep 5, 2020
@binarylogic binarylogic added Epic Larger, user-centric issue that contains multiple sub-issues and removed Epic Larger, user-centric issue that contains multiple sub-issues labels Dec 3, 2020
@jszwedko
Copy link
Member

This could be a follow-up, but I think it'd be useful to be able to aggregate histogram statistics as well to reduce the samples down to just p50, p95. p99, etc. This would reduce the amount of data being sent with the trade-off of less granularity downstream.

@bruceg bruceg mentioned this issue Feb 16, 2021
Open
@jeromekleinen
Copy link

I would also be interested in this kind of transform.

High level use case: detection of log stoppages based on metrics. We use an enterprise log platform which imposes license limits. Therefore, we filter logs based on use cases instead of sending all of the logs. It is possible that a system sent its logs, but due to the filtering it may seem like it hasn't sent anything leading to false positives.

I would like to use metrics based on the event stream before filtering to overcome this issue. Unfortunately, metric events are also impacting the license volume, so aggregating the metrics per host over fixed time intervals seems like the way to go.

@ross ross mentioned this issue Jun 17, 2021
Merged
4 tasks
@jszwedko
Copy link
Member

Noting, per #8628, this should also permit aggregating across metric tags.

@lambdaq lambdaq mentioned this issue May 13, 2022
Open
@jszwedko jszwedko mentioned this issue May 17, 2022
Open
esensar added a commit to esensar/vector that referenced this issue Jul 10, 2024
@esensar
feat(aggregate transform): add more aggregations to aggregate transform
ea1b98a 
Adds more functions to `aggregate` transform.

Closes: vectordotdev#3668
@esensar esensar mentioned this issue Jul 10, 2024
Merged
@jszwedko jszwedko mentioned this issue Jul 26, 2024
Open
@valleedelisle
Copy link

Would love to see this implemented. +1

github-merge-queue bot pushed a commit that referenced this issue Sep 9, 2024
@esensar @pront
feat(aggregate transform): add more aggregations to aggregate transfo…
f346a31 
…rm (#20836)

* feat(aggregate transform): add more aggregations to aggregate transform

Adds more functions to `aggregate` transform.

Closes: #3668

* Add changelog entry

* Add implementation for more aggregation functions

* Add test for min and max mode

* Implement Diff aggregation

* Add mean and stdev aggregations

* Update aggregate component docs

* Clean up mean calculation in metrics aggregation

Co-authored-by: Pavlos Rontidis <[email protected]>

* Remove `metric::` prefix in aggregate transform tests

* Emit errors when conflicting types are found during flush

* reduce code dulpication for MIN and MAX

* Clean up `record_count` in aggregate transform

Co-authored-by: Pavlos Rontidis <[email protected]>

* Remove unnecessary metric prefix in aggregate transform

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
AndrooTheChen pushed a commit to discord/vector that referenced this issue Sep 23, 2024
@esensar @pront @AndrooTheChen
feat(aggregate transform): add more aggregations to aggregate transfo…
1372afa 
…rm (vectordotdev#20836)

* feat(aggregate transform): add more aggregations to aggregate transform

Adds more functions to `aggregate` transform.

Closes: vectordotdev#3668

* Add changelog entry

* Add implementation for more aggregation functions

* Add test for min and max mode

* Implement Diff aggregation

* Add mean and stdev aggregations

* Update aggregate component docs

* Clean up mean calculation in metrics aggregation

Co-authored-by: Pavlos Rontidis <[email protected]>

* Remove `metric::` prefix in aggregate transform tests

* Emit errors when conflicting types are found during flush

* reduce code dulpication for MIN and MAX

* Clean up `record_count` in aggregate transform

Co-authored-by: Pavlos Rontidis <[email protected]>

* Remove unnecessary metric prefix in aggregate transform

---------

Co-authored-by: Pavlos Rontidis <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
domain: metrics Anything related to Vector's metrics events domain: transforms Anything related to Vector's transform components
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(aggregate transform): add more aggregations to aggregate transform esensar/vector
5 participants
@jamtur01 @jszwedko @binarylogic @jeromekleinen @valleedelisle