You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
An in-memory traffic shaper processor which evens out traffic so that output traffic is uniform
We use telegraf as a proxy and we receive data that is spiky in nature, in our every 10 minute we receive a spike and this affects our downstream systems since it needs to also process at the same rate, this leads to wastage of resource since the cpu, memory needs to be provisioned for peaks
Screenshot showing spiky nature of data
Expected behavior
Actual behavior
We have implemented a traffic shaper plugin in telegraf which evens out the spike and help us maintain a steady rate.
Example post using traffic shaper plugin we can see after 00:00 the traffic became uniform
Additional info
No response
The text was updated successfully, but these errors were encountered:
After discussing with the team, we would prefer that rate limiting like this occur in outputs instead of a processor. We have an open issue for this in: #14802 That is where I would want to continue this discussion.
The processor solution unfortunately adds a second buffer, which introduces another set of issues. You can see some examples of this #802 for a way to maintain the buffer outside of memory via a WAL and how to control the buffer itself with #15208 and #15217.
As such we do not wish to add a processor, but would take a PR to look at output rate limiting. I will close this issue and the PR.
Use Case
An in-memory traffic shaper processor which evens out traffic so that output traffic is uniform
We use telegraf as a proxy and we receive data that is spiky in nature, in our every 10 minute we receive a spike and this affects our downstream systems since it needs to also process at the same rate, this leads to wastage of resource since the cpu, memory needs to be provisioned for peaks
Screenshot showing spiky nature of data
Expected behavior
Actual behavior
We have implemented a traffic shaper plugin in telegraf which evens out the spike and help us maintain a steady rate.
Example post using traffic shaper plugin we can see after 00:00 the traffic became uniform
Additional info
No response
The text was updated successfully, but these errors were encountered: