-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
AWS Data Firehose HTTP Endpoint Input Plugin #15870
Comments
Next steps: Someone needs to implement this plugin. |
Hi @srebhan. I would be more than happy to work on this plugin. Infact, I already have a working solution for this plugin. I can share a draft PR for early review |
@syedmhashim sorry for the late reply! I would be very happy to review your work! |
@srebhan Thanks for your response. Here's a draft PR. I still need to write the tests but thought it'd be good to have some early feedback. Looking forward to your comments |
Use Case
I would like to request for an AWS Data Firehose HTTP Endpoint Input Plugin. I'd be using this in our monitoring infrastructure to transfer CloudWatch Logs from an AWS Data Firehose stream to a Telegraf server where I would be doing some processing and aggregation of the logs data and create metrics from the logs. These would then be pushed to a Loki server using the loki output plugin, from where these metrics would be pulled by our AWS Managed Grafana Server.
Expected behavior
Successful transfer of CloudWatch Logs from AWS Data Firehose stream to a Telegraf server.
Actual behavior
Initially, I decided to use the existing http_listener_v2 input plugin with the Data Firehose's HTTP Endpoint destination option. However, HTTP Endpoint destination option in Data Firehose have some requirements which I was not able to achieve using the http_listener_v2 input plugin. Some of the constraints are as follow:
For more details, please see here
Additional info
As an alternative, we instead push the logs from the Firehose to an S3 bucket. These logs then get pulled from the S3 bucket to an EC2 server and placed in folder from where Promtail scrapes and pushes to loki. We want to eliminate the S3 bucket and Promtail server and instead use Telegraf. I'm already working on the Data Firehose Input Plugin for this purpose and would be more than happy to share a PR.
The text was updated successfully, but these errors were encountered: