-
Notifications
You must be signed in to change notification settings - Fork 1.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sink(Elasticsearch): Dropping events with AWS auth strategy #20266
Comments
Thanks @joseluisjimenez1 . I think this is a specific case of #10870. We'd like to improve retries in Vector, generally, over time. |
Hey @jszwedko, it seems that could fit under #10870 , but the error loading aws credentials from the aws-sdk maybe is kind of different thing here? not 100% sure tbh... |
A note for the community
Problem
TL'DR : Elasticsearch sink drop events unintentionally even when Acknowledgments are enabled due to failure on loading AWS credentials.
I tried to be really concise, please let me know if can provide any extra information that maybe I missed. Thanks in advance.
Context:
aws
auth strategy.Description:
Vector is running smoothly until some increase of load arrives Kafka (for us, Kubernetes velero backups every hour). Sometimes those spikes do not drop anything, others drop a few and sometimes drop a lot.
But always same errors in the logs:
Things that have been tried:
Workaround:
Switch to basic authentication is the only way to avoid dropping events when those spikes comes that was found.
Proposal:
Vector should be able to handle credentials errors and apply back pressure instead of dropping events when:
EcsContainer
in this case.403 forbidden
due to lack of user permissions.Troubleshooting:
Seems like vector is using AWS rust SDK to sing the request to OpenSearch, but apparently, loads the credentials every single request and not use the cache that is defined?
Configuration
Version
0.36.0 -> 0.37.0
Debug Output
Example Data
No response
Additional Context
Vector is running at AWS ECS Fargate.
References
debug
#15196The text was updated successfully, but these errors were encountered: