-
Notifications
You must be signed in to change notification settings - Fork 4.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
OOTB support AWS Eventbridge #40006
OOTB support AWS Eventbridge #40006
Conversation
💚 CLA has been signed |
This pull request does not have a backport label.
To fixup this pull request, you need to add the backport labels for the needed
|
e82a2ca
to
314103e
Compare
Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane) |
2712439
to
953e22c
Compare
a5325e6
to
786192b
Compare
As a general comment; it looks like aws security-lake uses the same format for notifications but omits the detail-type field: https://docs.aws.amazon.com/security-lake/latest/userguide/subscriber-data-access.html#sample-notification Also really liking the part where non of these protocols look like their are versioned or anything. 😞 |
We have an integration for Amazon Security Lake which currently reads:
This change would also allow us to improve that integration, again the detail-type field is in question. I guess we should spin up a AWS security lake setup to verify. I hope it's there and the documentation is off as don't think it's safe to assume every event is object created when the field is not available. The integration is now quite limited as a lot of users have a retention long enough that there's enough objects in the bucket that polling becomes prohibitively expensive. |
02cae8e to cover 'I have made corresponding changes to the documentation' |
This pull request is now in conflicts. Could you fix it? 🙏
|
and a8eb074 to cover changelog item. |
@belimawr Can you do a final check? |
Hey @mjmbischoff |
@pierrehilbert As long as it 'stays on the radar' it should be fine. |
This pull request is now in conflicts. Could you fix it? 🙏
|
# Conflicts: # x-pack/filebeat/input/awss3/input_benchmark_test.go
This pull request is now in conflicts. Could you fix it? 🙏
|
@graphaelli (as Baptiste is off) @narph @lalit-satapathy could we have someone in your teams to review this PR please? |
I think what prevents our BK from running this PR is the following issue #40503 |
@@ -586,6 +586,13 @@ Please see https://docs.aws.amazon.com/AmazonS3/latest/userguide/ways-to-add-not | |||
for more details. SQS queue will be configured as a | |||
https://docs.aws.amazon.com/sns/latest/dg/sns-sqs-as-subscriber.html[subscriber to the SNS topic]. | |||
|
|||
[float] | |||
=== S3 -> EventBridge -> SQS setup |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a thought here: maybe we can add a cloudformation template to help setup s3 -> eventbridge -> sqs?
We are working on adding a template for the s3-sqs setup: #40642
(not required change for this pr)
@mjmbischoff There are some merge conflicts on this PR now. Would you mind rebasing it on |
# Conflicts: # go.mod # x-pack/filebeat/input/awss3/sqs_test.go
|
Adding code to OOTB support AWS Eventbridge generated events for S3 changes, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html (cherry picked from commit c37159e)
Proposed commit message
Adding OOTB support AWS Eventbridge generated events for S3 changes, see https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventBridge.html
Checklist
I have made corresponding change to the default configuration filesCHANGELOG.next.asciidoc
orCHANGELOG-developer.next.asciidoc
.Disruptive User Impact
The logic is triggered as a fall-through and is best effort.
Author's Checklist
How to test this PR locally
Since it relies on an AWS setup, testing locally beyond units tests is not possible.See /beats/x-pack/filebeat/input/awss3/_meta/terraform/README.md Don't have localstack so not truly 'local' but can run against aws.
Related issues
As it was discovered as part of support ticket there should be an ER linked.
Use cases
AWS has added a new way to notify on S3 changes, next to the existing plain SQS and SNS->SQS, AWS now also support using AWS EventBridge -> SQS
This PR achieves two goals: