Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws-lambda-event-sources: add filters to kafka event sources #26348

Closed
2 tasks
jdiebold opened this issue Jul 13, 2023 · 3 comments · Fixed by #26366
Closed
2 tasks

aws-lambda-event-sources: add filters to kafka event sources #26348

jdiebold opened this issue Jul 13, 2023 · 3 comments · Fixed by #26366
Labels
@aws-cdk/aws-lambda-event-sources @aws-cdk/aws-msk Related to Amazon Managed Streaming for Apache Kafka (Amazon MSK) effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2

Comments

@jdiebold
Copy link

Describe the feature

The AWS documentation (https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html#filtering-msk-smak) suggests that event filtering is also possible for MSK or self-managed Kafka.
Yet, the cdk doesn't offer this config for the (Self)ManagedKafkaEventSource construct.

Use Case

Filter the Kafka events before the lambda is called to reduce the number of lambda calls.

Proposed Solution

No response

Other Information

No response

Acknowledgements

  • I may be able to implement this feature request
  • This feature might incur a breaking change

CDK version used

2.87.0

Environment details (OS name and version, etc.)

n/A

@jdiebold jdiebold added feature-request A feature should be added or improved. needs-triage This issue or PR still needs to be triaged. labels Jul 13, 2023
@github-actions github-actions bot added the @aws-cdk/aws-msk Related to Amazon Managed Streaming for Apache Kafka (Amazon MSK) label Jul 13, 2023
@jdiebold jdiebold changed the title (module name): (short issue description) aws-lambda-event-sources: add filters to kafka event sources Jul 13, 2023
@pahud pahud closed this as completed Jul 13, 2023
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@pahud pahud reopened this Jul 13, 2023
@pahud
Copy link
Contributor

pahud commented Jul 13, 2023

reopening for close by mistake.

Looks like we can add a filter here:

const eventSourceMapping = target.addEventSourceMapping(
`KafkaEventSource:${Names.nodeUniqueId(target.node)}${this.innerProps.topic}`,
this.enrichMappingOptions({
eventSourceArn: this.innerProps.clusterArn,
startingPosition: this.innerProps.startingPosition,
sourceAccessConfigurations: this.sourceAccessConfigurations(),
kafkaTopic: this.innerProps.topic,
kafkaConsumerGroupId: this.innerProps.consumerGroupId,
}),

@pahud pahud added p2 effort/medium Medium work item – several days of effort and removed needs-triage This issue or PR still needs to be triaged. labels Jul 13, 2023
mergify bot added a commit to lpizzinidev/aws-cdk that referenced this issue Jul 14, 2023
mergify bot added a commit to lpizzinidev/aws-cdk that referenced this issue Jul 14, 2023
@mergify mergify bot closed this as completed in #26366 Jul 14, 2023
mergify bot pushed a commit that referenced this issue Jul 14, 2023
…26366)

This change grants the possibility to [specify event filtering](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html#filtering-msk-smak) on the `ManagedKafkaEventSource` and `SelfManagedKafkaEventSource` constructs via the `filters` property.

Closes #26348.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

colifran pushed a commit that referenced this issue Jul 17, 2023
…26366)

This change grants the possibility to [specify event filtering](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html#filtering-msk-smak) on the `ManagedKafkaEventSource` and `SelfManagedKafkaEventSource` constructs via the `filters` property.

Closes #26348.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
bmoffatt pushed a commit to bmoffatt/aws-cdk that referenced this issue Jul 29, 2023
…ws#26366)

This change grants the possibility to [specify event filtering](https://docs.aws.amazon.com/lambda/latest/dg/invocation-eventfiltering.html#filtering-msk-smak) on the `ManagedKafkaEventSource` and `SelfManagedKafkaEventSource` constructs via the `filters` property.

Closes aws#26348.

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-lambda-event-sources @aws-cdk/aws-msk Related to Amazon Managed Streaming for Apache Kafka (Amazon MSK) effort/medium Medium work item – several days of effort feature-request A feature should be added or improved. p2
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants