-
Notifications
You must be signed in to change notification settings - Fork 166
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DynamoDB Streams cannot list shards #181
Comments
Well, sounds more like the feature request exactly for the https:/awslabs/dynamodb-streams-kinesis-adapter. Feel free to raise a PR here with an Thank you. |
I don't think this is a feature request for https:/awslabs/dynamodb-streams-kinesis-adapter. Their class maps the AWS API and there is no I might as well create a support ticket for AWS to ask them if they are planning to provide Thank you! |
Hi @artembilan, I have create a PR, also created an issue in the adapter repo from AWS, so they can consider if it makes sense for them to add that functionality there. I've also requested AWS Support to add Regards |
Thanks, but looks like you misunderstood me a bit with the first point: I want to see a fix exactly in the repo, not that Binder where you have raised it. It sounds like such a feature really could be useful for any DynamoDB stream consuming via Kinesis adapter. Does it make sense? |
So, I raised PR in in the binder repo because:
Unfortunately, we cannot use older versions of the binder as there are bugs fixes that force us to use the latest version. But we need to start using DynamoDB Streams, I guess that is or will be the case for other users at some point. The idea was to fix the issue asap where it is materialising and then try to persuade AWS to include it as part of their adapter lib. Does that sound sensible to you? |
Sure! Will review and merge it later this week. |
Sounds good, thanks @artembilan. BTW, I think you closed this issue by mistake as it's not solved still. If you need any change in the PR just let me know. |
I closed it because the fix is not relevant to this project and according our discussion there is nothing to do exactly with this project. |
Adds custom adapter for DynamoDB Stream to the AmazonKinesis model to support `ListShards` operations. It uses `DescribeStream` API under the hood as `ListShards` is not available for DynamoStreams. Bumps LocalStack version from `0.1.22` to `0.2.2`. Resolves: spring-projects/spring-integration-aws#181
I'm trying to read records from a DynamoDB Stream. I'm getting the following error:
My
application.yaml
config is looks like this:The problem is that in the class
KinesisMessageChannelBinder
aKclMessageDrivenChannelAdapter
is created and theAmazonKinesis
to be used is set as follows:That basically means that when DynamoDB stream is being process the following class will be used to interact with it
AmazonDynamoDBStreamsAdapterClient
. That class doesn't supportlistShards
operations that are later used byKinesisMessageDrivenChannelAdapter#readShardList
. Seems like this problem was introduced in 10e96e2#diff-f7c5bb9b0447b4af040508dfb5f6a9fb0b23aa1c22cfa7c378aa5521dfc7f8f3R544.Given there were problems with
DescribeStream
throttling (that's why the issue was introduce) it's possible to sub-classAmazonDynamoDBStreamsAdapterClient
overloadinglistShards
to use describe stream for DynamoDB but list shards for Kinesis with minimal changes. I'm happy to do the change if you are OK with that approach. The change would have to be in https:/spring-cloud/spring-cloud-stream-binder-aws-kinesis but I have created the issue here as a change in this repo was the initial trigger for the issue.The text was updated successfully, but these errors were encountered: