wr.s3.list_objects not able to limit results by setting s3_additional_kwargs={"MaxKeys": num}

[mccauleyp]

Describe the bug

It's not possible to limit the number of keys returned by wr.s3.list_objects using the MaxKeys keyword from list_objects_v2 or the MaxItems keyword from the ListObjectsV2 paginator. It's getting passed through but I think isn't compatible with the paginator implementation:

https:/awslabs/aws-data-wrangler/blob/066b81a5778cca7ce5ea8d889a2fec9824d8996a/awswrangler/s3/_list.py#L97

You might consider making "PaginationConfig" configurable to the caller so that MaxItems can be set there, and/or maybe adding a flag to skip the paginator entirely if s3_additional_kwargs["MaxKeys"] < args["PaginationConfig"]["PageSize"].

P.S. Thanks for this great resource :)

Environment

Using wrangler version 2.11.0 in a Python 3.8 Lambda.

To Reproduce

This returns more than MaxKeys results:

wr.s3.list_objects(data_prefix, s3_additional_kwargs={"MaxKeys": 1})

This returns MaxKeys results:

client = boto3.client('s3')
client.list_objects_v2(Bucket=bucket, Prefix=prefix, MaxKeys=1)

Steps to reproduce the behavior.

P.S. Please do not attach files as it's considered a security risk. Add code snippets directly in the message body as much as possible.

[jaidisido]

Thanks for raising this @mccauleyp, it will be addressed in the PR above

[mccauleyp]

Awesome, thanks!