-
Notifications
You must be signed in to change notification settings - Fork 3.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
custom-resource-handlers: buffered asset download #29898
Labels
@aws-cdk/aws-s3
Related to Amazon S3
effort/small
Small work item – less than a day of effort
feature-request
A feature should be added or improved.
p2
Comments
nmussy
added
feature-request
A feature should be added or improved.
needs-triage
This issue or PR still needs to be triaged.
labels
Apr 19, 2024
khushail
added
investigating
This issue is being investigated and/or work is in progress to resolve the issue.
p2
effort/small
Small work item – less than a day of effort
and removed
needs-triage
This issue or PR still needs to be triaged.
investigating
This issue is being investigated and/or work is in progress to resolve the issue.
labels
Apr 19, 2024
This was referenced Apr 22, 2024
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
@aws-cdk/aws-s3
Related to Amazon S3
effort/small
Small work item – less than a day of effort
feature-request
A feature should be added or improved.
p2
Describe the feature
This issue is part feature request, part bug report.
S3 custom resource should be able to handle arbitrarily large asset files efficiently
Use Case
Deploying large asset files with an S3 custom resource causes the deployment Lambda to run out of memory The function's memory can be increased to compensate for bigger file sizes, but only up to a point and at additional costs.
Proposed Solution
It doesn't seem like the AWS CLI
s3
commands support buffered download. As proposed in #29862 (comment), using boto3'smultipart_threshold
should allow the file to be downloaded and written to disk in multiple parts.There might be other places where this change would be necessary, but this is the one that caused the initial issue:
aws-cdk/packages/@aws-cdk/custom-resource-handlers/lib/aws-s3-deployment/bucket-deployment-handler/index.py
Lines 171 to 180 in cfd336e
Other Information
See #29862 for the original issue
Acknowledgements
CDK version used
2.134.0
Environment details (OS name and version, etc.)
N/A
The text was updated successfully, but these errors were encountered: