Implement multipart copy and copying a particular version #308
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Summary of changes: - `s3_copy` now supports a `version` keyword argument that facilitates copying a specified version of an object. - A new function `s3_multipart_copy` to mirror `s3_multipart_upload` has been added, which calls `UploadPartCopy` in the API. - An explicit `cp(::S3Path, ::S3Path)` method has been implemented, which avoids the fallback `cp(::AbstractPath, ::AbstractPath)` method that reads the source file into memory before writing to the destination. - `cp(::S3Path, ::S3Path)` allows the user to opt into a multipart copy, in which case multipart is used when the source is larger than the specified part size (50 MiB by default). A multipart copy is unconditionally used when the source is at least 5 GiB. This behavior mimics that of the AWS CLI. Note that this now requires an additional API call to `HeadObject` in order to retrieve the source size.
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Comment on lines
507
to
512
| to_bucket, | ||
| to_path, | ||
| "$bucket/$path", | ||
| source, | ||
| Dict("headers" => headers); | ||
| aws_config=aws, | ||
| kwargs..., |
There was a problem hiding this comment.
[JuliaFormatter] reported by reviewdog 🐶
Suggested change
| to_bucket, | |
| to_path, | |
| "$bucket/$path", | |
| source, | |
| Dict("headers" => headers); | |
| aws_config=aws, | |
| kwargs..., | |
| to_bucket, to_path, source, Dict("headers" => headers); aws_config=aws, kwargs... |
|
bors try |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I've had these changes locally for months (possibly a year or more?) but hadn't committed or pushed them. I don't know if/when I'll have the bandwidth to ensure this gets over the finish line, so if someone is interested in picking this up then please feel free to do so.
Summary of changes:
s3_copynow supports aversionkeyword argument that facilitates copying a specified version of an object.s3_multipart_copyto mirrors3_multipart_uploadhas been added, which callsUploadPartCopyin the API.cp(::S3Path, ::S3Path)method has been implemented, which avoids the fallbackcp(::AbstractPath, ::AbstractPath)method that reads the source file into memory before writing to the destination.cp(::S3Path, ::S3Path)allows the user to opt into a multipart copy, in which case multipart is used when the source is larger than the specified part size (50 MiB by default). A multipart copy is unconditionally used when the source is at least 5 GiB. This behavior mimics that of the AWS CLI. Note that this now requires an additional API call toHeadObjectin order to retrieve the source size.