fix: check lowercase config keys with/without aws prefix in s3logstorefactory #2896
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ion-elgreco
requested review from
wjones127,
roeap,
rtyler and
hntd187
as code owners
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2896 +/- ##
==========================================
- Coverage 72.50% 72.49% -0.02%
==========================================
Files 131 131
Lines 40362 40421 +59
Branches 40362 40421 +59
==========================================
+ Hits 29266 29304 +38
- Misses 9216 9238 +22
+ Partials 1880 1879 -1 ☔ View full report in Codecov by Sentry. |
rtyler
force-pushed
the
fix/aws_logstorefactory
branch
3 times, most recently
from
abb01c8
to
8041814
Compare
…narios This should reduce bugs, reduce warnings, and improve performance when somebody is using this crate against an S3-alike service rather than AWS S3 directly. Currently I cannot think of a valid scenarijo where one would use AWS_ENDPOINT_URL against actual S3 buckets, so using that as the heuristic
rtyler
force-pushed
the
fix/aws_logstorefactory
branch
from
8041814
to
7055b09
Compare
|
keeping up with this branch, and my local dev environment is kind of a mess, but looks like this resolves my issue - "conditional_put", "CONDITIONAL_PUT", "aws_condititional_put", and "AWS_CONDITIONAL_PUT" all seem to work 👍 |
hntd187
approved these changes
Sep 23, 2024
| @@ -19,7 +19,7 @@ use deltalake_core::storage::StorageOptions; | |||
| use deltalake_core::DeltaResult; | |||
| use tracing::log::*; | |||
|
|
|||
| use crate::constants; | |||
| use crate::constants::{self, AWS_ENDPOINT_URL}; | |||
There was a problem hiding this comment.
prolly want to revert this and use consistent references here
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
We were only checking if the lowercase aws_copy_if_not_exists or aws_conditional_put are set to return the stores. But we should ignore the case and check for with and without aws prefix, since ObjectStore does this as well.
This pull request also short-circuits the loading of AWS configuration when using AWS_ENDPOINT_URL or scenarios where the client is not directly interacting with AWS S3 (e.g. Minio, Cloudflare, etc)