-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add S3 source against minio S3 object storage #1967
Comments
Related to #1161. "Object storage" which is not distantly located in the outer clouds provide faster latency due to smaller network lags and minio is a component often used to "bring data closer to computations", recently with support also for "legacy" protocols like ftp/sftp. |
S3 connections against non-AWS servers might be possible with the aws cli if allowing the AWS_ENDPOINT_URL environment variable to be used, see aws/aws-cli#1270 (comment) where this feature recently after 8 years since inception is now available as documented in https://aws.amazon.com/blogs/developer/new-improved-flexibility-when-configuring-endpoint-urls-with-the-aws-sdks-and-tools/. |
When setting up a connection to an S3 source and not using AWS but for example minio, the change in #1691 allows specifying the "endpoint-url" in source.yaml using a line that says "endpoint: https://my.s3.server.com". This setting cannot be made in the UI, as the S3 settings there does not provide a field for specifying the "endpoint-url".
It would neat to be able to make this setting from the UI and maybe directly from the CLI too, without having to manually edit the source.yaml-file.
Alternatives that I considered was to install and use "aws configure" to set the endpoint URL to use, but due to this long standing bug in the aws CLI, it doesn't work: aws/aws-cli#1270
I would like the "Amazon S3" settings to support also minio (and thereby also other compatible S3 sources), and ideally be renamed "S3" and allow configuration of the endpoint-url.
It would be nice if in addition (ie if not specified) the endpoint URL could be picked up from an environment variable. I also think the "minio client" is nicer to configure rather than the aws CLI if one needs to access data from various different S3 sources (other than just AWS).
The text was updated successfully, but these errors were encountered: