Add endpoint options to Amazon S3 Storage

[zinrai]

I want to use an Amazon S3 compatible service.
I want to add an endpoint option to the storage_options of Amazon S3 Storage.

[shevron]

Hi,

Looks good - I'm just making sure that this has been tested with plain S3 as well - that is, that passing None as the endpoint URL causes boto to fall back to the default URL, and not go crazy in some sense. Unfortunately, I don't have the means of testing it myself right now.

[zinrai]

I checked the behavior when set endpoint_url=None for boto3.

# docker pull datopian/giftless:0.5.0
0.5.0: Pulling from datopian/giftless
69692152171a: Pull complete
66a3c154490a: Pull complete
82a02637ec8f: Pull complete
d8b2fbd22335: Pull complete
e04b68c1294e: Pull complete
c82d0e158525: Pull complete
b9d75891fa63: Pull complete
86f9fad6fa7e: Pull complete
8493dceb4a6d: Pull complete
87e47e2283f7: Pull complete
c77b7c14ee90: Pull complete
25b75f059cd6: Pull complete
bfda31167a7a: Pull complete
dc37863108fb: Pull complete
f5426d00752f: Pull complete
Digest: sha256:62ece51318a7efc5b7efc76ce28d068b66c0ec90b47a55537ad118b45a765c00
Status: Downloaded newer image for datopian/giftless:0.5.0
docker.io/datopian/giftless:0.5.0

# docker run -d datopian/giftless:0.5.0
fab7a01309d63d02447bfd0ff124b54f24f0700ff7325ae9c537c3ec37b06040
# docker ps
CONTAINER ID   IMAGE                     COMMAND                  CREATED         STATUS         PORTS      NAMES
fab7a01309d6   datopian/giftless:0.5.0   "tini uwsgi -- -s 12…"   6 seconds ago   Up 6 seconds   5000/tcp   vigilant_chebyshev

# docker exec -it fab7a01309d6 /bin/bash
giftless@fab7a01309d6:~$ python3
Python 3.7.10 (default, May 12 2021, 16:15:12)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> s3 = boto3.resource('s3', endpoint_url=None, region_name="ap-northeast-1", aws_access_key_id="*****", aws_secret_access_key="*****)
>>> bucket = s3.Bucket("giftless-boto3-test")
>>> print(bucket)
s3.Bucket(name='giftless-boto3-test')
>>> print([obj_summary.key for obj_summary in bucket.objects.all()])
['hoge.txt']
>>>

giftless@fab7a01309d6:~$ python3
Python 3.7.10 (default, May 12 2021, 16:15:12)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> s3 = boto3.client('s3', endpoint_url=None, region_name="ap-northeast-1", aws_access_key_id="*****", aws_secret_access_key="*****")
>>> response = s3.list_objects_v2(Bucket="giftless-boto3-test")
>>> for object in response['Contents']:
...     print(object['Key'])
...
hoge.txt
>>>

[rufuspollock]

@zinrai this looks good and i'm merging. /cc @shevron hope that is ok.