s3 streaming upload allocates 5-10x more mem than the multipart chunk size (oom risk) and fails to saturate net tx (inefficient) #2390

jdanbrown · 2017-01-18T01:43:01Z

I'm trying to stream a large (>5GB) byte stream from stdin to s3 using:

$ aws s3 cp --expected-size=X - s3://foo

but I keep running into OOM kills (and also inefficient net tx) when I use large values for --expected-size. I did a little digging into resource usage using dstat and pv and I'm observing that:

aws s3 cp allocates 7-10x more mem than the multipart upload chunk size, whereas I'd expect it to allocate ~1x chunk size + a small constant overhead (<100MB)
aws s3 cp fails to saturate net tx, alternating between two modes of resource usage:
1. steadily increasing mem usage, draining stdin ~25+ MB/s, and uploading to s3 only ~12 MB/s
2. peaks mem usage and stops draining stdin while s3 upload speed increases to only ~25 MB/s
  - gsutil cp foo s3://foo achieves ~40-50 MB/s net tx from the same host
3. deallocates a bunch of mem and starts over at (i)

Here are sample commands I'm using to observe these, along with the resource usage I observed for each:

# --expected-size 1 GB
# chunk size ~100 KB (8 MB min?)
# max mem usage ~115 MB
# net tx ~12-25 MB/s
(cat /dev/zero | pv -terbf | aws s3 cp --expected-size=$((1024*1024*1024 * 1)) - s3://foo) 2>&1 | tr '\r' '\n'

# --expected-size 10 GB
# chunk size ~1 MB (8 MB min?)
# max mem usage ~115 MB
# net tx ~12-25 MB/s
(cat /dev/zero | pv -terbf | aws s3 cp --expected-size=$((1024*1024*1024 * 10)) - s3://foo) 2>&1 | tr '\r' '\n'

# --expected-size 100 GB
# chunk size ~10 MB
# max mem usage ~175 MB
# net tx ~12 MB/s <-> ~25 MB/s
(cat /dev/zero | pv -terbf | aws s3 cp --expected-size=$((1024*1024*1024 * 100)) - s3://foo) 2>&1 | tr '\r' '\n'

# --expected-size 1000 GB
# chunk size ~100 MB
# max mem usage ~945 MB
# net tx ~12 MB/s <-> ~25 MB/s
(cat /dev/zero | pv -terbf | aws s3 cp --expected-size=$((1024*1024*1024 * 1000)) - s3://foo) 2>&1 | tr '\r' '\n'

# --expected-size 10000 GB
# chunk size 1 GB
# max mem usage ~7220 MB
# net tx ~0 MB/s -> ~8-12 MB/s -> ~12 MB/s <-> ~25 MB/s
(cat /dev/zero | pv -terbf | aws s3 cp --expected-size=$((1024*1024*1024 * 10000)) - s3://foo) 2>&1 | tr '\r' '\n'

Sample output:

# Output from `aws s3 cp ... | pv` on the left of '||':
$ (cat /dev/zero | pv -terbf | aws s3 cp --expected-size=$((1024*1024*1024 * 1000)) - s3://foo) 2>&1 | tr '\r' '\n'

# Output from `dstat` on the right of '||':
$ dstat --cpu --mem --disk --net --top-mem

# Output lines manually zipped and aligned to reflect real time:
# - The gaps in `pv` output (on the left) are when `aws s3 cp` stops draining stdin

                             ||  ----total-cpu-usage---- ------memory-usage----- -dsk/total- -net/total- --most-expensive-
                             ||  usr sys idl wai hiq siq| used  buff  cach  free| read  writ| recv  send|  memory process
                             ||    0   0  99   0   0   0|7642M 6744M 8967M 40.1G|   0  8192B|   0     0 |aws         8692k
                             ||    2   0  98   0   0   0|7656M 6744M 8967M 40.1G|   0     0 |   0     0 |aws         28.0M
 128KiB 0:00:01 [80.1KiB/s]  ||    1   1  98   0   0   0|7751M 6744M 8967M 40.0G|   0     0 |6069B 2782B|aws          124M
53.8MiB 0:00:02 [ 111MiB/s]  ||    1   1  98   0   0   0|7866M 6744M 8967M 39.9G|   0     0 |   0     0 |aws          237M
 174MiB 0:00:03 [ 120MiB/s]  ||    2   1  97   0   1   0|7944M 6744M 8967M 39.8G|   0   240k|   0     0 |aws          314M
 254MiB 0:00:04 [80.7MiB/s]  ||    2   1  97   0   0   0|7998M 6744M 8967M 39.8G|   0  8192B|   0     0 |aws          370M
 310MiB 0:00:05 [55.3MiB/s]  ||    2   1  97   0   0   0|8064M 6744M 8967M 39.7G|   0    16k|  94k 8201k|aws          435M
 373MiB 0:00:06 [63.2MiB/s]  ||    1   0  98   0   0   0|8101M 6744M 8967M 39.7G|   0   112k|  81k 7336k|aws          472M
 415MiB 0:00:07 [42.2MiB/s]  ||    1   1  97   0   0   0|8144M 6744M 8967M 39.6G|   0    88k| 112k 9276k|aws          514M
 456MiB 0:00:08 [40.6MiB/s]  ||    1   0  99   0   0   0|8187M 6744M 8967M 39.6G|   0    80k| 210k   13M|aws          562M
 501MiB 0:00:09 [45.7MiB/s]  ||    1   1  98   0   0   0|8222M 6744M 8967M 39.5G|   0  8192B| 143k 9422k|aws          591M
 535MiB 0:00:10 [34.1MiB/s]  ||    1   0  99   0   0   0|8259M 6744M 8967M 39.5G|   0   104k| 156k   10M|aws          628M
 570MiB 0:00:11 [34.2MiB/s]  ||    1   1  98   0   0   0|8292M 6744M 8967M 39.5G|   0     0 | 171k   11M|aws          661M
 606MiB 0:00:12 [36.1MiB/s]  ||    1   0  98   0   0   0|8324M 6744M 8967M 39.4G|   0   296k| 168k   10M|aws          695M
 640MiB 0:00:13 [33.9MiB/s]  ||    1   1  98   0   0   0|8351M 6744M 8967M 39.4G|   0   136k| 157k 9905k|aws          720M
 664MiB 0:00:14 [24.1MiB/s]  ||    1   0  99   0   0   0|8377M 6744M 8967M 39.4G|   0  8192B| 183k   11M|aws          749M
 694MiB 0:00:15 [  30MiB/s]  ||    1   1  98   0   0   0|8406M 6744M 8967M 39.4G|   0    96k| 177k   10M|aws          776M
 722MiB 0:00:16 [28.6MiB/s]  ||    1   0  98   0   0   1|8435M 6744M 8967M 39.3G|   0     0 | 205k   11M|aws          807M
 750MiB 0:00:17 [27.4MiB/s]  ||    1   1  98   0   0   0|8466M 6744M 8967M 39.3G|   0    72k| 209k   12M|aws          836M
 778MiB 0:00:18 [27.8MiB/s]  ||    1   0  98   0   1   0|8490M 6744M 8967M 39.3G|   0     0 | 217k   12M|aws          859M
 803MiB 0:00:19 [  26MiB/s]  ||    1   1  98   0   0   0|8513M 6744M 8967M 39.3G|   0  8192B| 211k   13M|aws          884M
 828MiB 0:00:20 [24.4MiB/s]  ||    1   0  98   0   0   0|8534M 6744M 8967M 39.2G|   0     0 | 204k   12M|aws          903M
 848MiB 0:00:21 [20.1MiB/s]  ||    1   1  98   0   0   0|8561M 6744M 8967M 39.2G|   0     0 | 226k   13M|aws          931M
 872MiB 0:00:22 [24.2MiB/s]  ||    0   0  99   0   0   0|8574M 6744M 8967M 39.2G|   0   176k| 329k   19M|aws          943M
                             ||    0   1  98   0   0   0|8573M 6744M 8967M 39.2G|   0    16k| 403k   24M|aws          943M
                             ||    0   1  99   0   0   0|8575M 6744M 8967M 39.2G|   0  8192B| 429k   25M|aws          943M
                             ||    1   1  97   0   0   0|8573M 6744M 8967M 39.2G|   0   152k| 418k   24M|aws          943M
                             ||    1   1  97   0   0   1|8571M 6744M 8967M 39.2G|   0     0 | 407k   24M|aws          943M
                             ||    1   1  98   0   0   1|8575M 6744M 8967M 39.2G|   0  1608k| 419k   24M|aws          943M
                             ||    0   0  99   0   0   0|8574M 6744M 8967M 39.2G|   0     0 | 422k   24M|aws          943M
                             ||    0   1  99   0   0   0|8573M 6744M 8967M 39.2G|   0  8192B| 451k   25M|aws          943M
                             ||    0   0  99   0   0   0|8573M 6744M 8967M 39.2G|   0     0 | 424k   25M|aws          943M
                             ||    0   1  99   0   0   0|8573M 6744M 8967M 39.2G|   0     0 | 430k   24M|aws          943M
                             ||    0   0  98   0   0   1|8574M 6744M 8967M 39.2G|   0   224k| 425k   24M|aws          943M
                             ||    0   1  98   0   0   1|8574M 6744M 8967M 39.2G|   0    16k| 453k   25M|aws          943M
                             ||    0   0  99   0   0   0|8573M 6744M 8967M 39.2G|   0  8192B| 465k   25M|aws          943M
                             ||    0   1  98   0   0   1|8576M 6744M 8967M 39.2G|   0   120k| 439k   25M|aws          943M
 896MiB 0:00:36 [1.58MiB/s]  ||    0   0  99   0   0   0|8458M 6744M 8967M 39.3G|   0     0 | 408k   21M|aws          829M
 896MiB 0:00:36 [1.58MiB/s]  ||    1   1  98   0   0   1|8485M 6744M 8967M 39.3G|   0    96k| 207k   11M|aws          852M
 923MiB 0:00:37 [27.2MiB/s]  ||    1   0  99   0   0   0|8508M 6744M 8967M 39.3G|   0     0 | 219k   12M|aws          875M
 945MiB 0:00:38 [22.2MiB/s]  ||    1   1  98   0   0   1|8532M 6744M 8967M 39.2G|   0   112k| 207k   13M|aws          898M
 969MiB 0:00:39 [24.2MiB/s]  ||    1   0  98   0   0   1|8555M 6744M 8967M 39.2G|   0     0 | 207k   12M|aws          922M
 991MiB 0:00:40 [22.1MiB/s]  ||    1   1  98   0   0   1|8581M 6744M 8967M 39.2G|   0   104k| 284k   16M|aws          945M
1021MiB 0:00:41 [29.6MiB/s]  ||    0   0 100   0   0   0|8578M 6744M 8967M 39.2G|   0    32k| 444k   25M|aws          945M
                             ||    0   1  99   0   0   0|8578M 6744M 8967M 39.2G|   0    16k| 460k   24M|aws          945M
                             ||    0   0  99   0   0   0|8579M 6744M 8967M 39.2G|   0   176k| 448k   24M|aws          945M
   1GiB 0:00:45 [ 786KiB/s]  ||    1   1  98   0   1   0|8471M 6744M 8967M 39.3G|   0   168k| 257k   15M|aws          839M
   1GiB 0:00:45 [ 786KiB/s]  ||    1   0  99   0   0   0|8492M 6744M 8967M 39.3G|   0     0 | 215k   12M|aws          861M
1.02GiB 0:00:46 [23.6MiB/s]  ||    1   1  97   0   0   0|8516M 6744M 8967M 39.3G|   0   288k| 198k   12M|aws          883M
1.04GiB 0:00:47 [22.1MiB/s]  ||    1   0  98   0   1   0|8537M 6744M 8967M 39.2G|   0     0 | 212k   11M|aws          904M
1.07GiB 0:00:48 [22.2MiB/s]  ||    1   1  98   0   0   0|8565M 6744M 8967M 39.2G|   0    24k| 236k   14M|aws          933M
1.09GiB 0:00:49 [22.9MiB/s]  ||    0   0  99   0   0   0|8468M 6744M 8967M 39.3G|   0     0 | 242k   13M|aws          835M
1.12GiB 0:00:50 [30.8MiB/s]  ||    1   1  98   0   0   0|8366M 6744M 8967M 39.4G|   0   144k| 227k   12M|aws          733M
1.14GiB 0:00:51 [25.9MiB/s]  ||    1   0  99   0   0   1|8388M 6744M 8967M 39.4G|   0   176k| 219k   13M|aws          755M
1.17GiB 0:00:52 [23.3MiB/s]  ||    1   1  98   0   0   0|8409M 6744M 8967M 39.4G|   0    88k| 209k   13M|aws          778M
1.19GiB 0:00:53 [22.9MiB/s]  ||    1   0  98   0   1   0|8436M 6744M 8967M 39.3G|   0  8192B| 237k   13M|aws          804M
1.21GiB 0:00:54 [23.9MiB/s]  ||    0   1  98   0   0   0|8464M 6744M 8967M 39.3G|   0     0 | 289k   15M|aws          832M
1.24GiB 0:00:55 [27.9MiB/s]  ||    1   0  98   0   0   1|8479M 6744M 8967M 39.3G|   0   120k| 228k   12M|aws          852M
1.26GiB 0:00:56 [  24MiB/s]  ||    1   1  98   0   0   0|8503M 6744M 8967M 39.3G|   0     0 | 229k   12M|aws          874M
1.28GiB 0:00:57 [  21MiB/s]  ||    1   0  99   0   0   0|8399M 6744M 8967M 39.4G|   0     0 | 228k   12M|aws          767M
1.31GiB 0:00:58 [22.1MiB/s]  ||    1   1  98   0   0   0|8421M 6744M 8967M 39.3G|   0  8192B| 225k   12M|aws          791M
1.33GiB 0:00:59 [21.4MiB/s]  ||    0   0  99   0   0   0|8450M 6744M 8967M 39.3G|   0   112k| 249k   14M|aws          818M
1.35GiB 0:01:00 [25.2MiB/s]  ||    1   1  98   0   0   0|8473M 6744M 8967M 39.3G|   0     0 | 206k   13M|aws          843M
1.38GiB 0:01:01 [29.6MiB/s]  ||    1   1  98   0   0   0|8377M 6744M 8967M 39.4G|   0  1424k| 194k   11M|aws          748M
 1.4GiB 0:01:02 [  23MiB/s]  ||    1   1  98   0   0   0|8404M 6744M 8967M 39.4G|   0   120k| 227k   12M|aws          772M
1.43GiB 0:01:03 [29.5MiB/s]  ||    1   1  98   0   0   0|8432M 6744M 8967M 39.3G|   0    24k| 207k   12M|aws          799M
1.46GiB 0:01:04 [26.7MiB/s]  ||    1   1  98   0   0   0|8457M 6744M 8967M 39.3G|   0   160k| 270k   15M|aws          826M
1.48GiB 0:01:05 [25.7MiB/s]  ||    1   0  98   0   0   0|8480M 6744M 8967M 39.3G|   0    80k| 223k   13M|aws          847M
1.51GiB 0:01:06 [25.9MiB/s]  ||    1   1  98   0   0   0|8500M 6744M 8967M 39.3G|   0   128k| 233k   13M|aws          869M
1.53GiB 0:01:07 [21.1MiB/s]  ||    1   0  99   0   0   0|8523M 6744M 8967M 39.2G|   0     0 | 199k   11M|aws          894M
1.55GiB 0:01:08 [24.1MiB/s]  ||    2   1  97   0   0   1|8547M 6744M 8967M 39.2G|   0  8192B| 217k   11M|aws          914M
1.57GiB 0:01:09 [23.1MiB/s]  ||    1   0  99   0   0   0|8572M 6744M 8967M 39.2G|   0     0 | 298k   16M|aws          940M
 1.6GiB 0:01:10 [22.9MiB/s]  ||    1   1  98   0   0   0|8579M 6744M 8967M 39.2G|   0   240k| 363k   21M|aws          946M
1.62GiB 0:01:11 [28.6MiB/s]  ||    0   0  99   0   0   1|8578M 6744M 8967M 39.2G|   0     0 | 433k   25M|aws          946M
                             ||    0   1  98   0   0   1|8580M 6744M 8967M 39.2G|   0     0 | 460k   25M|aws          946M
                             ||    0   0  99   0   1   0|8578M 6744M 8967M 39.2G|   0   144k| 443k   25M|aws          946M
                             ||    0   1  98   0   0   1|8580M 6744M 8967M 39.2G|   0     0 | 447k   25M|aws          946M
                             ||    0   0  99   0   0   0|8579M 6744M 8967M 39.2G|   0    72k| 420k   25M|aws          946M
                             ||    0   1  99   0   0   0|8579M 6744M 8967M 39.2G|   0    16k| 458k   26M|aws          946M
                             ||    0   0  99   0   0   1|8578M 6744M 8967M 39.2G|   0     0 | 464k   24M|aws          946M
                             ||    0   1  98   0   0   1|8578M 6744M 8967M 39.2G|   0   104k| 425k   23M|aws          946M
                             ||    0   0  99   0   0   0|8577M 6744M 8967M 39.2G|   0     0 | 450k   25M|aws          946M
                             ||    0   1  99   0   0   0|8578M 6744M 8967M 39.2G|   0   104k| 463k   25M|aws          946M
1.63GiB 0:01:23 [49.2KiB/s]  ||    0   0  99   0   0   1|8456M 6744M 8967M 39.3G|   0   232k| 435k   23M|aws          825M
1.63GiB 0:01:23 [49.2KiB/s]  ||    1   1  98   0   0   0|8481M 6744M 8967M 39.3G|   0    96k| 209k   12M|aws          848M
1.65GiB 0:01:24 [23.4MiB/s]  ||    1   1  98   0   0   0|8501M 6744M 8967M 39.3G|   0    16k| 213k   12M|aws          870M
1.67GiB 0:01:25 [21.9MiB/s]  ||    1   1  98   0   0   0|8525M 6744M 8967M 39.2G|   0     0 | 234k   13M|aws          892M
1.69GiB 0:01:26 [21.6MiB/s]  ||    1   1  97   0   0   1|8544M 6744M 8967M 39.2G|   0     0 | 213k   12M|aws          915M
1.71GiB 0:01:27 [23.4MiB/s]  ||    0   1  99   0   0   0|8570M 6744M 8967M 39.2G|   0     0 | 291k   16M|aws          938M
1.74GiB 0:01:28 [23.8MiB/s]  ||    0   0  99   0   0   0|8578M 6744M 8967M 39.2G|   0   128k| 380k   20M|aws          946M
1.75GiB 0:01:29 [8.67MiB/s]  ||    0   0  99   0   0   0|7677M 6744M 8967M 40.1G|   0  8192B| 298k   16M|aws         49.0M

I'm running awscli 1.11.36 on k8s on aws ec2:

$ aws --version
aws-cli/1.11.36 Python/3.5.2 Linux/4.7.3-coreos-r3 botocore/1.4.93

Two other (old, closed) issues that sound possibly related:

The text was updated successfully, but these errors were encountered:

kyleknap · 2017-01-18T17:55:36Z

Thanks for the deep dive here.

This behavior seems to be as expected. Here is why it is expected:

For streaming uploads, the behavior is to stream in data into memory from stdin and allocate a chunk/part for each thread to upload. The entire chunk must be stored in memory because the stream cannot be rewound if errors are encountered and retries need to happen during the transfer. This means that the maximum memory should be chunksize * number_threads + some overhead (of 20 MB to 30 MB) which is about what you are seeing for the 1GB and 10GB expected size case. This calculated maximum is often reached when the input source can be read very fast, much faster than transferring to s3.
As to for the smaller uploads, the chunksize is set at 8MB instead of 100KB or 1MB because s3 has a minimum of 5MB for any part uploaded to s3. So the CLI cannot upload 100KB or 1MB parts because s3 will return errors
The reason there is an --expected-size argument is that since we are streaming the data in from stdin and that is not rewindable, there is no way to determine the size of the transfer. This is important because the maximum number of parts s3 allows in a multipart upload is 10,000 parts. So if the default configured chunksize * 10,000 is less than the size of the file, you will get errors from s3. So the --expected-size argument can help the CLI gauge what the appropriate chunksize is so it does not exceed this 10,000 part limit and fail the upload completely.

So for the case when you are setting the expected file size to 10,000 GB, the expected chunksize as you noted is 1GB due to the 10,000 part limit. However since there is 10 threads and each thread has to store the chunk/part into memory, that will increase max memory usage by 10x.

Based on the explanation, the best way to avoid the OOM errors would be to either decrease the thread count or use a file instead of stdin as the source of your data. I am not sure if there is much we can do architecturally in the CLI given the nature of unrewindable streams and the constraints of S3 API to improve this performance. Let us know if that makes sense. It might be worth improving our documentation to make this more clear.

jdanbrown · 2017-01-23T03:07:55Z

Ah ok, 10 threads explains the mem usage—thanks for the explanation. (Docs on controlling thread count, for reference.)

But what about streaming aws s3 cp only achieving ~12 or 25 MB/s net tx whereas gsutil cp foo s3://foo achieved ~40–50 MB/s? Based on your explanation of how streaming works I'd expect there's a way to make it saturate net tx.

MirkoRossini · 2017-10-05T14:08:49Z

Hello, I'm running into issues with aws s3 cp running out of memory, so I thought I would reply to this thread since it looks pertinent.
I'm trying to stream about 1 Terabyte of data to s3, I run this command:

stream | gzip | aws s3 cp --expected-size 858993459200 - s3:/path/to/file

Since I don't have much memory on the machine, I set up the following configuration in ~/.aws/config
[default] s3 = max_concurrent_requests = 1

There should be only one thread running, so I anticipate around 100M of memory usage.
Instead, the memory footprint keeps growing, ultimately to a size that my machine can't handle.

Am I misunderstanding the configuration effect?

kdaily · 2020-09-14T17:49:41Z

If this is still occurring with a more recent version of the AWS CLI, please let us know. Thanks!

github-actions · 2020-09-21T18:10:07Z

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.

vs-dsva · 2021-03-15T16:54:47Z

I am running this command:
pg_dump -v -h $PGHOST -U $PGUSER -p $PGPORT -Fc $DATABASE -B | aws s3 cp - s3://$AWS_S3_BUCKET/backup/$BACKUP_FILE --region $AWS_REGION --expected-size 60000000000
Memory is limited in my pod to 512MBytes:

CPU is pretty much constant

Network performance is pretty bad - 5Mbps:

kyleknap added response-needed s3 s3stream labels Jan 18, 2017

diehlaws added guidance Question that needs advice or information. needs-response and removed question labels Jan 4, 2019

hguturu mentioned this issue Jul 16, 2019

aws s3 cp streaming to stdout fails silently #3977

Closed

KaibaLopez removed the needs-response label Feb 26, 2020

kdaily added the response-requested Waiting on additional info and feedback. Will move to "closing-soon" in 7 days. label Sep 14, 2020

github-actions bot added closing-soon This issue will automatically close in 4 days unless further comments are made. closed-for-staleness and removed closing-soon This issue will automatically close in 4 days unless further comments are made. labels Sep 21, 2020

github-actions bot closed this as completed Sep 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

s3 streaming upload allocates 5-10x more mem than the multipart chunk size (oom risk) and fails to saturate net tx (inefficient) #2390

s3 streaming upload allocates 5-10x more mem than the multipart chunk size (oom risk) and fails to saturate net tx (inefficient) #2390

jdanbrown commented Jan 18, 2017

kyleknap commented Jan 18, 2017

jdanbrown commented Jan 23, 2017

MirkoRossini commented Oct 5, 2017

kdaily commented Sep 14, 2020

github-actions bot commented Sep 21, 2020

vs-dsva commented Mar 15, 2021

s3 streaming upload allocates 5-10x more mem than the multipart chunk size (oom risk) and fails to saturate net tx (inefficient) #2390

s3 streaming upload allocates 5-10x more mem than the multipart chunk size (oom risk) and fails to saturate net tx (inefficient) #2390

Comments

jdanbrown commented Jan 18, 2017

kyleknap commented Jan 18, 2017

jdanbrown commented Jan 23, 2017

MirkoRossini commented Oct 5, 2017

kdaily commented Sep 14, 2020

github-actions bot commented Sep 21, 2020

vs-dsva commented Mar 15, 2021