Compress responses to clients sending accept-encoding: gzip header

[NoopDog]

No description provided.

[melainalegaspi]

@NoopDog to investigate potential gains. @noah-aviel-dove to investigate if we can enable gzip compression on API gateway.

[hannes-ucsc]

https://ucsc-gi.slack.com/archives/C705Y6G9Z/p1618623078221900?thread_ts=1618590719.217100&cid=C705Y6G9Z

[hannes-ucsc]

We're looking at a 15% improvement

assuming that we can reduce the green portion of the timing diagram down to 25%

[nadove-ucsc]

It's very easy to enable compression on API gateway. All we need to do is specify a nonnegative value for minimum_compression_size in the chalice config.

$ curl https://service.nadove.dev.singlecell.gi.ucsc.edu/index/catalogs -H 'accept-encoding: gzip'  --output -; echo
�Ցъ�1
     E�%�3�L]��+�R�4j�ږ6#�0�n�+
�"��>��$�9�	,mp�l�d�q
                         ����4p�
��O����Ѓޠ/��G��݀����uA��	��H��R,�c>J?��jl��%�y�:\���m���ZK�֊�.�S�E��Fu��U�2L�?��B/�������'ƺu���2�
                                                                                                 Bk�2,�Z���pf�z�{���M&��7�D0���?�������U�oM�
$ curl https://service.nadove.dev.singlecell.gi.ucsc.edu/index/catalogs -H 'accept-encoding: gzip'  --output - | gunzip -
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   230  100   230    0     0    236      0 --:--:-- --:--:-- --:--:--   236
{"default_catalog":"dcp2","catalogs":{"dcp2":{"internal":false,"atlas":"hca","plugins":[{"type":"repository","name":"tdr","sources":["tdr:broad-jade-dev-data:snapshot/hca_dev_20201217_test4___20210126"]},{"type":"metadata","name":"hca"}]},"dcp2ebi":{"internal":false,"atlas":"hca","plugins":[{"type":"repository","name":"tdr","sources":["tdr:broad-jade-dev-data:snapshot/hca_dev_20201023_ebiv4___20201121"]},{"type":"metadata","name":"hca"}]},"it2":{"internal":true,"atlas":"hca","plugins":[{"type":"repository","name":"tdr","sources":["tdr:broad-jade-dev-data:snapshot/hca_dev_20201217_test4___20210126"]},{"type":"metadata","name":"hca"}]},"it2ebi":{"internal":true,"atlas":"hca","plugins":[{"type":"repository","name":"tdr","sources":["tdr:broad-jade-dev-data:snapshot/hca_dev_20201023_ebiv4___20201121"]},{"type":"metadata","name":"hca"}]}}}

[nadove-ucsc]

I tried to time the requests to find the inflection point where compression becomes inefficient for small responses, but the round-trip elapsed time was too inconsistent regardless of payload size or compression to find any discernible pattern.

[hannes-ucsc]

This isn't observable in prod, only on dev.

dev:

prod:

[hannes-ucsc]

Hannes to investigate ^^^

[hannes-ucsc]

The most recent prod build log reveals that TF recreated the API Gateway during that build. This was expected since the upgrading instructions on another issue required deleting the API Gateway resource prior to make deploy. The initial configuration for that newly created resource had minimum_compression_size set to 0 as expected:

https://gitlab.azul.data.humancellatlas.org/ucsc/azul/-/jobs/1630#L5189

We checked the API Gateway settings in the AWS console and found the compression setting to be disabled. We ran make plan with prod selected and prod checked out and saw that TF wanted to update the minimum_compression_size setting from -1 (disabled) to 0 (enabled for all payloads). We retried the prod deploy job and it updated the setting from -1 to 0.

https://gitlab.azul.data.humancellatlas.org/ucsc/azul/-/jobs/1988#L3049

The AWS console now reflects that

but

$ http -h 'https://service.azul.data.humancellatlas.org/version'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 4783
Content-Type: application/json
Date: Tue, 11 May 2021 23:54:47 GMT
Via: 1.1 9d99e2b9dedf1fed654116378ecfcced.cloudfront.net (CloudFront)
X-Amz-Cf-Id: bxns3mAcH-qn35YSlrlmpueE2KxmVyNNBLNWKwUN3qyquQir7nHPYA==
X-Amz-Cf-Pop: LAX3-C4
X-Amzn-Trace-Id: Root=1-609b1947-54a647995c9ee0ca52e6c5ea;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: fMDjPE9cIAMFnwg=
x-amzn-RequestId: 464f96c8-cc7f-4c29-84f9-a7203f2b53cd

whereas on dev it works (look for Content-Encoding: gzip):

$ http  -h 'https://service.dev.singlecell.gi.ucsc.edu/version'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 1914
Content-Type: application/json
Date: Tue, 11 May 2021 23:55:50 GMT
Via: 1.1 bd310f631d1314e239dc9cd04967c61f.cloudfront.net (CloudFront)
X-Amz-Cf-Id: Y1dKkUx_MiVVZLPzciqjUo0Lv27wiTByHJJRVzF2qM6LXK29YymGBw==
X-Amz-Cf-Pop: LAX3-C3
X-Amzn-Trace-Id: Root=1-609b1986-3e0013cb4a6a5cb52bf9bb9e;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: fMDs-E_sIAMFfqQ=
x-amzn-RequestId: d47f8bef-fa92-4354-bb10-e981ed945460

[hannes-ucsc]

I'll wait until tomorrow, maybe there's a delay in propagating of the change to the CF edge servers.

[theathorn]

@hannes-ucsc to file a ticket with AWS support.

[hannes-ucsc]

Case # 8354981671

[hannes-ucsc]

Response from AWS support:

From further review, I discovered that the last recorded stage update was done on 2021-05-05T00:14:00.000Z UTC. Please note that any changes to your api are not live to the public until the updates are redeployed. To deploy from within the web console, go to 'Resources' from the navigation menu on the left. Then under 'Actions', select 'Deploy Api' and select the 'prod' stage. After a few seconds, you should be able to test with the latest saved settings.

That fixed it:

Timings vary but improved by the expected 10-20%, barely noticeable, subjectively speaking.

However, we shouldn't need to manually redeploy the stage. TF should do that for us. I was gonna chalk this up to #1951 but that fix was deployed at the same time, so it should have been active. My 2nd best guess is that the cause is in the vicinity of
#2992 since that also messes with deployment stages.