Add query graceful shutdown for rolling updates

[illusional]

• Add terminationGracePeriodSeconds to query kubernetes deployment
• Implement app.on_shutdown signal handler to wait for all asyncio tasks to complete before returning.
• Upgrade aiohttp == 0.7.3 to address tasks being cancelled before the on_shutdown method is called: on_cleanup / on_shutdown are called after active tasks on the event loop are canceled aio-libs/aiohttp#3593

Testing

• Adding a "wait n seconds" method that slept for n seconds, and returned the value of an environment variable. This environment variable meant I could track which version of the deployment my script ran against.
• Taking the deploy.yaml from the deploy query step of the dev deploy, adding the TEST_VALUE environment variable with some value and saving it as new-deploy.yaml
• Issuing the first wait request (for 50 seconds) (https://internal.hail.populationgenomics.org.au/$NAMESPACE/query/api/v1alpha/wait?duration=50)
• Issuing the new deploy with:

  kubectl -n $NAMESPACE apply -f new-deploy.yaml
  kubectl -n $NAMESPACE rollout status --timeout=10m deployment query

• When the new pod is created (seen with kubectl --namespace $NAMESPACE get pod), issue the second request to the wait method.
• If all goes well, you should have:
  • termination logs like those below,
  • the first request successfully fulfilled with the response of env value being None (filled by the first pod)
  • The second request successfully filled, but has the value of the environment value, the one you set in the deploy.yaml (it got scheduled to the new node)

Termination logs:

{"severity": "INFO", "levelname": "INFO", "asctime": "2021-02-24 23:22:40,472", "filename": "query.py", "funcNameAndLine": "on_shutdown:253", "message": "On shutdown request received, with 2 tasks left", "hail_log": 1}
++ term
++ kill -TERM 7
+ true
+ '[' no == yes ']'
+ trap - SIGTERM SIGINT
+ wait 7
{"severity": "INFO", "levelname": "INFO", "asctime": "2021-02-24 23:23:26,004", "filename": "hail_logging.py", "funcNameAndLine": "log:40", "message": "https GET /michaelfranklin/query/api/v1alpha/wait done in 50.029999999998836s: 200", "remote_address": "10.28.127.3", "request_start_time": "[24/Feb/2021:23:22:35 +0000]", "request_duration": 50.029999999998836, "response_status": 200, "x_real_ip": "124.170.20.28", "hail_log": 1}
{"severity": "INFO", "levelname": "INFO", "asctime": "2021-02-24 23:23:26,005", "filename": "query.py", "funcNameAndLine": "on_shutdown:255", "message": "Tasks have all completed.", "hail_log": 1}

Test duration endpoint

@routes.get('/api/v1alpha/wait')
async def wait_seconds(request):
    """
    Wait query.duration seconds before returning the request.
    """
    duration = request.query.get('duration')
    try:
        duration = int(duration)
    except Exception as e:
        return web.json_response({
            'error': f'Invalid parameter duration "{duration}": {e}',
        }, status=422)

    await asyncio.sleep(int(duration))
    e = os.getenv("TEST_VALUE", "None")
    return web.json_response({"d": f"You waited '{duration}' seconds!!", "env": e})

[daniel-goldstein]

Sorry to step on your toes here a little bit @illusional! I updated aiohttp to 0.7.4 so you don't have to bump it here.

[illusional]

Too easy, thanks @daniel-goldstein! I've resolved the conflict ☺️

[illusional]

Hey @danking, just wondered if you could assign this PR to you :)

[danking]

Thanks for the ping!

[danking]

An elegant, if zany solution! Looks right to me.

[daniel-goldstein]

check-services error: on-shutdown needs a # pylint: disable=unused-argument

[illusional]

Thanks @daniel-goldstein, I've fixed my linting in my local hail now to catch this (and the other PR's) issues.