Add CommandComplete waiter to ssm GetCommandInvocation operation

[stobrien89]

Originated from: boto/botocore#1744
Linked PR: boto/botocore#1753

[stobrien89]

P48336721

[iainelder]

@stobrien89 , thanks for creating this.

Would it be a new waiter to accompany or replace the existing command-executed waiter?

I'd really like a waiter that doesn't fail because "doesn't exist yet" and doesn't fail because the command failed. I just need to know when I can retrive the result of the invocation.

I find that command-executed behaves very strangely in boto3 1.17.102. I created the following function to learn how SSM works:

def test_ec2():
    ssm = boto3.client("ssm")

    instance_id = "i-1111111111111111"

    response = ssm.send_command(
        InstanceIds=[instance_id],
        DocumentName="AWS-RunShellScript",
        Parameters={"commands": ["missing"]}
    )

    command_id = response["Command"]["CommandId"]

    waiter = ssm.get_waiter("command_executed")

    while True:
        try:
            waiter.wait(
                CommandId=command_id,
                InstanceId=instance_id
            )
        except WaiterError as ex:
            print("!" * 40)
            print(ex.last_response)

            error = ex.last_response.get("Error")

            if error:
                if error["Code"] == "InvocationDoesNotExist":
                    time.sleep(0.1)
                    continue

            if ex.last_response["Status"] == "Failed":
                break

            raise ex

    output = ssm.get_command_invocation(
        CommandId=command_id,
        InstanceId=instance_id
    )

    print(output)

The function just tries to run a command that shouldn't exist on an EC2 instance and gets the result.

It needs to run the waiter in a loop with a complex try-catch inside to get the behavior I want.

You can see the waiter first raises an exception because of an InvocationDoesNotExist error, and then raises another exception because the comand failed.

Apart from the request ID, the content of the exception from command failure seems identical to the invocation result.

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
{'Error': {'Message': '', 'Code': 'InvocationDoesNotExist'}, 'ResponseMetadata': {'RequestId': '11111111-1111-1111-1111-111111111111', 'HTTPStatusCode': 400, 'HTTPHeaders': {'server': 'Server', 'date': 'Sat, 03 Jul 2021 18:47:13 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '35', 'connection': 'keep-alive', 'x-amzn-requestid': '11111111-1111-1111-1111-111111111111'}, 'RetryAttempts': 0}}
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
{'CommandId': '22222222-2222-2222-2222-222222222222', 'InstanceId': 'i-1111111111111111', 'Comment': '', 'DocumentName': 'AWS-RunShellScript', 'DocumentVersion': '$DEFAULT', 'PluginName': 'aws:runShellScript', 'ResponseCode': 127, 'ExecutionStartDateTime': '2021-07-03T18:47:14.275Z', 'ExecutionElapsedTime': 'PT0.042S', 'ExecutionEndDateTime': '2021-07-03T18:47:14.275Z', 'Status': 'Failed', 'StatusDetails': 'Failed', 'StandardOutputContent': '', 'StandardOutputUrl': '', 'StandardErrorContent': '/var/lib/amazon/ssm/i-1111111111111111/document/orchestration/22222222-2222-2222-2222-222222222222/awsrunShellScript/0.awsrunShellScript/_script.sh: line 1: missing: command not found\nfailed to run commands: exit status 127', 'StandardErrorUrl': '', 'CloudWatchOutputConfig': {'CloudWatchLogGroupName': '', 'CloudWatchOutputEnabled': False}, 'ResponseMetadata': {'RequestId': '33333333-3333-3333-3333-333333333333', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'Server', 'date': 'Sat, 03 Jul 2021 18:47:19 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '807', 'connection': 'keep-alive', 'x-amzn-requestid': '33333333-3333-3333-3333-333333333333'}, 'RetryAttempts': 0}
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
{'CommandId': '22222222-2222-2222-2222-222222222222', 'InstanceId': 'i-1111111111111111', 'Comment': '', 'DocumentName': 'AWS-RunShellScript', 'DocumentVersion': '$DEFAULT', 'PluginName': 'aws:runShellScript', 'ResponseCode': 127, 'ExecutionStartDateTime': '2021-07-03T18:47:14.275Z', 'ExecutionElapsedTime': 'PT0.042S', 'ExecutionEndDateTime': '2021-07-03T18:47:14.275Z', 'Status': 'Failed', 'StatusDetails': 'Failed', 'StandardOutputContent': '', 'StandardOutputUrl': '', 'StandardErrorContent': '/var/lib/amazon/ssm/i-1111111111111111/document/orchestration/22222222-2222-2222-2222-222222222222/awsrunShellScript/0.awsrunShellScript/_script.sh: line 1: missing: command not found\nfailed to run commands: exit status 127', 'StandardErrorUrl': '', 'CloudWatchOutputConfig': {'CloudWatchLogGroupName': '', 'CloudWatchOutputEnabled': False}, 'ResponseMetadata': {'RequestId': '44444444-4444-4444-4444-444444444444', 'HTTPStatusCode': 200, 'HTTPHeaders': {'server': 'Server', 'date': 'Sat, 03 Jul 2021 18:47:19 GMT', 'content-type': 'application/x-amz-json-1.1', 'content-length': '807', 'connection': 'keep-alive', 'x-amzn-requestid': '44444444-4444-4444-4444-444444444444'}, 'RetryAttempts': 0}

[stobrien89]

Hi @iainelder,

This waiter would accompany the existing command-executed waiter, if implemented, as many people have been requesting this functionality. In the meantime, I'd suggest using a custom polling loop to get the desired behavior.

[stobrien89]

@iainelder,

Heard back from the ssm team today— They've added this request to their backlog, which is a good sign, but we don't have an estimated date of completion. I'll provide any additional updates I receive from them.

[github-actions]

This issue has not received any attention in 1 year. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[vparmeland]

Any updates?