Error tagging resources for target_group #16860
Comments
github-actions
bot
added
the
needs-triage
|
We're encountering the same issue occasionally on Terraform 0.12.28 with version 3.21.0 of the AWS provider and gathered some more information from CloudTrail following a failed execution. A successful CreateTargetGroup: {
"eventVersion": "1.08",
"userIdentity": <redacted>
"eventTime": "2021-01-18T17:52:26Z",
"eventSource": "elasticloadbalancing.amazonaws.com",
"eventName": "CreateTargetGroup",
"awsRegion": "eu-west-1",
"sourceIPAddress": <redacted>,
"userAgent": "aws-sdk-go/1.36.0 (go1.14.5; linux; amd64) exec-env/AWS_ECS_EC2 APN/1.0 HashiCorp/1.0 Terraform/0.12.28 (+https://www.terraform.io)",
"requestParameters": {
"unhealthyThresholdCount": 3,
"healthCheckTimeoutSeconds": 29,
"healthyThresholdCount": 2,
"protocol": "HTTP",
"matcher": {
"httpCode": "200"
},
"targetType": "instance",
"healthCheckPort": "8000",
"healthCheckPath": <redacted>,
"vpcId": <redacted>,
"port": 8000,
"healthCheckProtocol": "HTTP",
"healthCheckIntervalSeconds": 30,
"name": <redacted>,
"healthCheckEnabled": true
},
"responseElements": {
"targetGroups": [
{
"targetGroupArn": <redacted>,
"healthCheckPort": "8000",
"healthCheckPath": <redacted>,
"healthCheckEnabled": true,
"healthCheckTimeoutSeconds": 29,
"protocol": "HTTP",
"healthCheckProtocol": "HTTP",
"unhealthyThresholdCount": 3,
"healthCheckIntervalSeconds": 30,
"port": 8000,
"matcher": {
"httpCode": "200"
},
"targetGroupName": <redacted>,
"vpcId": <redacted>,
"protocolVersion": "HTTP1",
"targetType": "instance",
"healthyThresholdCount": 2
}
]
},
"requestID": <redacted>,
"eventID": <redacted>,
"readOnly": false,
"eventType": "AwsApiCall",
"apiVersion": "2015-12-01",
"managementEvent": true,
"eventCategory": "Management",
"recipientAccountId": <redacted>
}AddTags: {
"eventVersion": "1.08",
"userIdentity": <redacted>,
"eventTime": "2021-01-18T17:52:27Z",
"eventSource": "elasticloadbalancing.amazonaws.com",
"eventName": "AddTags",
"awsRegion": "eu-west-1",
"sourceIPAddress": <redacted>,
"userAgent": "aws-sdk-go/1.36.0 (go1.14.5; linux; amd64) exec-env/AWS_ECS_EC2 APN/1.0 HashiCorp/1.0 Terraform/0.12.28 (+https://www.terraform.io)",
"errorCode": "TargetGroupNotFoundException",
"errorMessage": "One or more target groups not found",
"requestParameters": {
"tags": <redacted>
"resourceArns": [
<same ARN as $.responseElements.targetGroups[0].targetGroupArn in CreateTargetGroup>
]
},
"responseElements": null,
"requestID": <redacted>,
"eventID": <redacted>,
"readOnly": false,
"eventType": "AwsApiCall",
"apiVersion": "2015-12-01",
"managementEvent": true,
"eventCategory": "Management",
"recipientAccountId": <redacted>
}After
Assuming the CloudTrail Events are always ordered correctly, it might be of significance that during creation of a successful target group the |
|
My team is also running this issue on TF 0.13.5 and AWS provider 3.22. It seems the provider tries to create tags even though the target group does not yet exist |
|
In order to reproduce this behavior, I ran the acc test about 20 times, but could not reproduce the error. TF version is |
|
Hi @shuheiktgw, many thanks for your investigation so far! I haven't tested this for v0.14.x due to some module incompatibilities yet. However, in v0.13.x (and also 0.12.x as posted by other guys above) the error occurs occasionally. Could you please have a look at ticket #12427 (I referred to this in the initial post). They had a pretty similar problem with other artifacts and solved it. Maybe their sync/wait-implementation could be reused for target groups?! Tags are pretty important for billing, tracing and automation. Would be great if you could somehow make this work reliably. Thanks in advance and best regards, |
|
Tackled it in #17280. Hope the PR will be reviewed soon. |
|
I'm seeing the same behavior as the original author, and I JUST updated to the latest terraform release (0.14.6). Even more concerning is that somehow it gets the ARN from the newly created target group, but somehow fails to consider it as valid. I'm also doing tagging. Another TG was created a few lines after I started create the above TG, and this is the relevant logs lines from that run. It sure smells like some weird race condition in the AWS provider code (trying to do the two requests at the same time), but the timing in my case from the creation and the tag appear to be many minutes apart. |
|
I have the same issue (Terraform 0.12.25), but in my case debug log shows that Terraform sends requests in series, only after it receives aws' response that the target group was created: And debug shows that only after terraform gets confirmation response from AWS about group creation it sends AddTags request: So for me it looks more like resource inconsistency on AWS' side than terraform issue. In terraform debug logs I always see AddTags request for Target Groups only after it receives target group creation confirmation - whether TG was created shortly after request or after some delay. |
YakDriver
removed
the
needs-triage
|
This should be resolved with #17280. If you continue to have issues, please open a new issue and let us know. |
|
Thanks for your work! Hopefully this solves the problem once and for all. Special thanks to @shuheiktgw and the other guys supporting this issue by describing their similar problem here. |
|
This has been released in version 3.29.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks! |
|
I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks! |
ghost
locked as resolved and limited conversation to collaborators
Community Note
Terraform CLI and Terraform AWS Provider Version
Terraform v0.13.5
hashicorp/aws 3.21.0
Note: Most recent core version is 0.14.3. However, a switch is not possible right now since some modules (e.g. terraform-aws-modules/rds/aws in version ~> 2.0) do not support the recent 0.14.x versions.
Affected Resource(s)
Terraform Configuration Files
The setting is pretty ordinary. An excerpt of the hcl definitions as follows:
Debug Output
Expected Behavior
TargetGroup could be found and related Tags have been created.
Actual Behavior
TargetGroup could not be found such that Tags cannot be created as well.
Steps to Reproduce
The bug occurs occasionally and cannot be reproduced in a reliable manner. However, the nature of this bug was found (and fixed) already for other resources in Error tagging resources #12427 and Error tagging resources #24395. I guess target_group resource was missed to fixed that time.
References
The text was updated successfully, but these errors were encountered: