Node groups' launch template "$Latest" version producing an infinite always-changing plan

[jaimehrubiks]

Description

Terraform will always show a difference in plan related to all the node groups (managed) launch template version and will apply the following change:

module.eks.module.node_groups.aws_eks_node_group.workers["general_purpose"] will be updated in-place
      <redacted>
      ~ launch_template {
            id      = "lt-08dadsf22asdd68e"
            name    = "ms-test-apps-general_purpose20210609165315229000000001"
          ~ version = "2" -> "$Latest"
        }

Where "2" can be different on each launch template. After the apply, a change is produced in the node groups. If you plan/apply again, same plan appears, in an infinite loop.

Versions

Module: v17.1.0

Terraform:

Terraform v0.14.11
+ provider registry.terraform.io/gavinbunney/kubectl v1.11.1
+ provider registry.terraform.io/hashicorp/aws v3.45.0
+ provider registry.terraform.io/hashicorp/cloudinit v2.2.0
+ provider registry.terraform.io/hashicorp/helm v2.1.2
+ provider registry.terraform.io/hashicorp/kubernetes v1.11.4
+ provider registry.terraform.io/hashicorp/local v1.4.0
+ provider registry.terraform.io/hashicorp/null v2.1.2
+ provider registry.terraform.io/hashicorp/random v2.3.1
+ provider registry.terraform.io/hashicorp/template v2.2.0
+ provider registry.terraform.io/terraform-aws-modules/http v2.4.1

Reproduction

Easiest way to reproduce the behavior:

1. Apply plan on v17.1.0
2. Apply again. You will see the change in the plan without making any changes to the node groups. From now on, a new plan will show always even without making changes on the terraform code.

Code Snippet to Reproduce

  node_groups_defaults = {
    subnets                   = data.terraform_remote_state.thisvpc-infrastructure.outputs.private_subnets_ids
    public_ip                 = false
    key_name                  = module.global_variables.admin_keyname
    iam_role_arn              = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/${data.terraform_remote_state.thisvpc-infrastructure.outputs.iam_role_node_instance_role_name}"
    create_launch_template    = true
    k8s_labels                = {
      "cluster" = var.cluster_name,
      "managed" = "true",
    }
  }

    general_purpose = { 
      desired_capacity = 1
      min_capacity     = 1
      max_capacity     = 10
      instance_types = ["t3.xlarge"]
      labels = {
        "node-role" = "worker"
      }
    }

Expected behavior

No launch template update if the node group is not changed.

Actual behavior

Launch template changes version from a static number to $Latest

Additional context

This issue is introduced on version v17.1.0

Probably introduced here #1419

[jaimehrubiks]

I think the issue comes from the removal of the ternary operator in the merged PR.

"$Latest" string is not the same as the output of https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/launch_template#latest_version

Edit: Not sure. I will be testing this today/tomorrow

Edit2:
I think this ternary operator condition is also wrong. Will keep testing tomorrow.
each.value["launch_template_version"] != null

Edit3:
This seems to work
version = each.value["launch_template_version"] != "$Latest" ? each.value["launch_template_version"] : aws_launch_template.workers[each.key].latest_version

But I want to do more tests, I'll let you know when I am more confident

Edit 4: Found the root cause:
https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_node_group

version - (Required) EC2 Launch Template version number. While the API accepts values like $Default and $Latest, the API will convert the value to the associated version number (e.g. 1) on read and Terraform will show a difference on next plan. Using the default_version or latest_version attribute of the aws_launch_template resource or data source is recommended for this argument.

Before creating a PR, I want to play around this feature, and see if we can delay the rolling update without knowing (or checking it in aws console) in advance the current template version. That could be good. I'll do the PR after testing if this is possible. I will try this by using "update_default_version" field on the launch template. Will start tomorrow

[llamahunter]

We have daily automated drift detection of our tf deployments, so this is kind of an annoyingly noisy side effect of our update to the latest aws eks module. Any ETA on the PR being accepted?

[toneill818]

Running into the same issue, each plan will have a change for the templates.

[antonbabenko]

This has been fixed in #1447 and v17.3.0 has been just released.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.