feat!: Add support for Outposts, remove node security group, add support for addon preserve and most_recent configurations

[bryantbiggs]

Description

List of backwards incompatible changes

• The cluster_id output used to output the name of the cluster. This is due to the fact that the cluster name is a unique constraint and therefore its set as the unique identifier within Terraform's state map. However, starting with local EKS clusters created on Outposts, there is now an attribute returned from the aws eks create-cluster API named id. The cluster_id has been updated to return this value which means that for current, standard EKS clusters created in the AWS cloud, no value will be returned (at the time of this writing) for cluster_id and only local EKS clusters on Outposts will return a value that looks like a UUID/GUID. Users should switch all instances of cluster_id to use cluster_name before upgrading to v19. Reference
• Minimum supported version of Terraform AWS provider updated to v4.45 to support latest features provided via the resources utilized.
• Minimum supported version of Terraform updated to v1.0
• Individual security group created per EKS managed node group or self managed node group has been removed. This configuration went mostly un-used and would often cause confusion ("Why is there an empty security group attached to my nodes?"). This functionality can easily be replicated by user's providing one or more externally created security groups to attach to nodes launched from the node group.
• Previously, var.iam_role_additional_policies (one for each of the following: cluster IAM role, EKS managed node group IAM role, self-managed node group IAM role, and Fargate Profile IAM role) accepted a list of strings. This worked well for policies that already existed but failed for policies being created at the same time as the cluster due to the well known issue of unkown values used in a for_each loop. To rectify this issue in v19.x, two changes were made:
  1. var.iam_role_additional_policies was changed from type list(string) to type map(string) -> this is a breaking change. More information on managing this change can be found below, under Terraform State Moves
  2. The logic used in the root module for this variable was changed to replace the use of try() with lookup(). More details on why can be found here
• The cluster name has been removed from the Karpenter module event rule names. Due to the use of long cluster names appending to the provided naming scheme, the cluster name has moved to a ClusterName tag and the event rule name is now a prefix. This guarantees that users can have multiple instances of Karpenter withe their respective event rules/SQS queue without name collisions, while also still being able to identify which queues and event rules belong to which cluster.

Added

• Support for setting preserve as well as most_recent on addons.
  • preserve indicates if you want to preserve the created resources when deleting the EKS add-on
  • most_recent indicates if you want to use the most recent revision of the add-on or the default version (default)

Modified

• cluster_security_group_additional_rules and node_security_group_additional_rules have been modified to use lookup() instead of try() to avoid the well known issue of unkown values within a for_each loop
• block_device_mappings previously required a map of maps but has since changed to an array of maps. Users can remove the outer key for each block device mapping and replace the outermost map {} with an array []. There are no state changes required for this change.
• node_security_group_ntp_ipv4_cidr_block previously defaulted to ["0.0.0.0/0"] and now defaults to ["169.254.169.123/32"] (Referenc: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html)
• node_security_group_ntp_ipv6_cidr_block previously defaulted to ["::/0"] and now defaults to ["fd00:ec2::123/128"] (Referenc: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html)
• create_kms_key previously defaulted to false and now defaults to true. Clusters created with this module now default to enabling secret encryption by default with a customer managed KMS key created by this module
• cluster_encryption_config previously used a type of list(any) and now uses a type of any -> users can simply remove the outer [...] brackets on v19.x
  • cluster_encryption_config previously defaulted to [] and now defaults to {resources = ["secrets"]} to encrypt secrets by default
• cluster_endpoint_public_access previously defaulted to true and now defaults to false. Clusters created with this module now default to private only access to the cluster endpoint
  • cluster_endpoint_private_access previously defaulted to false and now defaults to true
• The addon configuration now sets "OVERWRITE" as the default value for resolve_conflicts to ease addon upgrade management. Users can opt out of this by instead setting "NONE" as the value for resolve_conflicts
• The kms module used has been updated from v1.0.2 to v1.1.0 - no material changes other than updated to latest

⚠️ See UPGRADE-19.0.md for full list of changes and upgrade guidance

Motivation and Context

• Add support for latest EKS features provided by the AWS provider such as Outposts support and addition addon configuration settings
• Add support for Terraform AWS provider v4.x feature releases
• Resolves Attribute-Based Instance Type Selection for node groups #1800
• Resolves add datasource for aws eks describe-addon-versions #1908
• Resolves Adding additional policy to node group IAM role #2053
• Resolves Add support for "preserve" argument for addons #2054
• Resolves Added support for network_card_index #2261
• Resolves fix: Invalid for_each argument with var.create_cluster_primary_security_group_tags = false #2264
• Resolves feat: Allow AWS EKS addon timeouts to be configured #2273
• Resolves Invalid for_each argument on aws_default_tags #2281
• Resolves aws_cloudwatch_event_rule limits cluster name to 35 characters #2311

Breaking Changes

• Yes, see UPGRADE-19.0.md for full list of changes and upgrade guidance

How Has This Been Tested?

• I have updated at least one of the examples/* to demonstrate and validate my change(s)
• I have tested and validated these changes using one or more of the provided examples/* projects

• I have executed pre-commit run -a on my pull request

[antonbabenko]

Looks pretty good! Just a few minor comments here and there.

[antonbabenko]

Fixed in 87ced51

When specifying create_kms_key = false, this error happens.

cluster_encryption_config = {
    provider_key_arn = aws_kms_key.eks.arn
    resources        = ["secrets"]
}
create_kms_key = false

╷
│ Error: Unsupported attribute
│ 
│   on .terraform/modules/eks/main.tf line 350, in resource "aws_iam_policy" "cluster_encryption":
│  350:         Resource = var.create_kms_key ? [module.kms.key_arn] : [for config in var.cluster_encryption_config : config.provider_key_arn]
│ 
│ Can't access attributes on a primitive-typed value (string).
╵

[antonbabenko]

After a small fix in the upgrade guide, it looks great to me. I have successfully applied this PR and migrated from v18. No problems so far :)

[bryantbiggs]

FYI - we are validating deployment of EKS on Outposts in local mode and further changes may be added

[antonbabenko]

LGTM (still)

[bryantbiggs]

bug found in provider, converting to draft until fixed and supported hashicorp/terraform-provider-aws#27560

[antonbabenko]

This PR is included in version 19.0.0 🎉

[github-actions]

I'm going to lock this pull request because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.