Every time apply is invoked, rebuilds aws_elasticsearch_domain even with no changes

[nodesocket]

Every time I invoke terraform apply even if none of the aws_elasticsearch_domain properties were changed, Terraform reconfigures the Elasticsearch domain which takes a very long time and requires double the resources and copying all the data.

I believe this may be related to passing a dynamic variable into the module for access_policies. Is this a known issue?

Terraform Version

Terraform v0.11.1

Affected Resource(s)

aws_elasticsearch_domain    

Terraform Configuration Files

module definition

variable "domain_name" {
    description = "Domain name of the Elasticsearch cluster"
}

variable "allowed_ips" {
    type = "list"
    description = "A list of allowed IPs that can connect to the Elasticsearch cluster"
}

variable "version" {
    description = "The version of Elasticsearch to run in the cluster"
    default = "5.5" // latest version
}

variable "data_node_count" {
    description = "The number of Elasticsearch data nodes"
    default = 2 // must be an even number
}

variable "data_node_instance_type" {
    description = "The instance type of each Elasticsearch data node"
    default = "r4.large.elasticsearch"
}

variable "master_node_count" {
    description = "The number of Elasticsearch dedicated master nodes"
    default = 3
}

variable "master_node_instance_type" {
    description = "The instance type of each Elasticsearch dedicated master node"
    default = "t2.medium.elasticsearch"
}

variable "ebs_volume_size" {
    description = "The EBS volume size of each Elasticsearch node"
    default = 200
}

resource "aws_elasticsearch_domain" "elasticsearch" {
    domain_name = "${var.domain_name}"
    elasticsearch_version = "${var.version}"

    cluster_config {
        instance_count = "${var.data_node_count}"
        instance_type = "${var.data_node_instance_type}"
        dedicated_master_enabled = true
        dedicated_master_count = "${var.master_node_count}"
        dedicated_master_type = "${var.master_node_instance_type}"
        zone_awareness_enabled = true
    }

    ebs_options {
        ebs_enabled = true
        volume_type = "gp2" // general purpose SSD
        volume_size = "${var.ebs_volume_size}"
    }

    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                }
            }
        ]
    }
    CONFIG

    snapshot_options {
        automated_snapshot_start_hour = 1 // 1:00am
    }
}

module instance

variable "justin_home_ip_addr" {
    type = "string"
    default = "1.2.3.4/32"
    description = "Justin home IP address"
}

module "elasticsearch" {
    source = "../../modules/elasticsearch"
    domain_name = "my-elasticsearch-dev"
    allowed_ips = [
        "${var.justin_home_ip_addr}",
        "2.3.4.5/32"
    ]
}

[apparentlymart]

Hi @nodesocket! Sorry for this weird behavior.

Are you able to share the terraform plan output showing Terraform's plan to replace the instance? I'm mainly curious to see just which attributes are changing, so if you're not able to share the full output it'd be useful to at least know if access_policies is changing, and if so whether the significant content is changing or just whitespace or bracket formatting.

Normalizing IAM policies is a common source of this sort of problem in the AWS provider since each AWS service seems to normalize them slightly differently; we may need to tweak the normalization logic a little to cover changes made by the ElasticSearch service.

[nodesocket]

@apparentlymart see the following related issue which has the plan and a change I made recommended by @jbardin. Still having the bug though.

hashicorp/terraform#16930 (comment)

[bflad]

@nodesocket would you be okay if we consolidated this repository's discussion down to #576 since it seems to be related to that older issue (if its nothing strange happening from your upstream issue). Thanks in advance!

[nodesocket]

Sure, consolidating to #576 makes sense. Thanks.

[bflad]

Thanks, closing this issue given above response.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!