Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

lifecycle ignore_changes not working, still showing ~ update in-place #16930

Closed
nodesocket opened this issue Dec 15, 2017 · 12 comments
Closed

lifecycle ignore_changes not working, still showing ~ update in-place #16930

nodesocket opened this issue Dec 15, 2017 · 12 comments
Labels
waiting-response An issue/pull request is waiting for a response from the community

Comments

@nodesocket
Copy link

Due to an outstanding bug in the AWS provider and Elasticsearch service I am looking to ignore the access_policies property in a aws_elasticsearch_domain_policy.

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"

    lifecycle {
        ignore_changes = [
            "access_policies"
        ]
    }

    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                }
            }
        ]
    }
CONFIG
}

However when I run plan, I am still seeing ~ update in-place related to the access_policies. How can I completely ignore changes to access_policies and not apply?

  ~ module.elasticsearch.aws_elasticsearch_domain.elasticsearch
      access_policies:                      "{\"Statement\":[{\"Action\":\"es:*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}},\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Resource\":\"arn:aws:es:us-west-2:391582443503:domain/duocircle-elasticsearch-dev/*\"}],\"Version\":\"2012-10-17\"}" => "    {\n        \"Version\": \"2012-10-17\",\n        \"Statement\": [\n            {\n                \"Action\": [\n                    \"es:*\"\n                ],\n                \"Principal\": {\n                    \"AWS\": \"*\"\n                },\n                \"Effect\": \"Allow\",\n                \"Condition\": {\n                    \"IpAddress\": {\"aws:SourceIp\": [\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}\n                }\n            }\n        ]\n    }\n    "
@jbardin
Copy link
Member

jbardin commented Dec 15, 2017

Hi @nodesocket,

I'm not able to reproduce this, at least in my config the ignore_changes is working correctly. What version of Terraform are you using?

Is there anything else in that aws_elasticsearch_domain_policy block that might be changing, and is this the exactly config verbatim?

@jbardin jbardin added the waiting-response An issue/pull request is waiting for a response from the community label Dec 15, 2017
@nodesocket
Copy link
Author

@jbardin:

Running v0.11.1.

Here is the full configuration I am using. Do I need to also apply the ignore_changes on the parent aws_elasticsearch_domain?

variable "domain_name" {
    description = "Domain name of the Elasticsearch cluster"
}

variable "allowed_ips" {
    type = "list"
    description = "A list of allowed IPs that can connect to the Elasticsearch cluster"
}

variable "version" {
    description = "The version of Elasticsearch to run in the cluster"
    default = "5.5" // latest version
}

variable "data_node_count" {
    description = "The number of Elasticsearch data nodes"
    default = 2 // must be an even number
}

variable "data_node_instance_type" {
    description = "The instance type of each Elasticsearch data node"
    default = "r4.large.elasticsearch"
}

variable "master_node_count" {
    description = "The number of Elasticsearch dedicated master nodes"
    default = 3
}

variable "master_node_instance_type" {
    description = "The instance type of each Elasticsearch dedicated master node"
    default = "t2.medium.elasticsearch"
}

variable "ebs_volume_size" {
    description = "The EBS volume size of each Elasticsearch node"
    default = 200
}

resource "aws_elasticsearch_domain" "elasticsearch" {
    domain_name = "${var.domain_name}"
    elasticsearch_version = "${var.version}"

    cluster_config {
        instance_count = "${var.data_node_count}"
        instance_type = "${var.data_node_instance_type}"
        dedicated_master_enabled = true
        dedicated_master_count = "${var.master_node_count}"
        dedicated_master_type = "${var.master_node_instance_type}"
        zone_awareness_enabled = true
    }

    ebs_options {
        ebs_enabled = true
        volume_type = "gp2" // general purpose SSD
        volume_size = "${var.ebs_volume_size}"
    }

    snapshot_options {
        automated_snapshot_start_hour = 1 // 1:00am
    }
}

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"

    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                }
            }
        ]
    }
    CONFIG

    lifecycle {
        ignore_changes = [
            "access_policies"
        ]
    }
}

@nodesocket
Copy link
Author

@jbardin

I tried adding:

    lifecycle {
        ignore_changes = [
            "access_policies"
        ]
    }

To the resource aws_elasticsearch_domain as well as the resource aws_elasticsearch_domain_policy, but still seeing ~ update in-place:

  ~ module.elasticsearch.aws_elasticsearch_domain.elasticsearch
      access_policies: "{\"Statement\":[{\"Action\":\"es:*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}},\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Resource\":\"arn:aws:es:us-west-2:391582443503:domain/duocircle-elasticsearch-dev/*\"}],\"Version\":\"2012-10-17\"}" => "    {\n        \"Version\": \"2012-10-17\",\n        \"Statement\": [\n            {\n                \"Action\": [\n                    \"es:*\"\n                ],\n                \"Principal\": {\n                    \"AWS\": \"*\"\n                },\n                \"Effect\": \"Allow\",\n                \"Condition\": {\n                    \"IpAddress\": {\"aws:SourceIp\": [\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}\n                }\n            }\n        ]\n    }\n    "

Any ideas? This is blocking since rebuilding our 100GB Elasticsearch cluster every terraform apply is a no go.

@jbardin
Copy link
Member

jbardin commented Dec 19, 2017

@nodesocket,

I can't seem to replicate this, even using the exact config you've provided.
What version of aws provider are you using?
Can you provide a full debug log from terraform plan?

@nodesocket
Copy link
Author

@jbardin are you using it as a module as well?

variable "web_public_ip_addrs" {
    type = "list"
    default = [
        "51.1.2.3/32",
        "51.1.2.4/32"
    ]
}

variable "justin_home_ip_addr" {
    default = "1.2.3.4/32"
}

variable "adam_home_ip_addr" {
    default = "2.3.4.5/32"
}

module "elasticsearch" {
    source = "../../modules/elasticsearch"
    domain_name = "my-elasticsearch-dev"
    allowed_ips = [
        "${var.justin_home_ip_addr}",
        "${var.adam_home_ip_addr}",
        "${var.web_public_ip_addrs}"
    ]
}
MacBook-Pro ➜  dev git:(master) terraform -v
Terraform v0.11.1
+ provider.aws v1.3.0

@jbardin
Copy link
Member

jbardin commented Dec 20, 2017

Just tried as a module, passing things in various ways that I thought might break, but still no luck here.

I did notice something, which may be leading us in the right direction, is that technically the allowed_ips declaration you have here should contain a nested list, but it gets flattened out. In theory this should have required something like:

"${concat(list(var.justin_home_ip_addr, var.adam_home_ip_addr), var.web_public_ip_addrs)}"

Now regardless of what that outputs, the access_policies field should still be ignored anyway, but this is the only thing I've managed to spot that looks out of the ordinary.

@nodesocket
Copy link
Author

nodesocket commented Dec 22, 2017

@jbardin ok, I changed the declaration to:

module "elasticsearch" {
    source = "../../modules/elasticsearch"
    domain_name = "my-elasticsearch-dev"
    allowed_ips = "${
        concat(
            list(
                var.justin_home_ip_addr,
                var.adam_home_ip_addr
            ),
            var.web_public_ip_addrs
        )
    }"
}

Still seeing it wanted to do ~ update in-place. I'm thinking is just needs to possibly do this once to cleanup? Should I run apply and then check?

@nodesocket
Copy link
Author

nodesocket commented Dec 22, 2017

@jbardin darn it.

Just ran apply and after around 16 minutes of the reconfigure, then running a plan and still seeing the same thing.... Argggg.

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ module.elasticsearch.aws_elasticsearch_domain.elasticsearch
      access_policies: "{\"Statement\":[{\"Action\":\"es:*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}},\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Resource\":\"arn:aws:es:us-west-2:391582443503:domain/duocircle-elasticsearch-dev/*\"}],\"Version\":\"2012-10-17\"}" => "    {\n        \"Version\": \"2012-10-17\",\n        \"Statement\": [\n            {\n                \"Action\": [\n                    \"es:*\"\n                ],\n                \"Principal\": {\n                    \"AWS\": \"*\"\n                },\n                \"Effect\": \"Allow\",\n                \"Condition\": {\n                    \"IpAddress\": {\"aws:SourceIp\": [\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}\n                }\n            }\n        ]\n    }\n    "


Plan: 0 to add, 1 to change, 0 to destroy.

@jbardin
Copy link
Member

jbardin commented Dec 22, 2017

I'm starting to wonder if the access_policies is a red herring, and there's another part of the diff that triggering the change but remaining hidden for some reason.

Could you capture the full log output from a plan with TF_LOG=trace? You can use TF_LOG_PATH= to save the output to a file if you want. Scan the file for sensitive info first in you're going to post it here, or you can encrypt it with our public key. You can send it directly to me if you prefer too.

@nodesocket
Copy link
Author

nodesocket commented Dec 26, 2017

Ok, I manually deleted everything in the directory .terraform/plugins and then did a terraform init -upgrade and got further perhaps. But, now when I run plan I am seeing a panic: runtime error: invalid memory address or nil pointer dereference which I reported in a new issue hashicorp/terraform-provider-aws#2772.

Once I get past that, I will advise on this original issue.

@bflad
Copy link
Contributor

bflad commented Jan 23, 2018

@nodesocket sorry you have been having trouble with this! Looks like hashicorp/terraform-provider-aws#2772 was fixed in terraform-provider-aws version 1.7.0 so you should be good for testing again. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

@ghost
Copy link

ghost commented Apr 5, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@ghost ghost locked and limited conversation to collaborators Apr 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
waiting-response An issue/pull request is waiting for a response from the community
Projects
None yet
Development

No branches or pull requests

3 participants