Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Terraform considers my elasticsearch domain policy to be modified every time it runs. #576

Closed
hashibot opened this issue Jun 13, 2017 · 28 comments · Fixed by #4942
Closed
Labels
bug Addresses a defect in current functionality. service/elasticsearch Issues and PRs that pertain to the elasticsearch service.

Comments

@hashibot
Copy link

This issue was originally opened by @yeehaa123 as hashicorp/terraform#12390. It was migrated here as part of the provider split. The original body of the issue is below.


Hi there,

Thank you for opening an issue. Please note that we try to keep the Terraform issue tracker reserved for bug reports and feature requests. For general usage questions, please see: https://www.terraform.io/community.html.

Terraform Version

Run terraform -v to show the version. If you are not running the latest version of Terraform, please upgrade because your issue may have already been fixed.
8.8

Affected Resource(s)

Please list the resources as a list, for example:

  • aws_elasticsearch_domain_policy"
    If this issue appears to affect multiple resources, it may be an issue with Terraform's core, so please mention this.

Terraform Configuration Files

data "aws_iam_policy_document" "es" {
  statement {
    sid = "admin"
    principals = [
      {
        type        = "AWS"
        identifiers = ["*"]
      },
    ]
    actions = [
      "es:*"
    ]
    condition {
      test = "IpAddress"
      variable = "aws:SourceIp"
      values = [
        "95.97.17.4"
      ]
    }
  }

  statement {
    sid = "client"
    principals = [
      {
        type        = "AWS"
        identifiers = ["*"]
      },
    ]
    actions = [
      "es:ESHttpGet"
    ]
  }
}

resource "aws_elasticsearch_domain_policy" "main" {
  domain_name = "${aws_elasticsearch_domain.domain.domain_name}"
  access_policies = "${data.aws_iam_policy_document.es.json}"
}

Debug Output

Please provider a link to a GitHub Gist containing the complete debug output: https://www.terraform.io/docs/internals/debugging.html. Please do NOT paste the debug output in the issue; just paste a link to the Gist.

Panic Output

If Terraform produced a panic, please provide a link to a GitHub Gist containing the output of the crash.log.

Expected Behavior

What should have happened?

Actual Behavior

What actually happened?

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

  1. terraform apply

Important Factoids

Are there anything atypical about your accounts that we should know? For example: Running in EC2 Classic? Custom version of OpenStack? Tight ACLs?

  • This behavior is especially problematic, because this resource takes a long time to modify. Min 15 min, up to an hour.
  • even though it's said to be fixed with pr 8648

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:
-5067
-8648

@hashibot hashibot added the bug Addresses a defect in current functionality. label Jun 13, 2017
@sirlancealittle
Copy link

sirlancealittle commented Jun 14, 2017

experiencing the same issue:

Terraform Version

$ terraform -v
Terraform v0.9.8

Affected Resource

  • aws_elasticsearch_domain

Terraform Configuration File

resource "aws_elasticsearch_domain" "sandbox" {
    domain_name                     = "sandbox"
    elasticsearch_version           = "5.3"

    cluster_config {
        instance_type               = "t2.medium.elasticsearch"
        instance_count              = 3
    }

    ebs_options {
        ebs_enabled                 = true
        volume_size                 = 20
        volume_type                 = "gp2"
    }

    advanced_options {
        "rest.action.multi.allow_explicit_index" = "true"
    }

    access_policies                 = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Condition": {
                "IpAddress": { "aws:SourceIp": "${ aws_eip.nat.public_ip }/32" }
            }
        }
    ]
}
POLICY

    tags {
        Domain =  "sandbox"
    }
}

Debug Output

https://gist.github.com/sirlancealittle/0142c73aa7ace8ad6ea3f9db1bd6ab28

Expected Behavior

I would expect that this not force a change

Actual Behavior

A change is forced on every run

Steps to Reproduce

terraform plan || terraform apply

Interesting Factoids

It appears that a reference to the arn of the elasticsearch domain is being added by the provider during the apply, however when state is read back and compared to the inline access policy it is finding that the arn reference is missing and registering it as a change.

The first attempt to work around this was to add the arn reference to the policy but it appears inline policy does not allow self references.

Terraform Configuration

resource "aws_elasticsearch_domain" "sandbox" {
    domain_name                     = "sandbox"
    elasticsearch_version           = "5.3"

    cluster_config {
        instance_type               = "t2.medium.elasticsearch"
        instance_count              = 3
    }

    ebs_options {
        ebs_enabled                 = true
        volume_size                 = 20
        volume_type                 = "gp2"
    }

    advanced_options {
        "rest.action.multi.allow_explicit_index" = "true"
    }

    access_policies                 = <<POLICY
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Resource": "${ self.arn }",     # << ADDED THIS
            "Condition": {
                "IpAddress": { "aws:SourceIp": "${ aws_eip.nat.public_ip }/32" }
            }
        }
    ]
}
POLICY

    tags {
        Domain = "Sandbox"
    }
}

Debug Output

$ tf plan

1 error(s) occurred:

* module root: 1 error(s) occurred:

* resource 'aws_elasticsearch_domain.sandbox' config: cannot contain self-reference self.arn

@brendanjerwin
Copy link

I, too, am experiencing this. its pretty harsh since changes to es domains take at least 10 minutes to apply... Is there any discussion somewhere as to a fix? Any workarounds?

@brendanjerwin
Copy link

duplicate of #205 ?

@brendanjerwin
Copy link

Here is an example of the policy changes its applying:

Before

 {
        "Statement": [
            {
                "Action": [
                    "es:RemoveTags",
                    "es:ListTags",
                    "es:ListDomainNames",
                    "es:ESHttpPut",
                    "es:ESHttpPost",
                    "es:ESHttpHead",
                    "es:ESHttpGet",
                    "es:ESHttpDelete",
                    "es:DescribeElasticsearchDomains",
                    "es:DescribeElasticsearchDomainConfig",
                    "es:DescribeElasticsearchDomain",
                    "es:AddTags"
                ],
                "Condition": {
                    "IpAddress": {
                        "aws:SourceIp": "10.0.0.0/16"
                    }
                },
                "Effect": "Allow",
                "Principal": {
                    "AWS": [
                        "arn:aws:iam::11111111111111:root",
                        "arn:aws:iam::111111111111111:user/user1",
                        "arn:aws:iam::111111111111111:user/user2t"
                    ]
                },
                "Resource": "arn:aws:es:us-east-1:1111111111111111:domain/es-domain-here/*",
                "Sid": "system-access"
            }
        ],
        "Version": "2012-10-17"
    }

After

 {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "system-access",
                "Effect": "Allow",
                "Action": [
                    "es:RemoveTags",
                    "es:ListTags",
                    "es:ListDomainNames",
                    "es:ESHttpPut",
                    "es:ESHttpPost",
                    "es:ESHttpHead",
                    "es:ESHttpGet",
                    "es:ESHttpDelete",
                    "es:DescribeElasticsearchDomains",
                    "es:DescribeElasticsearchDomainConfig",
                    "es:DescribeElasticsearchDomain",
                    "es:AddTags"
                ],
                "Principal": {
                    "AWS": [
                        "arn:aws:iam::11111111111111:root",
                        "arn:aws:iam::111111111111111:user/user1",
                        "arn:aws:iam::111111111111111:user/user2t"
                    ]
                },
                "Condition": {
                    "IpAddress": {
                        "aws:SourceIp": "10.0.0.0/16"
                    }
                }
            }
        ]
    }
}

Diff

screen shot 2017-09-01 at 7 54 04 am

@brendanjerwin
Copy link

Ah, I switched to using https://www.terraform.io/docs/providers/aws/r/elasticsearch_domain_policy.html and it seems to have made it happy. (why is the other method of applying policies even allowed?)

@sjourdan
Copy link

I confirm using the ES domain resource policy fixed the same issue for me, thanks @brendanjerwin

@nodesocket
Copy link

I am also having this issue, see #2636. @brendanjerwin what is the workaround? Thanks.

@brendanjerwin
Copy link

@nodesocket
Copy link

nodesocket commented Dec 13, 2017

@brendanjerwin darn, switching to a separate aws_elasticsearch_domain_policy resource did not fix it. Still rebuilding my ES cluster every time I invoke apply. I think it is because I am using an interpolation function jsonencode() which dynamically builds up JSON?

Any ideas or tricks to try?

variable "allowed_ips" {
    type = "list"
    description = "A list of allowed IPs that can connect to the Elasticsearch cluster"
}

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"

    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                }
            }
        ]
    }
    CONFIG
}

@nodesocket
Copy link

nodesocket commented Dec 15, 2017

Wanted to add the plan to show what Terraform is saying is different. The two JSON's appears to be identical, excluding whitespace. Any ideas?

  ~ module.elasticsearch.aws_elasticsearch_domain.elasticsearch
      access_policies:                      "{\"Statement\":[{\"Action\":\"es:*\",\"Condition\":{\"IpAddress\":{\"aws:SourceIp\":[\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}},\"Effect\":\"Allow\",\"Principal\":{\"AWS\":\"*\"},\"Resource\":\"arn:aws:es:us-west-2:391582443503:domain/duocircle-elasticsearch-dev/*\"}],\"Version\":\"2012-10-17\"}" => "    {\n        \"Version\": \"2012-10-17\",\n        \"Statement\": [\n            {\n                \"Action\": [\n                    \"es:*\"\n                ],\n                \"Principal\": {\n                    \"AWS\": \"*\"\n                },\n                \"Effect\": \"Allow\",\n                \"Condition\": {\n                    \"IpAddress\": {\"aws:SourceIp\": [\"69.181.22.201/32\",\"142.134.194.183/32\",\"54.149.205.143/32\",\"54.148.222.11/32\",\"54.68.193.51/32\",\"54.186.172.23/32\",\"54.186.60.165/32\",\"54.191.158.99/32\",\"54.149.154.28/32\",\"54.148.229.97/32\",\"54.149.206.185/32\",\"54.186.27.61/32\",\"54.191.214.3/32\",\"54.148.30.215/32\",\"54.186.22.84/32\",\"52.28.30.98/32\",\"52.58.5.29/32\",\"52.58.7.81/32\",\"52.58.7.120/32\",\"52.29.162.96/32\",\"52.29.144.204/32\",\"52.29.142.239/32\",\"52.29.118.68/32\",\"54.213.22.21/32\",\"54.200.247.200/32\",\"52.10.130.167/32\",\"52.10.99.51/32\",\"52.26.49.97/32\",\"54.68.34.165/32\",\"54.69.62.154/32\",\"54.149.26.35/32\",\"54.149.35.133/32\",\"54.186.218.12/32\",\"54.200.129.228/32\",\"54.148.113.140/32\",\"54.148.38.162/32\",\"54.148.165.188/32\",\"54.68.138.64/32\",\"54.149.88.251/32\",\"54.149.240.58/32\",\"54.186.10.118/32\",\"54.187.218.212/32\",\"54.187.213.119/32\",\"54.187.206.49/32\",\"54.187.71.48/32\",\"54.148.218.146/32\",\"54.149.34.179/32\",\"54.186.22.84/32\",\"54.186.57.195/32\",\"54.187.63.214/32\",\"52.28.251.132/32\",\"52.58.109.202/32\",\"52.28.147.211/32\",\"52.58.97.209/32\",\"52.58.19.153/32\",\"52.28.246.64/32\",\"52.28.59.28/32\",\"52.28.6.212/32\",\"52.58.96.151/32\",\"52.29.156.81/32\",\"34.223.207.43/32\"]}\n                }\n            }\n        ]\n    }\n    "

@Puneeth-n
Copy link
Contributor

the issue is Aws reformats your policy and therefore terraform reports it as a change. use the policy that aws reformats after you terraform and update your policy in terraform. it’s an old known issue.

@nodesocket
Copy link

@Puneeth-n can you point to the issue that describes this? We should try and get a fix. Just FYI, I tried using ignore_changes on the aws_elasticsearch_domain_policy but still having problems. See hashicorp/terraform#16930 (comment)

@Puneeth-n
Copy link
Contributor

@nodesocket
Copy link

nodesocket commented Jan 23, 2018

Update

I was able to work around this issue by using:

    lifecycle {
        ignore_changes = [
            "access_policies"
        ]
    }

But even breaking the policy into a separate aws_elasticsearch_domain_policy resource still causes a rebuild of the Elasticsearch cluster each apply:

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"

    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                }
            }
        ]
    }
    CONFIG
}

@ebarault
Copy link

ebarault commented Feb 8, 2018

i confirm using aws_elasticsearch_domain_policy solved that issue for me

@frittentheke
Copy link

frittentheke commented Mar 7, 2018

It's the order of the elements ("version" moved to the end) that causes the (apparent) change. Using an inline-policy you might be able to work around this, but when using aws_iam_policy_document data to create a policy you are lost -> this is a bug.

@adamgotterer
Copy link

adamgotterer commented May 16, 2018

aws_elasticsearch_domain_policy did not prevent a modify on every plan for me on 1.18.0. I tried inlining, loading from a file, and rendering from a template. All approaches appear to be adding line breaks and white space which is causing the diff:

  ~ module.elasticsearch.aws_elasticsearch_domain_policy.es
      access_policies:                                    "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":\"*\",\"Action\":[\"es:ESHttpPut\",\"es:ESHttpPost\",\"es:ESHttpHead\",\"es:ESHttpGet\"],\"Resource\":\"arn:aws:es:us-east-1:615190663081:domain/staging-knotch/*\"}]}" => "  {\n      \"Version\": \"2012-10-17\",\n      \"Statement\": [\n          {\n              \"Action\": [\n                \"es:ESHttpPut\",\n                \"es:ESHttpPost\",\n                \"es:ESHttpHead\",\n                \"es:ESHttpGet\"\n              ],\n              \"Principal\": \"*\",\n              \"Effect\": \"Allow\"\n          }\n      ]\n  }\n  "

@bflad
Copy link
Contributor

bflad commented May 16, 2018

@adamgotterer the plan looks like the AWS API is returning a Resource field ("Resource": "arn:aws:es:us-east-1:615190663081:domain/staging-knotch/*") and your configuration is trying to remove it. What happens if you add this to your policy configuration?

@adamgotterer
Copy link

@bflad adding resource fixed it for me. Thanks!

@nodesocket
Copy link

@adamgotterer I still encounter this problem, every time Terraform modifies the aws_elasticsearch_domain_policy. Where did you add resource to fix it?

@adamgotterer
Copy link

@nodesocket my resources look like:

resource "aws_elasticsearch_domain" "es" {
 ...
}

resource "aws_elasticsearch_domain_policy" "es" {
  domain_name = "${aws_elasticsearch_domain.es.domain_name}"
  access_policies = <<POLICIES
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Action": [
                "es:ESHttpPut",
                "es:ESHttpPost",
                "es:ESHttpHead",
                "es:ESHttpGet"
              ],
              "Principal": "*",
              "Effect": "Allow",
              "Resource": "${aws_elasticsearch_domain.es.arn}/*"
          }
      ]
  }
  POLICIES
}

I removed access_policies from aws_elasticsearch_domain and migrated to aws_elasticsearch_domain_policy. When looking at the plan diff I noticed "/*" was being added to the resource path, so double check for that.

@nodesocket
Copy link

nodesocket commented May 16, 2018

@adamgotterer yeah, still seeing aws_elasticsearch_domain_policy being updated every apply. Ideas?

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"
    access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:*"
            ],
            "Principal": {
                "AWS": "*"
            },
            "Effect": "Allow",
            "Condition": {
                "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
            }
        }
    ]
}
CONFIG
}

@adamgotterer
Copy link

adamgotterer commented May 16, 2018

@nodesocket you are missing the resource key.

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"
    access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "es:*"
            ],
            "Principal": {
                "AWS": "*"
            },
            "Effect": "Allow",
            "Condition": {
                "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
            },
            "Resource": "${aws_elasticsearch_domain. elasticsearch.arn}" #<==========
        }
    ]
}
CONFIG
}

@nodesocket
Copy link

nodesocket commented May 16, 2018

Adding resource stopped Terrform from changing the policy each apply, but it also broke my access to Elasticsearch. Note, my Terraform definition I tried was:

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"
    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                },
                "Resource": "${aws_elasticsearch_domain.elasticsearch.arn}"
            }
        ]
    }
CONFIG

@nodesocket
Copy link

The trick is:

"Resource": "${aws_elasticsearch_domain.elasticsearch.arn}/*"

Adding /*. Sigh. Thanks for the help @adamgotterer

@tomelliff
Copy link
Contributor

I think this can be closed because people seem to be happy with either using the aws_elasticsearch_domain_policy resource or you can build the resource ARN for the access_policy parameter in the aws_elasticsearch_domain resource manually with something like this:

resource "aws_elasticsearch_domain" "es" {
  domain_name           = "${var.domain}"
  ...

  access_policies = <<CONFIG
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": "es:*",
            "Principal": "*",
            "Effect": "Allow",
            "Condition": {
                "IpAddress": {"aws:SourceIp": ["66.193.100.22/32"]}
            },
            "Resource": "arn:aws:es:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:domain/${var.domain}/*"
        }
    ]
}
CONFIG
}

I'll raise a PR for the docs to add the resource element to make it a bit easier for future users.

@bflad
Copy link
Contributor

bflad commented Jun 22, 2018

Documentation updated via #4942 and will release with version 1.25.0 of the AWS provider, likely middle of next week. If there are further issues here, please open a new issue. Thanks!

@ghost
Copy link

ghost commented Apr 5, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 5, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. service/elasticsearch Issues and PRs that pertain to the elasticsearch service.
Projects
None yet
Development

Successfully merging a pull request may close this issue.