Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(aws_elasticsearch_domain) panic: runtime error: invalid memory address or nil pointer dereference #2772

Closed
nodesocket opened this issue Dec 26, 2017 · 16 comments
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/elasticsearch Issues and PRs that pertain to the elasticsearch service.
Milestone

Comments

@nodesocket
Copy link

nodesocket commented Dec 26, 2017

Running Terraform v0.11.1 with AWS provider 1.6. I have a crash.log as well if needed, but don't want to post it publically as it seems to contain sensitive information.

panic: runtime error: invalid memory address or nil pointer dereference
2017-12-26T16:02:56.764-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x26a808c]
2017-12-26T16:02:56.764-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:
2017-12-26T16:02:56.764-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: goroutine 30 [running]:
2017-12-26T16:02:56.764-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsElasticSearchDomainRead(0xc420aa65b0, 0x2f81da0, 0xc420010b40, 0x0, 0x4d57fc0)
2017-12-26T16:02:56.764-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_elasticsearch_domain.go:500 +0x94c
2017-12-26T16:02:56.764-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc42095fa40, 0xc420b974a0, 0x2f81da0, 0xc420010b40, 0xc420619018, 0xc420d56d01, 0x80000000018)
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:355 +0x199
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc420a6d810, 0xc420b97450, 0xc420b974a0, 0x5aa7d90, 0x0, 0x18)
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc420183c40, 0xc420c33040, 0xc420c33150, 0x0, 0x0)
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: reflect.Value.call(0xc420374cc0, 0xc420184378, 0x13, 0x3343a66, 0x4, 0xc42036ff20, 0x3, 0x3, 0x8, 0xc42062df10, ...)
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/usr/local/go/src/reflect/value.go:434 +0x906
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: reflect.Value.Call(0xc420374cc0, 0xc420184378, 0x13, 0xc420adbf20, 0x3, 0x3, 0x4, 0x500000003, 0x8)
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/usr/local/go/src/reflect/value.go:302 +0xa4
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: net/rpc.(*service).call(0xc42037d2c0, 0xc4201660f0, 0xc4201805b0, 0xc420574580, 0xc420143000, 0x2b37500, 0xc420c33040, 0x16, 0x2b37540, 0xc420c33150, ...)
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/usr/local/go/src/net/rpc/server.go:381 +0x142
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: created by net/rpc.(*Server).ServeCodec
2017-12-26T16:02:56.765-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/usr/local/go/src/net/rpc/server.go:475 +0x36b
2017/12/26 16:02:56 [ERROR] root.elasticsearch: eval: *terraform.EvalRefresh, err: aws_elasticsearch_domain.elasticsearch: unexpected EOF
2017/12/26 16:02:56 [ERROR] root.elasticsearch: eval: *terraform.EvalSequence, err: aws_elasticsearch_domain.elasticsearch: unexpected EOF
2017/12/26 16:02:56 [TRACE] [walkRefresh] Exiting eval tree: module.elasticsearch.aws_elasticsearch_domain.elasticsearch
2017/12/26 16:02:56 [TRACE] dag/walk: upstream errored, not walking "module.elasticsearch.aws_elasticsearch_domain_policy.main"
2017/12/26 16:02:56 [TRACE] dag/walk: upstream errored, not walking "provider.aws (close)"
2017/12/26 16:02:56 [TRACE] dag/walk: upstream errored, not walking "root"
2017/12/26 16:02:56 [DEBUG] plugin: waiting for all plugin processes to complete...
2017-12-26T16:02:56.770-0500 [DEBUG] plugin: plugin process exited: path=/Users/justin/Sites/dev/terraform/environments/dev/.terraform/plugins/darwin_amd64/terraform-provider-aws_v1.6.0_x4
2017-12-26T16:02:56.770-0500 [WARN ] plugin: error closing client during Kill: err="connection is shut down"



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https:/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Here is the config:

variable "domain_name" {
    description = "Domain name of the Elasticsearch cluster"
}

variable "allowed_ips" {
    type = "list"
    description = "A list of allowed IPs that can connect to the Elasticsearch cluster"
}

variable "version" {
    description = "The version of Elasticsearch to run in the cluster"
    default = "5.5" // latest version
}

variable "data_node_count" {
    description = "The number of Elasticsearch data nodes"
    default = 2 // must be an even number
}

variable "data_node_instance_type" {
    description = "The instance type of each Elasticsearch data node"
    default = "r4.large.elasticsearch"
}

variable "master_node_count" {
    description = "The number of Elasticsearch dedicated master nodes"
    default = 3
}

variable "master_node_instance_type" {
    description = "The instance type of each Elasticsearch dedicated master node"
    default = "t2.small.elasticsearch"
}

variable "ebs_volume_size" {
    description = "The EBS volume size of each Elasticsearch node"
    default = 200
}

resource "aws_cloudwatch_log_group" "es_index_slow_logs" {
    name = "es_index_slow_logs"
    retention_in_days = 14
}

resource "aws_cloudwatch_log_group" "es_search_slow_logs" {
    name = "es_search_slow_logs"
    retention_in_days = 14
}

resource "aws_elasticsearch_domain" "elasticsearch" {
    domain_name = "${var.domain_name}"
    elasticsearch_version = "${var.version}"

    cluster_config {
        instance_count = "${var.data_node_count}"
        instance_type = "${var.data_node_instance_type}"
        dedicated_master_enabled = true
        dedicated_master_count = "${var.master_node_count}"
        dedicated_master_type = "${var.master_node_instance_type}"
        zone_awareness_enabled = true
    }

    ebs_options {
        ebs_enabled = true
        volume_type = "gp2" // general purpose SSD
        volume_size = "${var.ebs_volume_size}"
    }

    log_publishing_options {
        enabled = true
        log_type = "INDEX_SLOW_LOGS"
        cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es_index_slow_logs.name}"
    }

    log_publishing_options {
        enabled = true
        log_type = "SEARCH_SLOW_LOGS"
        cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es_search_slow_logs.name}"
    }

    snapshot_options {
        automated_snapshot_start_hour = 1 // 1:00am
    }
}

resource "aws_elasticsearch_domain_policy" "main" {
    domain_name = "${aws_elasticsearch_domain.elasticsearch.domain_name}"

    lifecycle {
        ignore_changes = [
            "access_policies"
        ]
    }

    access_policies = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Action": [
                    "es:*"
                ],
                "Principal": {
                    "AWS": "*"
                },
                "Effect": "Allow",
                "Condition": {
                    "IpAddress": {"aws:SourceIp": ${jsonencode(var.allowed_ips)}}
                }
            }
        ]
    }
CONFIG
}
@atsushi-ishibashi
Copy link
Contributor

atsushi-ishibashi commented Dec 27, 2017

Please replace name with arn

log_publishing_options {
    enabled = true
    log_type = "INDEX_SLOW_LOGS"
    cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es_index_slow_logs.name}"
}

log_publishing_options {
    enabled = true
    log_type = "SEARCH_SLOW_LOGS"
    cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es_search_slow_logs.name}"
}

@nodesocket
Copy link
Author

nodesocket commented Dec 27, 2017

@atsushi-ishibashi thanks. Changed name to arn but still crashing with panic: runtime error: invalid memory address or nil pointer dereference.

unexpected EOF

@atsushi-ishibashi
Copy link
Contributor

@nodesocket Uhhh.. Could you give me more error logs?
Btw, did you associate resource policy with cwl like below?

resource "aws_cloudwatch_log_resource_policy" "example" {
  policy_name = "hogehoge"
  policy_document = <<CONFIG
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": [
        "logs:PutLogEvents",
        "logs:PutLogEventsBatch",
        "logs:CreateLogStream"
      ],
      "Resource": "${aws_cloudwatch_log_group.es_index_slow_logs.arn}"
    }
  ]
}
CONFIG
}

@nodesocket
Copy link
Author

@atsushi-ishibashi I only created the two cloudwatch log groups:

resource "aws_cloudwatch_log_group" "es_index_slow_logs" {
    name = "es_index_slow_logs"
    retention_in_days = 14
}

resource "aws_cloudwatch_log_group" "es_search_slow_logs" {
    name = "es_search_slow_logs"
    retention_in_days = 14
}

Must I also create a cloudwatch resource policy for each group as well?

@atsushi-ishibashi
Copy link
Contributor

atsushi-ishibashi commented Dec 27, 2017

@nodesocket Yeah, you must create a cloudwatch resource policy. To be exact, you have to assign
the policy to elasticsearch to put logs to these two cwl groups.

resource "aws_cloudwatch_log_resource_policy" "example" {
  policy_name = "hogehoge"
  policy_document = <<CONFIG
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "es.amazonaws.com"
      },
      "Action": [
        "logs:PutLogEvents",
        "logs:PutLogEventsBatch",
        "logs:CreateLogStream"
      ],
      "Resource": [
        "${aws_cloudwatch_log_group.es_index_slow_logs.arn}",
        "${aws_cloudwatch_log_group. es_search_slow_logs.arn}"
      ]
    }
  ]
}
CONFIG
}

@nodesocket
Copy link
Author

@atsushi-ishibashi thanks, updated and added a aws_cloudwatch_log_resource_policy, but still Terraform is crashing. Here is the full crash.log.

https://gist.github.com/nodesocket/84c8ad96fda3e146852652ae2f2bf152

@atsushi-ishibashi
Copy link
Contributor

atsushi-ishibashi commented Dec 27, 2017

The reason is Enabled: false but I couldn't find why it becomes unable...
Could you try not to specify enabled = true? The default value is true. Of course, it is bug if it works well 🙇

2017-12-27T11:40:25.591-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:     LogPublishingOptions: {
2017-12-27T11:40:25.591-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       INDEX_SLOW_LOGS: {
2017-12-27T11:40:25.591-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:         Enabled: false
2017-12-27T11:40:25.591-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       },
2017-12-27T11:40:25.591-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       SEARCH_SLOW_LOGS: {
2017-12-27T11:40:25.591-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:         Enabled: false
2017-12-27T11:40:25.591-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:       }
2017-12-27T11:40:25.591-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:     },

@nodesocket
Copy link
Author

nodesocket commented Dec 29, 2017

@atsushi-ishibashi really appreciate the help, but I removed enabled = true but unfortunately still seeing the Terraform crash.

Here is the updated relevant sections:

resource "aws_cloudwatch_log_group" "es_index_slow_logs" {
    name = "es_index_slow_logs"
    retention_in_days = 14
}

resource "aws_cloudwatch_log_group" "es_search_slow_logs" {
    name = "es_search_slow_logs"
    retention_in_days = 14
}

resource "aws_cloudwatch_log_resource_policy" "es_slow_logs" {
    policy_name = "es_slow_logs"
    policy_document = <<CONFIG
    {
        "Version": "2012-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {
                "Service": "es.amazonaws.com"
            },
            "Action": [
                "logs:PutLogEvents",
                "logs:PutLogEventsBatch",
                "logs:CreateLogStream"
            ],
            "Resource": [
                "${aws_cloudwatch_log_group.es_index_slow_logs.arn}",
                "${aws_cloudwatch_log_group.es_search_slow_logs.arn}"
            ]
        }]
    }
CONFIG
}

Relevant aws_elasticsearch_domain:

    log_publishing_options {
        log_type = "INDEX_SLOW_LOGS"
        cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es_index_slow_logs.arn}"
    }

    log_publishing_options {
        log_type = "SEARCH_SLOW_LOGS"
        cloudwatch_log_group_arn = "${aws_cloudwatch_log_group.es_search_slow_logs.arn}"
    }

@atsushi-ishibashi
Copy link
Contributor

Uhh.. I submitted the PR which avoid nil pointer and is already merged.
But the primal problem is that your terraform codes didn't configure log_publishing_options accoding to gist... 😕

@jen20 jen20 added bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. labels Dec 29, 2017
@jen20
Copy link
Contributor

jen20 commented Dec 29, 2017

This is fixed by #2787. Thanks for reporting, @nodesocket, and thanks for fixing, @atsushi-ishibashi!

@jen20 jen20 closed this as completed Dec 29, 2017
@atsushi-ishibashi
Copy link
Contributor

atsushi-ishibashi commented Dec 30, 2017

@jen20 Not yet. Actually we can avoid nil pointer, but we couldn't understand why log_publishing_options wasn't set. So there will be diff in state after read function.
Could you reopen this one and help us? 🙇

@jen20 jen20 reopened this Dec 30, 2017
@nodesocket
Copy link
Author

@atsushi-ishibashi and @jen20 so besides this nil pointer issue, is log_publishing_options going to work with this patch after? Thanks for all the help

@nodesocket
Copy link
Author

nodesocket commented Jan 2, 2018

@atsushi-ishibashi, @jen20 ok this makes zero sense. I commented out the following blocks:

# resource "aws_cloudwatch_log_group" "es_index_slow_logs" {
#     name = "es_index_slow_logs"
#     retention_in_days = 14
# }

# resource "aws_cloudwatch_log_group" "es_search_slow_logs" {
#     name = "es_search_slow_logs"
#     retention_in_days = 14
# }

# resource "aws_cloudwatch_log_resource_policy" "es_slow_logs" {
#     policy_name = "es_slow_logs"
#     policy_document = <<CONFIG
#     {
#         "Version": "2012-10-17",
#         "Statement": [{
#             "Effect": "Allow",
#             "Principal": {
#                 "Service": "es.amazonaws.com"
#             },
#             "Action": [
#                 "logs:PutLogEvents",
#                 "logs:PutLogEventsBatch",
#                 "logs:CreateLogStream"
#             ],
#             "Resource": [
#                 "${aws_cloudwatch_log_group.es_index_slow_logs.arn}",
#                 "${aws_cloudwatch_log_group.es_search_slow_logs.arn}"
#             ]
#         }]
#     }
# CONFIG
# }

And manually did rm -rf .terraform/modules and terraform get and then terraform plan but still seeing the crash. This makes zero sense. Could the remote state (using S3) be corrupt? Here is the crash information:

MacBook-Pro ➜  dev git:(master) ✗ terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

aws_elasticsearch_domain.elasticsearch: Refreshing state... (ID: arn:aws:es:us-west-2:391582443503:domain/duocircle-elasticsearch-dev)

Error: Error refreshing state: 1 error(s) occurred:

* module.elasticsearch.aws_elasticsearch_domain.elasticsearch: 1 error(s) occurred:

* module.elasticsearch.aws_elasticsearch_domain.elasticsearch: aws_elasticsearch_domain.elasticsearch: unexpected EOF


panic: runtime error: invalid memory address or nil pointer dereference
2018-01-02T15:19:00.533-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x26a808c]
2018-01-02T15:19:00.533-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4:
2018-01-02T15:19:00.533-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: goroutine 97 [running]:
2018-01-02T15:19:00.533-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/aws.resourceAwsElasticSearchDomainRead(0xc4209cd0a0, 0x2f81da0, 0xc420078900, 0x0, 0x4d57fc0)
2018-01-02T15:19:00.533-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/aws/resource_aws_elasticsearch_domain.go:500 +0x94c
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Resource).Refresh(0xc42067a4d0, 0xc420c1bf40, 0x2f81da0, 0xc420078900, 0xc4204ba130, 0xc420da1b01, 0x80000000018)
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/resource.go:355 +0x199
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema.(*Provider).Refresh(0xc4209cc2a0, 0xc420c1bef0, 0xc420c1bf40, 0x5aa7d90, 0x0, 0x18)
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/helper/schema/provider.go:308 +0x9a
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin.(*ResourceProviderServer).Refresh(0xc42000ba60, 0xc420d4dbe0, 0xc420d4de40, 0x0, 0x0)
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/opt/teamcity-agent/work/222ea50a1b4f75f4/src/github.com/terraform-providers/terraform-provider-aws/vendor/github.com/hashicorp/terraform/plugin/resource_provider.go:549 +0x4e
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: reflect.Value.call(0xc420370d80, 0xc420390e50, 0x13, 0x3343a66, 0x4, 0xc420cf7f20, 0x3, 0x3, 0x0, 0x0, ...)
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/usr/local/go/src/reflect/value.go:434 +0x906
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: reflect.Value.Call(0xc420370d80, 0xc420390e50, 0x13, 0xc42037bf20, 0x3, 0x3, 0x0, 0x0, 0x0)
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/usr/local/go/src/reflect/value.go:302 +0xa4
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: net/rpc.(*service).call(0xc420353e40, 0xc42008e910, 0xc4200175b0, 0xc4203ab100, 0xc42000bfe0, 0x2b37500, 0xc420d4dbe0, 0x16, 0x2b37540, 0xc420d4de40, ...)
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/usr/local/go/src/net/rpc/server.go:381 +0x142
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: created by net/rpc.(*Server).ServeCodec
2018-01-02T15:19:00.534-0500 [DEBUG] plugin.terraform-provider-aws_v1.6.0_x4: 	/usr/local/go/src/net/rpc/server.go:475 +0x36b
2018/01/02 15:19:00 [ERROR] root.elasticsearch: eval: *terraform.EvalRefresh, err: aws_elasticsearch_domain.elasticsearch: unexpected EOF
2018/01/02 15:19:00 [ERROR] root.elasticsearch: eval: *terraform.EvalSequence, err: aws_elasticsearch_domain.elasticsearch: unexpected EOF
2018/01/02 15:19:00 [TRACE] [walkRefresh] Exiting eval tree: module.elasticsearch.aws_elasticsearch_domain.elasticsearch
2018/01/02 15:19:00 [TRACE] dag/walk: upstream errored, not walking "module.elasticsearch.aws_elasticsearch_domain_policy.main"
2018/01/02 15:19:00 [TRACE] dag/walk: upstream errored, not walking "provider.aws (close)"
2018/01/02 15:19:00 [TRACE] dag/walk: upstream errored, not walking "root"
2018-01-02T15:19:00.538-0500 [DEBUG] plugin: plugin process exited: path=/Users/justin/Sites/elastic-byte-clients/duocircle/DuoCircleInfrastructure/terraform/environments/dev/.terraform/plugins/darwin_amd64/terraform-provider-aws_v1.6.0_x4
2018/01/02 15:19:00 [DEBUG] plugin: waiting for all plugin processes to complete...
2018-01-02T15:19:00.538-0500 [WARN ] plugin: error closing client during Kill: err="connection is shut down"



!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
A crash log has been placed at "crash.log" relative to your current
working directory. It would be immensely helpful if you could please
report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version. That
information is available on the first line of crash.log. You can also
get it by running 'terraform --version' on the command line.

[1]: https:/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

@radeksimko radeksimko added the service/elasticsearch Issues and PRs that pertain to the elasticsearch service. label Jan 12, 2018
@voltechs
Copy link

I'm attempting to upgrade our infrastructure from v0.9.11 to v0.11.2, getting the following. https://gist.github.com/voltechs/aa24f1fc400e7271981216a7f881c3f6

@radeksimko
Copy link
Member

@nodesocket the fix was released in 1.7.0 just a few days ago. Can you try upgrading via terraform init -upgrade? Thanks.

Regardless I'd be keen to see the debug log from terraform commands you ran (minus any secrets) because I was unable to reproduce this crash.

As we are unable to reproduce this I'm closing it for now, but happy to re-open once we have a repro case (i.e. when there's a chance of resolving it).

@voltechs This is a different bug unrelated to this issue, we can keep track of it in #3005 which I just opened.

Thanks all.

@bflad bflad added this to the v1.7.0 milestone Jan 16, 2018
@ghost
Copy link

ghost commented Apr 8, 2020

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

@ghost ghost locked and limited conversation to collaborators Apr 8, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Addresses a defect in current functionality. crash Results from or addresses a Terraform crash or kernel panic. service/elasticsearch Issues and PRs that pertain to the elasticsearch service.
Projects
None yet
Development

No branches or pull requests

6 participants