runtime error: invalid memory address or nil pointer -- v1.1.0-rc1

[rymancl]

Terraform Version

(running on wsl if it matters)

Terraform v1.1.0-rc1
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.68.0

Terraform Configuration Files

I'm not able to post the code for this, but essentially it's a large project that references about 60 [local] modules. Inside those modules are various IAM resources (roles, policies, attachments, etc).

Debug Output

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.

[1]: https:/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

runtime error: invalid memory address or nil pointer dereferencemodule.upstreametl.aws_iam_policy.upstreametl_generic_pipeline_policy: Refreshing state... [id=arn:aws:iam::ACCOUNTID:policy/UpstreamETL-Generic-Pipeline-Policy]

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.

[1]: https:/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

runtime error: invalid memory address or nil pointer dereference
!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.

[1]: https:/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

runtime error: invalid memory address or nil pointer dereferencemodule.prodfit.aws_iam_role_policy_attachment.prodfit_awsreadonly_policy_attachment: Refreshing state... [id=PRODFit-Generic-Pipeline-Role-20210903114328825000000012]
module.prodfit.aws_iam_policy.prodfit_generic_pipeline_policy: Refreshing state... [id=arn:aws:iam::ACCOUNTID:policy/PRODFit-Generic-Pipeline-Policy]

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

Terraform crashed! This is always indicative of a bug within Terraform.
Please report the crash with Terraform[1] so that we can fix this.

When reporting bugs, please include your terraform version, the stack trace
shown below, and any additional information which may help replicate the issue.

[1]: https:/hashicorp/terraform/issues

!!!!!!!!!!!!!!!!!!!!!!!!!!! TERRAFORM CRASH !!!!!!!!!!!!!!!!!!!!!!!!!!!!

runtime error: invalid memory address or nil pointer dereferencemodule.enrca.aws_iam_role_policy.enrca_iam_admin_pipeline_role_policy: Refreshing state... [id=ENRCA-IAM-Admin-Pipeline-Role:ENRCA-IAM-Admin-Pipeline-Role-Policy]
goroutine 3326 [running]:
runtime/debug.Stack()
        /usr/local/go/src/runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
        /usr/local/go/src/runtime/debug/stack.go:16 +0x19
github.com/hashicorp/terraform/internal/logging.PanicHandler()
        /home/circleci/project/project/internal/logging/panic.go:44 +0xb5
panic({0x221d240, 0x3ed5b20})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/hashicorp/terraform/internal/instances.(*expanderModule).onlyResourceInstances(0xc004937270, {{}, 0x4d, {0xc000988240, 0x1e}, {0xc0007483c0, 0x21}}, {0xc002c54900, 0x90, 0x4})
        /home/circleci/project/project/internal/instances/expander.go:392 +0x95
github.com/hashicorp/terraform/internal/instances.(*expanderModule).resourceInstances(0x9e, {0xc000206c80, 0x0, 0xc000590000}, {{}, 0x4d, {0xc000988240, 0x1e}, {0xc0007483c0, 0x21}}, ...)
        /home/circleci/project/project/internal/instances/expander.go:387 +0x2e5
github.com/hashicorp/terraform/internal/instances.(*expanderModule).resourceInstances(0xc003770078, {0xc000206c80, 0x1, 0x1}, {{}, 0x4d, {0xc000988240, 0x1e}, {0xc0007483c0, 0x21}}, ...)
        /home/circleci/project/project/internal/instances/expander.go:385 +0x293
github.com/hashicorp/terraform/internal/instances.(*Expander).ExpandResource(0xc002ba0080, {{}, {0xc000206c80, 0x1, 0x1}, {{}, 0x4d, {0xc000988240, 0x1e}, {0xc0007483c0, ...}}})
        /home/circleci/project/project/internal/instances/expander.go:157 +0x16f
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstanceOrphan).deleteActionReason(0xc002f66220, {0x2aecc00, 0xc0013a1260})
        /home/circleci/project/project/internal/terraform/node_resource_plan_orphan.go:204 +0x23b
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstanceOrphan).managedResourceExecute(0xc002f66220, {0x2aecc00, 0xc0013a1260})
        /home/circleci/project/project/internal/terraform/node_resource_plan_orphan.go:140 +0x58c
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstanceOrphan).Execute(0x0, {0x2aecc00, 0xc0013a1260}, 0x30)
        /home/circleci/project/project/internal/terraform/node_resource_plan_orphan.go:49 +0x90
github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc0012976c0, {0x2aecc00, 0xc0013a1260}, {0x7f1883d70fb0, 0xc002f66220})
        /home/circleci/project/project/internal/terraform/graph_walk_context.go:133 +0xc2
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x2553ec0, 0xc002f66220})
        /home/circleci/project/project/internal/terraform/graph.go:74 +0x2f0
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc000cf99e0, {0x2553ec0, 0xc002f66220}, 0xc00290ff80)
        /home/circleci/project/project/internal/dag/walk.go:381 +0x2f1
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update
        /home/circleci/project/project/internal/dag/walk.go:304 +0xf85

Expected Behavior

Plan should succeed

Actual Behavior

Plan crashes

Steps to Reproduce

1. tfenv use 1.0.11
2. terraform init -backend-config=backend.txt
3. terraform workspace select nonprod
4. terraform plan

success

1. tfenv use 1.1.0-rc1
2. terraform init -backend-config=backend.txt
3. terraform workspace select nonprod
4. terraform plan

crash

Additional Context

This is my first time testing out a non-main release, apologies if I'm just missing something obvious. I'm happy to provide any additional info/context if I'm able to.

References

[rymancl]

For whatever reason, if I add in a moved {} block for a module (one unrelated to those mentioned in the crash log), the plan succeeds. I have no clue why. I have not ran an apply with any moved {} blocks.

[rymancl]

Debugging further, this seems kinda like an issue with using for_each on a module declaration [to control if it is deployed on the current workspace].

If I leave my module as is...

module "team1" {
  source = "./team1"
  # arguments omitted
}

plan works fine on v1.1.0-rc1.

If I add a for_each, I get the crash...

module "team1" {
  source = "./team1"
  for_each = contains(["nonprod"], terraform.workspace) ? toset(["true"]) : toset([])
  # arguments omitted
}

This usage of for_each does not crash on v1.0.11. (The plan wants to destroy & re-create, but that is what I expect).

Also, this is with no moved {} blocks in place. If I add

moved {
  from = module.team1
  to   = module.team1["true"]
}

there is no crash as I'd expect.

Is v1.1.0-rc1 forcing the usage of moved {} in situations like this?

[apparentlymart]

Hi @rymancl! Thanks for trying the release candidate and reporting this crash.

A panic like this is never correct Terraform behavior, so indeed this is not a symptom of us trying to force any particular usage of moved blocks. (If it were then it would be a user-oriented error message rather than a panic, but it would also not be within the spirit of our Terraform v1.0 Compatibility Promises.)

We've spent a little time investigating this so far and haven't yet achieved a reproduction, though your latest comment arrived after we'd already been working for a while and so it will hopefully give us some more information to work with on a second attempt.

Unfortunately we're getting very close to the v1.1.0 final release now and so out of pragmatism we're going to leave this one unsolved for v1.1.0 and instead hope to have it solved in time for v1.1.1. While we never like to leave known bugs in final releases, it seems like this is a problem encountered during the plan step rather than the apply step, and so it fails prior to making any changes to remote objects and thus only blocks upgrading to Terraform v1.1, as opposed to causing difficult-to-resolve problems in production.

With that said then, we will prioritize debugging this as soon as possible, but we'll have to ask for you and anyone else who encounters a similar problem to remain on the latest Terraform v1.0.x release until we've been able to resolve this bug.

Thanks again for reporting this! We'll share more updates when we have them.

[alisdair]

Thanks for reporting this! I'm glad you were able to find the moved workaround for this situation. It looks like there's a bug in the handling of implied change from no-key to zero-key due to adding a for_each or count to a module which previously didn't have either in place. See #29605 for the PR which updated this behaviour as part of the moved work.

I have a minimal reproduction, with the following config:

main.tf:

module "foo" {
  source = "./modules/foo"
  # count = 1
}

modules/foo/main.tf:

resource "null_resource" "default" {
  count = 1
}

Steps to reproduce:

1. terraform init
2. terraform apply -auto-approve
3. Uncomment the count argument in the root module call
4. terraform plan: panics

The use of count in the module's resource seems to be required, although I'm not yet sure why. Still digging in!

[rymancl]

Thanks for the info @apparentlymart & @alisdair !

@alisdair - Sadly my workaround doesn't work 100%. If I'm on a workspace where my module doesn't get deployed , it'll still panic.

terraform workspace select nonprod

module "team1" {
  source = "./team1"
  for_each = contains(["production"], terraform.workspace) ? toset(["true"]) : toset([])
  # arguments omitted
}

crash

Unfortunately, this setup of applying modules conditionally based on workspace, and not having to destroy active resources, was my whole reason for testing the moved {} blocks, so I guess I'll have to wait until this is fixed.
Correct me if I'm wrong, but I don't think there is a way do dynamic/calculated moves. Hypothetically:

moved {
  from = module.team1
  to   = contains(["production"], terraform.workspace) ? module.team1["true"] : module.team1
}

╷
│ Error: Invalid expression
│
│ on _moves.tf line 3, in moved:
│ 3: to = contains(["production"], terraform.workspace) ? module.team1["true"] : module.team1
│
│ A single static variable reference is required: only attribute access and indexing with constant keys. No calculations, function calls, template expressions, etc are allowed here.

EDIT: I think the dynamic moves may be a little deeper of a topic. Assuming everything is fixed, what should actually be the result of

moved {
  from = module.team1
  to   = module.team1["true"]
}

when I try to plan on the nonprod workspace using the module declaration above?

[nathanielram]

Also having this issue. My code works fine on 1.0.11. We do have a few counts on some of our modules if that helps in any way. I can investigate further when I have some free time.

[xfournet]

While we never like to leave known bugs in final releases, it seems like this is a problem encountered during the plan step rather than the apply step, and so it fails prior to making any changes to remote objects and thus only blocks upgrading to Terraform v1.1, as opposed to causing difficult-to-resolve problems in production.

In our situation it leads to a difficult problem to be solved. The upgrade to Terraform 1.1.0 went well the first times, but since a change in our parameter make the plan to fail with the same error than this issue. I'm not sure I can downgrade Terraform since my state is already written by 1.1.0

I have a minimal repro case

main.tf

variable "param" {
  type = string
}

module "m1" {
  source = "./m1/"
  for_each = toset([var.param])
}

m1/main.tf

resource "null_resource" "provision" {
  for_each = toset(["test"])
}

Then

terraform init
terraform apply -auto-approve '-var=param=test'

terraform plan '-var=param=test'
---> works

terraform plan '-var=param=test2'
---> crash!

[apparentlymart]

Hi @xfournet,

The state snapshot format has not changed between Terraform v1.0 and Terraform v1.1, so in principle you should still be able to downgrade back to v1.0 at this point. If you try that and find that it doesn't work, please let me know and I will investigate a different workaround.

[xfournet]

Thanks @apparentlymart i was able to downgrade without problem

[crw]

Closed #30139 as a dupe of this issue.

[zswanson]

Chiming in that I'm seeing a similar crash on v1.1.0 after adding a for_each to an existing module in my root composition. I had to work around this by destroying the module first, adding the for_each and then applying. The crash log in my case didn't reference a particular resource though.

runtime error: invalid memory address or nil pointer dereferencegoroutine 5219 [running]:
runtime/debug.Stack()
	runtime/debug/stack.go:24 +0x65
runtime/debug.PrintStack()
	runtime/debug/stack.go:16 +0x19
github.com/hashicorp/terraform/internal/logging.PanicHandler()
	github.com/hashicorp/terraform/internal/logging/panic.go:44 +0xb5
panic({0x2e20cc0, 0x4ad1070})
	runtime/panic.go:1038 +0x215
github.com/hashicorp/terraform/internal/instances.(*expanderModule).onlyResourceInstances(0xc003d25270, {{}, 0x4d, {0xc000a21110, 0xc}, {0xc000a21120, 0x7}}, {0xc0012e5880, 0x18, 0x4})
	github.com/hashicorp/terraform/internal/instances/expander.go:392 +0x95
github.com/hashicorp/terraform/internal/instances.(*expanderModule).resourceInstances(0x26, {0xc000722800, 0x0, 0xc002e3c000}, {{}, 0x4d, {0xc000a21110, 0xc}, {0xc000a21120, 0x7}}, ...)
	github.com/hashicorp/terraform/internal/instances/expander.go:387 +0x2e5
github.com/hashicorp/terraform/internal/instances.(*expanderModule).resourceInstances(0xc002324108, {0xc000722800, 0x1, 0x1}, {{}, 0x4d, {0xc000a21110, 0xc}, {0xc000a21120, 0x7}}, ...)
	github.com/hashicorp/terraform/internal/instances/expander.go:385 +0x293
github.com/hashicorp/terraform/internal/instances.(*Expander).ExpandResource(0xc000b0aa60, {{}, {0xc000722800, 0x1, 0x1}, {{}, 0x4d, {0xc000a21110, 0xc}, {0xc000a21120, ...}}})
	github.com/hashicorp/terraform/internal/instances/expander.go:157 +0x16f
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstanceOrphan).deleteActionReason(0xc00199f860, {0x36f1ce0, 0xc002f15500})
	github.com/hashicorp/terraform/internal/terraform/node_resource_plan_orphan.go:182 +0x445
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstanceOrphan).managedResourceExecute(0xc00199f860, {0x36f1ce0, 0xc002f15500})
	github.com/hashicorp/terraform/internal/terraform/node_resource_plan_orphan.go:140 +0x58c
github.com/hashicorp/terraform/internal/terraform.(*NodePlannableResourceInstanceOrphan).Execute(0x0, {0x36f1ce0, 0xc002f15500}, 0x60)
	github.com/hashicorp/terraform/internal/terraform/node_resource_plan_orphan.go:49 +0x90
github.com/hashicorp/terraform/internal/terraform.(*ContextGraphWalker).Execute(0xc0013dfea0, {0x36f1ce0, 0xc002f15500}, {0x75a8360, 0xc00199f860})
	github.com/hashicorp/terraform/internal/terraform/graph_walk_context.go:133 +0xc2
github.com/hashicorp/terraform/internal/terraform.(*Graph).walk.func1({0x3158180, 0xc00199f860})
	github.com/hashicorp/terraform/internal/terraform/graph.go:74 +0x2f0
github.com/hashicorp/terraform/internal/dag.(*Walker).walkVertex(0xc000c75860, {0x3158180, 0xc00199f860}, 0xc004c11900)
	github.com/hashicorp/terraform/internal/dag/walk.go:381 +0x2f1
created by github.com/hashicorp/terraform/internal/dag.(*Walker).Update
	github.com/hashicorp/terraform/internal/dag/walk.go:304 +0xf85

[zswanson]

Similar crash if I modify an existing set used in a for_each such as adding or removing an item.
@alisdair - does your fix address this use-case too?

[alisdair]

@zswanson Not as far as I can tell. Please submit a new issue with a reproduction case and a stack trace and we can investigate. Thanks!

[Xerkus]

Workaround:
Try to perform targeted destroy operation for to-be-removed modules before applying change that would end up with removal of a module instance defined with for_each :

terraform destroy -target='module.module_with_for_each["plan_wants_to_remove"]'

Now that plan no longer have to remove the module the terraform plan will work just fine.

[RobertoUa]

I had the same crash when I tried to do a plan with -target. And without -target it worked fine.
The project is pretty big, so I don't know how to pinpoint the source of the issue.

[alisdair]

We've merged the fix for this issue, which will be included in the upcoming 1.1.1 release. Thanks all for your reports!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.