TFC local execution doesn't seem to respect the -lock-timeout option

[chapmanc]

Terraform Version

Terraform v0.12.29

 terraform -v
Terraform v0.12.29
+ provider.null v3.1.0
+ provider.time v0.7.0

Your version of Terraform is out of date! The latest version
is 0.14.7. You can update by downloading from https://www.terraform.io/downloads.html

Terraform v0.14.7

Terraform v0.14.7
+ provider registry.terraform.io/hashicorp/null v3.1.0
+ provider registry.terraform.io/hashicorp/time v0.7.0

Terraform Configuration Files

terraform {
  backend "remote" {
    organization = "chappie"

    workspaces {
      name = "testing"
    }
  }
}
resource "null_resource" "previous" {}

resource "time_sleep" "wait_100s" {
  depends_on = [null_resource.previous]

  create_duration = "100s"
}

# This resource will create (at least) 100 seconds after null_resource.previous
resource "null_resource" "next" {
  depends_on = [time_sleep.wait_100s]
}

Debug Output

This is difficult to show as it is a result of multiple runs.

Crash Output

N/A

Expected Behavior

When specifying -lock-timeout=120s terraform plans should hang until the specified timeout has completed and then fail with a lock error.

Error: Error locking state: Error acquiring the state lock: workspace already locked (lock ID: "chappie/testing")

Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.

Actual Behavior

When specifying -lock-timeout=120s terraform plans immediately fail with a lock error.

Steps to Reproduce

1. terraform login
2. terraform init
3. After running for i in {1..10}; do terraform plan -lock=true -lock-timeout=120s & done I would expect that it would fail after 120s due to the lock timeout with ~ 1-2 plans succeeding. Instead 1 plan begins and all the other plans fail due to:

Error: Error locking state: Error acquiring the state lock: workspace already locked (lock ID: "chappie/testing")

Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.

Additional Context

References

@alisdair

[alisdair]

Thanks for reporting this! I'm able to reproduce it on Terraform 0.14 and the current main branch.

Note for repro steps:

• Local operations on the workspace is essential
• Try starting one apply and then a plan while the apply is running

This doesn't seem to affect all locking backends (I tested Consul, which behaves as you'd expect). Trace logs on the subsequent plan operation show that a lock attempt is made and instantly fails:

2021-02-19T15:00:53.772-0500 [TRACE] backend/local: requesting state lock for workspace "default"
╷
│ Error: Error acquiring the state lock

For anyone picking this up: my best guess here is that the remote backend is either not receiving a clistate.Locker with the correct timeout, or it is failing to pass it to its local backend.

[alisdair]

My guess was wrong, the timeout value is making it to the appropriate places.

The problem is that the remote backend is not populating the Info field of the LockError when locking fails, which prevents lock retry.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.