backend: Validate remote backend Terraform version

[alisdair]

When using the enhanced remote backend, a subset of all Terraform operations are supported. Of these, only plan and apply can be executed on the remote infrastructure (e.g. Terraform Cloud). Other operations run locally and use the remote backend for state storage.

This causes problems when the local version of Terraform does not match the configured version from the remote workspace. If the two versions are incompatible, an import or state mv operation can cause the remote workspace to be unusable until a manual fix is applied.

To prevent this from happening accidentally, this commit introduces a check that the local Terraform version and the configured remote workspace Terraform version are compatible. This check is skipped for commands which do not write state, and can also be disabled by the use of a new command-line flag, -ignore-remote-version.

Terraform version compatibility is defined as:

• For all releases before 0.14.0, local must exactly equal remote, as two different versions cannot share state;
• 0.14.0 to 1.0.x are compatible, as we will not change the state version number until at least Terraform 1.1.0;
• Versions after 1.1.0 must have the same major and minor versions, as we will not change the state version number in a patch release.

If the two versions are incompatible, a diagnostic is displayed, advising that the error can be suppressed with -ignore-remote-version. When this flag is used, the diagnostic is still displayed, but as a warning instead of an error.

Commands which will not write state can assert this fact by calling the helper meta.ignoreRemoteBackendVersionConflict, which will disable the checks. Those which can write state should instead call the helper meta.remoteBackendVersionCheck, which will return diagnostics for display.

In addition to these explicit paths for managing the version check, we have an implicit check in the remote backend's state manager initialization method. Both of the above helpers will disable this check. This fallback is in place to ensure that future code paths which access state cannot accidentally skip the remote version check.

Release plans

This change is intended to ship with 0.15.0, and we also intend to backport to earlier release series. The current plan is 0.14, 0.13, and 0.12.

Screenshot

Here's a session which demonstrates both the error and warning forms of the diagnostics:

[codecov]

Codecov Report

Merging #26947 (c5c1f31) into master (b8d448d) will increase coverage by 0.05%.
The diff coverage is 58.57%.

Impacted Files	Coverage Δ
command/state_mv.go	40.95% <4.34%> (-0.40%)	⬇️
command/state_rm.go	46.93% <8.33%> (-1.49%)	⬇️
command/untaint.go	36.84% <17.39%> (+0.03%)	⬆️
command/meta_backend.go	56.82% <22.22%> (-0.59%)	⬇️
command/taint.go	38.23% <27.27%> (-0.04%)	⬇️
command/import.go	53.14% <44.44%> (-0.40%)	⬇️
command/state_meta.go	47.70% <50.00%> (+0.08%)	⬆️
command/state_push.go	34.93% <71.42%> (+0.76%)	⬆️
backend/remote/backend_mock.go	62.99% <72.72%> (+0.41%)	⬆️
backend/remote/backend.go	62.59% <100.00%> (+4.54%)	⬆️
... and 21 more

[apparentlymart]

This seems like a good compromise! Thanks for working on this.

It does seem a little unfortunate that we had to add new CLI argument surface area to so many commands only for this one backend, but I don't have a better idea and it is at least consistent with our sense that the local vs. remote distinction will probably eventually become a first-class thing separate from what we otherwise think of as "backends" (state storage options for local operations).

One cosmetic feedback inline, but it's subjective so feel free to ignore it if you disagree with it.

[apparentlymart]

Seems like GitHub helpfully lost my inline feedback. Thanks. 🙄

The gist of it was a couple ideas for the error messages:

• Mention the remote organization name as well as the workspace name, so it's clear which workspace we're talking about and visually distinct from local workspace names? (organization/workspace instead of just workspace)
• Explicitly use the words "remote workspace" in the message so we're extra clear that we're talking about remote workspaces rather than local workspaces?

[alisdair]

Thanks for the suggestions! I revised the diagnostic and I think it's clearer. Example:

Error: Terraform version mismatch

The local Terraform version (0.14.0-dev) does not match the configured version
for remote workspace alisdair-v2/sfv (0.13.5). If you're sure you want to
upgrade the state, you can force Terraform to continue using the
-ignore-remote-version flag. This may result in an unusable workspace.

[e-moshaya]

This change broke our pipelines.. We use terraform_remote_state with backend set to remote and Terraform Cloud as the backend for all our workspaces. However, in all the workspaces we have Execution Mode set to local only and not remote. However, With the introduction of this change, we're required to changes the Execution Mode to remote and update the terraform version inline with the version required in the current workspace. Can't the terraform_remote_state verify the version of the remote state via the terraform_version in the state file instead?

[alisdair]

@e-moshaya Sorry about that! The behaviour you're seeing was unintentional, and the intended behaviour is as you proposed. See #27197 for a pending fix, which I hope to release today.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.