feat: terraform enterprise support with migrations (#3)

BREAKING CHANGE: almost a rewrite so we expect some things to possibly be broken

- Added commands for querying s3 backend
- Added Terraform Enterprise support
- Added migrations from s3 backend to terraform enterprise
- Added commands for interacting with Terraform Enterprise
- Added auto-creation of Terraform Enterprise Projects for workspaces
- Improved documentation to cover changes
- Added integration tests
- Replaced environment variable lookup vars with environment variable referencing from within configuration files

.github/workflows/ci.yml

@@ -5,15 +5,6 @@ on:
  - '**'
 
 jobs:
- lint:
- name: Lint
- runs-on: ubuntu-latest
- steps:
- - uses: actions/checkout@v2
- - name: golangci-lint
- run: |
- curl -sSfL https://raw.githubusercontent.com/golangci/golangci-lint/master/install.sh | sh -s
- ./bin/golangci-lint run -v
  test:
  runs-on: ubuntu-latest
  steps:

.gitignore

@@ -7,5 +7,7 @@ dist/
 .terraform/
 *.tfstate
 terraform.backend.tf.json
+terraform.tf.json
 terraform.tfstate
-terraform.tfvars.json
+terraform.tfstate.backup
+terraform.tfvars.json

.gitpod.yml

@@ -0,0 +1,14 @@
+
+image:
+ file: .gitpod/Dockerfile
+
+tasks:
+ - command: bash .gitpod/init.sh && go mod download && . ~/.bashrc
+
+vscode:
+ extensions:
+ - EditorConfig.EditorConfig
+ - esbenp.prettier-vscode
+ - golang.go
+ - ms-vscode.makefile-tools
+ - redhat.vscode-yaml

.gitpod/Dockerfile

@@ -0,0 +1,26 @@
+FROM gitpod/workspace-go
+
+ARG AWS_DEST="awscliv2.zip"
+
+USER root
+
+RUN apt-get install -yq \
+ curl \
+ unzip \
+ less
+
+RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -L -o "${AWS_DEST}" \
+ && unzip ${AWS_DEST} \
+ && ./aws/install \
+ && rm -rf ${AWS_DEST} ./amazonaws
+
+USER gitpod
+
+# We want a newer version of go that in the image, so we're going to overwrite it
+RUN curl "https://go.dev/dl/go1.20.4.linux-amd64.tar.gz" -L -o "go1.20.4.linux-amd64.tar.gz" \
+ && rm -rf ~/go \
+ && tar -C ~/ -xzf go1.20.4.linux-amd64.tar.gz
+
+RUN git clone --depth=1 https:/tfutils/tfenv.git ~/.tfenv \
+ && echo 'export PATH="$HOME/.tfenv/bin:$PATH"' >> ~/.bashrc.d/300-tfenv \
+ && echo 'export AWS_PROFILE=main' >> ~/.bashrc.d/300-aws

.gitpod/init.sh

@@ -0,0 +1,35 @@
+#!/usr/bin/env sh
+
+# AWS
+
+[[ -d /home/gitpod/.aws ]] || mkdir /home/gitpod/.aws
+cat <<- AWSFILE > /home/gitpod/.aws/config
+[default]
+cli_pager=
+output=json
+
+# profile pinned in environment (~/.bashrc.d/300-aws)
+[profile main]
+sso_start_url=${AWS_SSO_START_URL}
+sso_region=${AWS_SSO_REGION}
+sso_account_id=${AWS_SSO_ACCOUNT_ID}
+sso_role_name=${AWS_SSO_ROLE_NAME}
+region=${AWS_REGION}
+AWSFILE
+
+echo "Generated: /home/gitpod/.aws/config"
+
+# TERRAFORM
+
+[[ -d /home/gitpod/.terraform.d ]] || mkdir /home/gitpod/.terraform.d
+cat <<- TFRCFILE > ~/.terraform.d/credentials.tfrc.json
+{
+ "credentials": {
+ "app.terraform.io": {
+ "token": ${TF_TOKEN_app_terraform_io}
+ }
+ }
+}
+TFRCFILE
+
+echo "Generated: /home/gitpod/.terraform.d/credentials.tfrc.json"

.vscode/launch.json

@@ -0,0 +1,24 @@
+{
+ // Use IntelliSense to learn about possible attributes.
+ // Hover to view descriptions of existing attributes.
+ // For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
+ "version": "0.2.0",
+ "configurations": [
+ {
+ "name": "List workspaces",
+ "type": "go",
+ "request": "launch",
+ "mode": "auto",
+ "program": "cmd/twrapper/main.go",
+ "args": ["cloud", "workspace", "list"]
+ },
+ {
+ "name": "Delete workspaces",
+ "type": "go",
+ "request": "launch",
+ "mode": "auto",
+ "program": "cmd/twrapper/main.go",
+ "args": ["cloud", "workspace", "delete", "--approve", "--workspace", "twrapper-example-wprkspace"]
+ },
+ ]
+}

.vscode/settings.json

@@ -0,0 +1,3 @@
+{
+ "ansible.python.interpreterPath": "/usr/bin/python3"
+}

CONTRIBUTING.md

@@ -0,0 +1,45 @@
+# Welcome Twrapper contributing guide
+
+Thank you for investing your time in contributing to our project!
+
+## New contributor guide
+
+This guide will get you setup to not only contribute code to twrapper, but also integration test it.
+
+- [Testing](docs/testing)
+
+## Getting started
+
+TODO
+
+### Issues
+
+#### Create a new issue
+
+TODO
+
+#### Solve an issue
+
+TODO
+
+### Make Changes
+
+#### Make changes in the app
+
+TODO
+
+#### Make changes to the docs
+
+TODO
+
+### Commit your update
+
+TODO
+
+### Pull Request
+
+TODO
+
+### Your PR is merged
+
+:boom: Thank you!

Dockerfile

@@ -1,11 +1,11 @@
-FROM golang:1.16.1-alpine AS build
+FROM golang:1.20.0-alpine AS build
 ARG DEST_DIR="/go/src/twrapper"
 COPY . $DEST_DIR
 WORKDIR $DEST_DIR
 RUN apk update && apk add git openssh
 RUN go build -o /usr/bin/twrapper ./cmd/twrapper
 
-FROM golang:1.16.1-alpine AS tfenv
+FROM golang:1.20.0-alpine AS tfenv
 ARG DEST_DIR="/go/src/tfenv"
 ARG TFENV_VERSION="v2.2.0"
 RUN apk update && apk add git openssh && \

Makefile

@@ -1,13 +1,7 @@
 SRC=./cmd/twrapper
 DIST=./dist
 
-test : mods
- go test -v ./...
-
-test_quiet : mods
- go test ./...
-
-dist: clean test
+dist: clean check
  GOOS=darwin GOARCH=amd64 go build -o ${DIST}/twrapper-darwin-amd64 ${SRC}
  GOOS=darwin GOARCH=arm64 go build -o ${DIST}/twrapper-darwin-arm64 ${SRC}
  GOOS=linux GOARCH=386 go build -o ${DIST}/twrapper-linux-386 ${SRC}
@@ -28,6 +22,16 @@ mods:
  go mod download
 
 check : test
- staticcheck ./cmd/twrapper/
- staticcheck ./pkg/aws/
- staticcheck ./pkg/terraform/
+ staticcheck ./...
+
+test : mods
+ go test -v ./...
+
+test_quiet : mods
+ go test ./...
+
+integration_tests:
+ go test --tags=integration_test ./...
+
+vet:
+ go vet ./...

README.md

@@ -1,14 +1,18 @@
 # Twrapper Terraform Wrapper
 
+## NOTICE
+
+We will be deprecating, this project very soon. Once we've migrated to Terraform Cloud, we'll no longer maintain this
+project.
+
 ## About
 
 This is not an alternative to or a replacement for Terragrunt!
 
-At adaptavist we have numerous private GIT repos with Terraform modules ready to be remotely 
-triggered enabling some level of self-service. However, we need to wrap terraform to automate 
-backend configuration and ensuring a suitable role can be assumed. We already had something
-written in python but we want to replace it with something lighter weight and pre-bundled in a 
-container.
+At adaptavist we have numerous private GIT repos with Terraform modules ready to be remotely triggered enabling some
+level of self-service. However, we need to wrap terraform to automate backend configuration and ensuring a suitable role
+can be assumed. We already had something written in python but we want to replace it with something lighter weight and
+pre-bundled in a container.
 
 Our intentions with this wrapper
 
@@ -17,98 +21,9 @@ Our intentions with this wrapper
 - Restrict a module to a role
 - Allow searching through a list of roles to find a suitable one to run the module
 - Ensure Account IDs are specified for the roles we use when working accross accounts
-- Force a unique backend key by using UUIDv4
 - Enable running most Terraform commands
-
-## Configuration
-
-```yaml
-# module/twrapper.yaml
-# error if these vars are not set or are empty
-required_vars:
- - TF_VAR_example_var_name
-# Tells Twrapper we are working on AWS
-aws:
- # If you want to provide a role to terraform, you must tell it the variable used to apply it.
- role_tf_var: aws_assume_role_arn
- # role_arn pins your module to a specific role
- role_arn: arn:aws:iam::00000000:role/AdminisratorAccess
- # try these roles if role name is not set, only used if AccountIDVar is set
- try_role_names: [ OrganizationAccountAccessRole ]
- # when using try_role_names, ACCOUNT_ID must be present in the env
- account_id_var: ACCOUNT_ID
-# Tell Twrapper what environment variable should be used for the key_id
-backend_key_id_var: KEY_ID # VAR used to get the key id
-# Also support local for development, may add support for more in time
-backend_type: s3
-# optional - defaults to {key_id}.tfstate
-backend_key_template: "module-{key_id}.tfstate"
-backend_config:
- bucket: "backend-bucket-name"
- region: "us-east-1"
- dynamodb_table: "backend-table-name"
- encrypt: true
-```
-
-### Example Configs
-
-#### Required Terraform Variables
-
-You may want Terraform to validate your variables but you may be using a third-party module. So Twrapper
-provides a real basic feature for checking variables are set and not empty, but it will not validate
-the value beyond that.
-
-```yaml
-required_vars:
- - TF_VAR_my_var_1
- - TF_VAR_my_var_2
-```
-
-#### Backend configuration
-
-```yaml
-# What type of backend
-backend_type: s3
-# The environment variable which has the backend key
-backend_key_id_var: KEY_ID
-# override the key/path format for your backend
-backend_key_template: "module-{key_id}.tfstate"
-# Main backend configure, this is generally static
-backend_config:
- bucket: "backend-bucket-name"
- region: "us-east-1"
- dynamodb_table: "backend-table-name"
- encrypt: true
-```
-
-#### AWS Configuration
-
-Chances are you're deploying to some sort of IaaS platform. and you need an appropriate session. We
-aim to resolve this by trying to resolve which roles can be used. We understand that as busineses
-grow the role names may not be consistent and therefore the ability to try others is helpful.
-
-```yaml
-# Tell the AWS package to try and assume one of the two roles and assign the ARN of the working
-# role to terraform as "assume_role_arn"
-aws:
- role_tf_var: assume_role_arn
- account_id_var: AWS_ACCOUNT_ID
- try_role_names:
- - MyAdminRole
- - MyOrganizationRole
-
-# Fix your module to a single role on a single account, and assign to the "assume_role_arn" variable
-aws:
- role_tf_var: assume_role_arn
- role_arn: arn:aws:iam::00000000:role/RoleName
-```
-
+- Migrating to Terraform Cloud
 
 ## Usage
 
-```bash
-twrapper [terraform args...]
-twrapper plan # run a terraform plan
-twrapper apply # runs an apply but adds -auto-approve
-twrapper destroy # runs a destroy but add -force
-```
+Please check out the [documentation](docs/) for usage.