Terraform logo

I have been using Terraform for a couple of years. Today, I couldn’t imagine managing cloud infrastructure without it. This article gives technical tips that can simplify daily use.

Use terraform instead of vendor specific language

Ever written or tried to decipher CloudFormation templates ? Experience is often painful.

I find Terraform syntax so much easier to the eyes, and to the mind. Cherry on the cake, learning Terraform once will help you with any cloud provider or even on prem infrastructure.

Plus documentation and community support are excellent.

Always use remote state and locking

Infrastructure is made of shared resources. These resources need to be maintained or referenced by several team members or several computers or repositories. By default the state of your infrastructure is stored in a local state file. This is not practical for teams, nor secure (no backup and risk of collision if 2 process deploy at the same time).

For any collaborative work to occur, store (and backup) the state of your infrastructure in a shared location.

Terraform can be configured to fetch and save state to a specific backend instead of (instead of using a local .tfstate file). The most common scenario is to share the state (and provide a locking mechanism) through the use of an S3 bucket and a DynamoDB table.

Setting up a shared state is chicken and egg problem. You need to create the bucket and DB (with Terraform), but also reference it in your terraform config.

It means a 2 steps action:

  1. you first create the backend using a terraform local state file
  2. then you configure terraform to use this backend to store subsequent state updates.

1.Create a shared bucket + dynamo db table

The bucket will store your .tfstate file and dynamodb will act as a lock.

resource "aws_s3_bucket" "terraform_state" {
  bucket = "my-custom-tf-bucket135468745684124874212"
  # Enable versioning so we can see the full revision history of our
  # state files
  versioning {
    enabled = true
  }
  # Enable server-side encryption by default
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

resource "aws_dynamodb_table" "terraform_locks" {
  name         = "my-tf-locks-bucket135468745684124874212"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "LockID"
  attribute {
    name = "LockID"
    type = "S"
  }
}

2. Configure terraform to point to this bucket

Bucket name needs to be unique to your account, and key has to be unique to your environment (The key is the location of your .tfstate file inside the bucket, like a path).

/* add a backend definition at the beginning of your tf file */
terraform {
  backend "s3" {
    # Replace this with your bucket name!
    bucket         = "my-custom-tf-bucket135468745684124874212"
    key            = "my-prod-envt/s3/terraform.tfstate"
    region         = "eu-west-1"
    # Replace this with your DynamoDB table name!
    dynamodb_table = "my-tf-locks-bucket135468745684124874212"
    encrypt        = true
    profile = "default"
  }
}

A recommended way of creating the backend is to use an external module like https://github.com/scottwinkler/terraform-aws-s3backend which creates additional roles and provide relevant outputs to ease later integration with other accounts. See https://gitlab.com/demeringo/blueprint-terraform/ for a example of use.

Use latest 0.12 syntax (and formatter and syntax check)

The updated terraform syntax remove the need for quotes or special characters for the most for trivial interpolation. Embrace it, that makes code easier to read.

If you maintain old scripts, use terraform 0.12upgrade command and terraform will rewrite and reformat your script to the latest syntax.

Terraform provides basic syntax check with the the build in terraform validate command. You should format your prior to any commit with terraform fmt -recursive. It will ease the life of the team on subsequent merges or diff.

Boost your editor

Use an editor that provides syntax highlighting, autocompletion, formatting and linting. As I primarily use VS code, I stick to the following extensions:

Static analysis of TF code

You can use static analysis tools to ensure that your terraform code is valid and does not creates security holes even before running it.

Run Terraform in CI/CD

This is a subject in itself, it will need its own article.

See https://gitlab.com/demeringo/blueprint-terraform/ for a working example with Gitlab ci.

Principle:

  • Execute terraform (or other tools) from docker containers
  • Use variables for AWS keys and backend
  • Automate static checks
  • Use a manual approval step to deploy to production.

Organize your code

As soon as our terraform code grows behind simple use case, we (and other maintainers in the team) benefit from having a structured approach. It simplifies managing multiple environments, reusing code and most importantly having a shared understanding in the team.

Terraform does not enforces much conventions by itself, but we can see common practice emerge.

https://www.terraform-best-practices.com/examples/terraform/medium-size-infrastructure

Use variables files wisely

Terraform variables could be declared inline in your main terraform file (or any file loaded by TF). However , externalizing variables to separate file leads to more maintainable and modular code.

Terraform variables can be declared in file and passed in command line. File name does not matter but the usage is to name it *.tfvars. This is perfect for anything that is environment or execution specific. terraform apply -var-file="testing.tfvars"

But Terraform also automatically loads variables according to file name.

  • terraform.tfvars (exact name matching requiered)
  • *.auto.tfvars (extension matching)

Combining theses mechanisms allow to provide default values and a neat way to override them for specific environments.

The following article highlights a very good way to name variables files for increased clarity: https://blog.d2si.io/2020/02/13/structurer-son-projet-terraform/.

Separate environment without using workspaces

Yevgeniy Brikman, the author of Terraform up And Running (https://www.terraformupandrunning.com/) gives recommendations to organize files by environments.

In particular, we do not rely on build in Terraform workspaces to segregate environments (dev, prod) but rather on an explicit directory layout that reflects the various environments. It offers an explicit flat view of infrastructure and ease compare environment specificities.

  • Example of layout here vs workspaces

Split your code into modules (or stacks)

Use modules for code that you want to reuse accross environments or instances.

This french article proposes a nice split of infrastructure into “stacks” (like shared network, app1, app2). The State of each stack is stored in its own file, which ease reuse (import state) but also independent life cycles (different teams).

Use existing community maintained modules

Except for learning or very specific needs, you should rely on external modules for any complex infrastructure.

For example, it does not make sense to script a VPC creation from scratch. You may achieve quicker and likely more robust results by using a community - maintained VPC module like: https://github.com/terraform-aws-modules/terraform-aws-vpc.

In the same fashion, you may use a module to create an S3 bucket for storage of remote state. It will abstract most of the configuration of roles and encryption for free.

Modules can be found on public registries or git repositories:

Remember to explicitly write the version of the module you use in your TF code.

Bonus tips

Retrieve latest Ubuntu AMI (instead of hard-coding its ID)

resource "aws_instance" "myec2" {
  ami           = data.aws_ami.latest_ubuntu_ami.id
  instance_type = "t2.micro"
}

data "aws_ami" "latest_ubuntu_ami" {
  most_recent = true
  owners      = ["099720109477"]

  filter {
    name = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-bionic-18.04-amd64-server-*"]
  }

  filter {
    name = "virtualization-type"
    values = ["hvm"]
  }

  filter {
    name = "root-device-type"
    values = ["ebs"]
  }
}

Use TF to manage local stuff

You can perform local operations using terraform provider local and edit local files with the template provider.

Read about terraform

  • Terraform up and running. This book, in its second edition is the reference. Well written and covers all aspects of Terraform. https://www.terraformupandrunning.com/
  • Terraform in Action (https://www.manning.com/books/terraform-in-action). Although still a Work In Progress (to be published in 2020 but already accessible through Manning Early Access Program), it is a very good book that goes deep into the internals of Terraform and also its integration with other tools.

Other sources: