Terraform in Production: State, Modules, and CI/CD Without the Foot-Guns

Introduction

Building an anomaly-detection pipeline on AWS entirely through Terraform taught me the pleasant half of infrastructure-as-code: reproducible environments from a single apply. Working with Terraform beyond a solo project teaches the unpleasant half: state conflicts, drift, and the 3,000-line main.tf nobody wants to touch.

This post is the setup I now start every project with, remote state done right, an environment layout that avoids both copy-paste and workspace confusion, module discipline, and a CI/CD flow where plan is a code-review artifact.

State: The Part You Get One Chance to Do Right

Terraform state is a database pretending to be a file. Local state means the infrastructure's source of truth lives on one laptop; two people applying concurrently means corruption. Remote state with locking is non-negotiable from day one:

terraform {
  backend "s3" {
    bucket       = "acme-terraform-state"
    key          = "prod/network/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true          # S3-native locking (Terraform ≥ 1.10)
  }
}

(On older Terraform, locking uses a DynamoDB table via dynamodb_table, either way, locking on is the point.)

Beyond the backend:

• Treat state as sensitive. State contains every attribute Terraform knows, including generated passwords and connection strings. Encrypt it, restrict IAM access to it, version the bucket so you can recover from a bad write.
• Split state by blast radius. One state file for everything means every plan touches everything and every mistake can too. Split by layer and lifecycle, network, data-stores, app, and connect them with terraform_remote_state or data sources. A bad apply in app should be incapable of deleting the VPC.

Environments: Directories, Not Branches

The two failure modes are copy-pasting entire configs per environment (they drift apart within a month) and using long-lived git branches per environment (merging infrastructure between branches is misery). The layout that has aged best for me:

infra/
├── modules/                  # reusable building blocks
│   ├── vpc/
│   ├── ecs-service/
│   └── rds-postgres/
└── envs/
    ├── dev/
    │   ├── main.tf           # composes modules with dev-sized inputs
    │   └── backend.tf        # its own state
    └── prod/
        ├── main.tf
        └── backend.tf

Each environment is a thin composition root, a handful of module calls with environment-specific inputs, and has its own state. The modules hold all real logic. Environments differ by variable values, visible in one file, not by divergent resource code.

Modules: Small Interfaces, No Grab-Bags

Module design is API design. The rules I hold the line on:

• A module does one thing: "an ECS service with its ALB target group and autoscaling," not "our platform." Grab-bag modules with 60 variables become unmaintainable and un-reviewable.
• Few required variables, opinionated defaults. Callers should express intent (service_name, cpu, desired_count), not restate implementation details.
• Version your modules: git tags or a registry, and pin versions in callers. An unversioned shared module means every consumer changes at once, including prod, on someone else's commit.
• Pin providers too:

terraform {
  required_version = "~> 1.10"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
  }
}

The Pipeline: Plan in PR, Apply on Merge

The core insight that makes Terraform work in a team: a plan is a diff, and diffs are for review. Nobody should run apply from a laptop against shared environments, not for ceremony, but because the plan that gets applied should be the plan that was reviewed.

The GitHub Actions shape:

on:
  pull_request:
    paths: ["infra/**"]
  push:
    branches: [main]
    paths: ["infra/**"]

permissions:
  id-token: write        # OIDC, no long-lived AWS keys in CI
  contents: read

jobs:
  plan:
    if: github.event_name == 'pull_request'
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-plan
      - run: terraform init && terraform validate
      - run: terraform plan -out=tfplan -no-color > plan.txt
      # post plan.txt as a PR comment for review

  apply:
    if: github.ref == 'refs/heads/main'
    environment: production          # required-reviewer gate lives here
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/terraform-apply
      - run: terraform init && terraform apply -auto-approve

Details that matter:

• OIDC federation, not stored keys. GitHub's OIDC token assumes a scoped IAM role; there are no AWS credentials to leak or rotate. Use separate roles for plan (read-heavy) and apply (write).
• Gate apply with a protected environment, so production applies need an approval click even after merge.
• Add terraform fmt -check, tflint, and a security scanner (trivy/checkov) to the PR job, cheap, and they catch the classics like world-open security groups before review does.

Drift: The Slow Killer

Every console click-fix creates drift, reality diverging from code, and drift turns your next plan into a minefield of surprise changes. Two habits keep it contained: run a scheduled nightly plan against each environment and alert on any unexpected diff (drift you learn about in the morning is a cleanup; drift you discover mid-incident is a catastrophe), and when infrastructure was changed by hand out of necessity, bring it into code with import the same week rather than letting the exception become the norm.

Takeaways

• Remote, encrypted, locked state from the first commit; split state by blast radius.
• Environments are thin directories composing versioned modules, differences are variable values, not divergent code.
• Plan in PRs as a reviewable artifact; apply only from CI, via OIDC-scoped roles, behind an environment gate.
• Detect drift nightly, and re-import manual fixes promptly.

Terraform's value was never "write HCL instead of clicking." It's that infrastructure becomes reviewable, diffable, and revertible, and every practice above exists to protect exactly that property.