Terraform State: The 3AM Lesson in Why You Never Edit It By Hand

2025-04-11 · terraform · iac · devops

Some mistakes teach you their lesson immediately. Terraform state corruption teaches it at 3AM, in production, with a stale lock file.

The “quick fix”

A resource was stuck: terraform apply kept failing on a timeout for one S3 bucket whose region had drifted. Someone — me — had the bright idea to open terraform.tfstate and remove the offending resource entry. It’s JSON, right? How bad can it be?

Bad in three compounding ways:

State is a graph, not a list. Other resources depended on that bucket’s ID via depends_on and interpolated attributes. Removing the node left dangling references that failed on the next apply, in resources that had nothing to do with the original problem.
State is the source of truth for drift detection. Terraform no longer believed the bucket existed, so it planned a fresh create against a bucket that very much existed — which fails, which is better than the alternative: a create that succeeds with a suffixed name and leaves the original orphaned, unmanaged, and billed.
The serial number. State has a monotonic version counter. Hand-edits that don’t bump it correctly poison every subsequent lock comparison.

The actual fix

# What I should have done in the first place:
terraform state rm aws_s3_bucket.stuck_bucket        # unmanage, don't lie about existence
# or better, if the resource truly is gone upstream:
terraform import aws_s3_bucket.stuck_bucket real-bucket-name

state rm and import exist precisely because state surgery is graph surgery. The CLI updates every dependent edge correctly, bumps the serial, and keeps the lock protocol intact.

The setup that prevents all of this

Remote state (S3 + DynamoDB locking, or Terraform Cloud): nobody has a local terraform.tfstate to hand-edit, and concurrent applies lock properly
Mandatory plan review: no apply without a human reading the plan; an unexpected -/+ or -destroy stops the line
prevent_destroy lifecycle blocks on databases and buckets: the one resource class where terraform’s honest mistake becomes your outage

The deep lesson is the same one from every IaC tool: the state file isn’t a cache, it’s the database of record for your infrastructure. You would not hand-EDIT a production database with a text editor at 3AM. Treat state with the same respect, and the 3AM never happens.