Refactoring Terraform state into separate files in GCP GCS

Recently I needed to refactor some Terraform configuration, which had grown into quite a large single state file, into smaller separate directories with their own state files.

This is a little bit tricky but the Terraform CLI tool helps a lot.

The first thing is to get the existing single state file JSON:

terraform state pull | tee terraform.tfstate

This should work for any storage backend.

Alternatively you can use gsutil to copy the Terraform state file from a GCS bucket:

gsutil cp gs://your-bucket/your-prefix/terraform.tfstate ./terraform.tfstate

Once you’ve got a local copy of the large single Terraform state file, you can start working with the Terraform CLI tool to extract resources out into separate smaller files.

Extracting a single resource looks like this:

terraform state mv \
  -state=terraform.tfstate \
  -state-out=extracted.tfstate \
  'module.entity-smart-commissioning-status["prod"]'{,}

The {,} on the end is a little Bash trick to repeat that argument, e.g. foobar{,} expands to foobar foobar. This keeps the resource name the same when moving it into the new state file.

You can run that command for each resource that needs extracting, and for several different extracted Terraform state files if that applies in your case.

Once that is done, you can push the trimmed down original state file back to the storage backend with:

terraform state push terraform.tfstate

Again, there is also the alternative of doing this via gsutil to upload the file to a GCP bucket without going via Terraform:

gsutil cp ./terraform.tfstate gs://your-bucket/your-prefix/terraform.tfstate

You can use either of the commands to push the new extracted state files to the correct locations from their respective Terraform directories.