How to store and leverage Terraform remote state and AWS S3

In this article we are going to walk through configuring a Terraform remote state file with an AWS S3 backend that deploys an AWS instance. We will then use that remote state file to programmatically stop and start the instance using Terraform output for the instance id and boto3 with python.

Configuring remote state

The following HCL will create a remote state backend in us-west-2, deploy an aws instance in the free tier and set two outputs for the instance id and ip. You can use an existing S3 bucket, create a new one in the console or reference this terraform code that will create the bucket for you.

main.tf

terraform {
  backend "s3" {
    bucket = "YOUR_UNIQUE_BUCKET_NAME"
    key    = "demo/instance/terraform.tfstate"
    region = "us-west-2"
  }
  }

provider "aws" {
  profile = "default"
  region  = "us-west-2"
}

resource "aws_instance" "app_server" {
  ami           = "ami-830c94e3"
  instance_type = "t2.micro"

  tags = {
    Name = "RemoteStateInstance"
  }
}

output "instance_id" {
  description = "ID of the EC2 instance"
  value       = aws_instance.app_server.id
}

output "instance_public_ip" {
  description = "Public IP address of the EC2 instance"
  value       = aws_instance.app_server.public_ip
}

Once this Terraform config is applied it will create a terraform.tfstate file in your S3 bucket path demo/instance/terraform.tfstate. You have successfully deployed Terraform state to a remote backend!

Using Python with Boto3 to stop and start the AWS Instance

Boto3 is an AWS python SDK that will let us programmatically interact with AWS services. It can be configured for authentication to your AWS account with the AWS CLI or a credentials file.

For this example we are going to use boto3 to stop and start the deployed AWS instance using the instance_id we set as a Terraform output.

The following Python script will take an environment variable S3_INSTANCE_ID and provide the user with options to stop or start the instance with the supplied id.

The instance id can be obtained from the terraform output command or by reviewing the state file in S3.

instance_id = "i-0825d0daa987363d9"
instance_public_ip = "18.236.141.53"

You can export the instance id as an environment variable in your local environment and the script will read it with os.environ.get.

export S3_INSTANCE_ID=i-0825d0daa987363d9

Now executing the script with the --stop flag will look for the specific instance in the environment and stop it in EC2. If the id you provide for the variable S3_INSTANCE_ID is not in your AWS region then the script will exit.

python instance_tfstate.py  --stop

i-0825d0daa987363d9
Success {'StoppingInstances': [{'CurrentState': {'Code': 64, 'Name': 'stopping'}, 'InstanceId': 'i-0825d0daa987363d9', 'PreviousState': {'Code': 16, 'Name': 'running'}}], 'ResponseMetadata': {'RequestId': '3b6b71de-d2fa-483e-8166-c2696d8bed07', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '3b6b71de-d2fa-483e-8166-c2696d8bed07', 'cache-control': 'no-cache, no-store', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'content-type': 'text/xml;charset=UTF-8', 'content-length': '579', 'date': 'Wed, 08 Sep 2021 12:40:31 GMT', 'server': 'AmazonEC2'}, 'RetryAttempts': 0}}

Using Terraform remote state outputs with dynamic changes

Rather than manually adding instance ID’s to your environment whenever your configuration or environment changes you can use CloudTruth to dynamically pickup changes from the Terraform remote state file. This means if you apply and delete the EC2 instance you can automatically pick up new instance id from the remote state file with no code or environment changes.

CloudTruth allows you to read directly from a remote state file stored in S3 and programmatically access anything within the state file as a parameter by integrating directly with AWS S3.

We can setup a CloudTruth Project and reference the instance_id value with the CloudTruth CLI or the UI. The following CLI commands create the project and our parameter that has our external instance id.

cloudtruth project set terraform_state 

You can put your integrated account and bucket name in the CLI.

cloudtruth --project terraform_state parameter set S3_INSTANCE_ID --fqn aws://YOUR_ACCOUNT/us-west-2/s3/?r=YOUR_BUCKET/demo/instance/terraform.tfstate --jmes outputs.instance_id.value

We use the jmespath outputs.instance_id.value to pull the value directly out of the state file. The state file output looks like this:

  "outputs": {
    "instance_id": {
      "value": "i-0825d0daa987363d9",
      "type": "string"
    }

The CloudTruth UI

From the projects page create a new project called terraform_state.

Add a new parameter named S3_INSTANCE_ID.

Set an External value as the S3 bucket destination path YOUR_UNIQUE_BUCKET_NAME/demo/instance/terraform.tfstate in us-west-2 and use the JMESPATH selector outputs.instance_id.value. Hit save to set the value to the instance id output from the remote state file.

Now you can use our Rest API or the CloudTruth run command to pass the Terraform remote state output directly to your infrastructure scripts!

This example uses CloudTruth run which pulls in the environment from the project terraform_state we created.

cloudtruth --project terraform_state run -- python instance_tfstate.py  --start

i-0825d0daa987363d9
Success {'StartingInstances': [{'CurrentState': {'Code': 0, 'Name': 'pending'}, 'InstanceId': 'i-0825d0daa987363d9', 'PreviousState': {'Code': 80, 'Name': 'stopped'}}], 'ResponseMetadata': {'RequestId': '2b135efe-790c-4bbb-8497-03ede4da5126', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '2b135efe-790c-4bbb-8497-03ede4da5126', 'cache-control': 'no-cache, no-store', 'strict-transport-security': 'max-age=31536000; includeSubDomains', 'content-type': 'text/xml;charset=UTF-8', 'content-length': '579', 'date': 'Wed, 08 Sep 2021 15:28:56 GMT', 'server': 'AmazonEC2'}, 'RetryAttempts': 0}}

The instance is now Running in AWS EC2!

Don’t forget to cleanup your infrastructure with terraform destroy.

Summary

We walked through how to deploy Terraform remote state to an AWS S3 backend and deployed an EC2 instance in the free tier while specifying the instance id as a Terraform output. We then leveraged the instance id with a sample Python script using boto3 that can stop and start the instance with a manual environment variable and reviewed how we can dynamically get Terraform state output with CloudTruth.