Using the run_cmd function to configure Terragrunt variables

Terraform is a great tool for provisioning infrastructure, but its usability tends to suffer at scale. Many users end up using some form of wrapper like Terragrunt to act as a management layer. You can think of it like needing a Makefile to drive a compiler processing a number of source files. Scale makes everything harder, so even with a wrapper, there is significant complexity with managing all the variables and module inputs. This is especially true w.r.t. how they relate to each other and across multiple environments. As such, there is a benefit to using other tools to manage this configuration state, and one of the easiest ways to integrate Terragrunt with these tools is through use of its run_cmd function.

Terragrunt DRY configuration

Terragrunt is one of the more popular wrappers for Terraform. It does a great job of helping you manage multiple modules with inputs, and tie it all together in a DRY fashion. Where it begins to break down is in managing the complexity of hundreds of input variables and secrets and how they relate to each other across multiple environments. Fortunately, the extensibility of Terragrunt gives you many options to configure variables from an external data source.

With vanilla Terraform, one has two sources of data for configuration. The most obvious method is through Terraform variables, for which you can supply values for in a few ways. They can be set as defaults inline with the variable declaration, or overridden via tfvars files or environment variables. The second method is through use of a provider or data source. However, those aren’t always a good fit as they can’t directly set the values of existing Terraform variables. It can also be messy referencing them across your codebase, but that can be handled through use of local variables.

When driving Terraform with Terragrunt, one would like to have a single source of configuration that can drive Terraform variables. This source can also drive Terragrunt inputs to modules or even the parameters that control Terragrunt itself. While you can accomplish this through a number of directory layouts and files, this becomes extremely complex at scale. It becomes very difficult to understand how config relates across the system and make changes to it. What we need is a way to separate out the management of configuration from Terragrunt. Then we can make use of the Terragrunt hooks to connect the two systems.

To demonstrate how to accomplish this, lets start with some simple Terraform that takes a variable.

hello.tf

variable "message" {
  default = "default message"
}

output "hello" {
  value = "Hello ${var.message}"
}

Running it with Terraform gives the following:

$ terraform apply -auto-approve
...<snip>...
hello = "Hello default message"

The simplest Terragrunt configuration that can pass in a value for the message variable, would look like:

terragrunt.hcl

terraform {
  extra_arguments "set_env" {
    commands = ["plan", "apply", "destroy"]
    env_vars = {
      TF_VAR_message = "terragrunt"
    }
  }
}

We see that the value was passed and used correctly when run with Terragrunt:

$ terragrunt apply -auto-approve
...<snip>...
hello = "Hello terragrunt"
$ 

Configure Terragrunt variables with the run_cmd

Now that we’ve shown we can use Terragrunt to easily set Terraform variables, lets try and do it dynamically. We will use an external source for variables rather than being hardcoded in Terragrunt or through local files.

terragrunt.hcl

terraform {
  extra_arguments "set_env" {
    commands = ["plan", "apply", "destroy"]
    env_vars = {for k,v in local.myconfig : "TF_VAR_${k}" => v }
  }
}

locals {
  myconfig = jsondecode(run_cmd("sh", "-c", "echo '{\"message\": \"echo\"}'"))
}

To break this down, in the locals section we use sh+echo to simulate a run_cmd command that generates some json config. We then parse that with the jsondecode function so that myconfig ends up being a native HCL map.

In the env_vars section, we iterate over myconfig to generate a new map of environment variables and append the TF_VAR_ prefix that Terraform recognizes as an environment variable.

When run, we see it works as expected:

$ terragrunt apply -auto-approve
{"message": "echo"}
...<snip>...
hello = "Hello echo"
$ 

Note that the Terragrunt run_cmd echos the result of the command when it runs. To prevent this, we need to pass --terragrunt-quiet as the first argument to run_cmd You’d typically need to do this if some of your variables are secret, but doing so also serves to de-clutter your output for day-day execution.

Providing an external source for configuring Terragrunt run_cmd variables

Now that the plumbing is in place, we need to plug in the actual external command that will populate your config. If that command generates a simple json hash, then your work is done. Just plug it in in place of the sh/echo and go back to focusing on building your infrastructure. However, in the real world, things are rarely that simple. The command may not be generating json or may be generating complex json that is difficult to use.

Decoding options

You have a few options If the command output is not in json. If it is in yaml or csv, then you can use the Terraform functions yamldecode or csvdecode in place of the jsondecode we used in the example above. If it is in neither of those formats, then you may be able to use some combination of the Terraform string and collection functions (split/replace/regex/etc) to manually parse the output into a useable form. As a last resort, you can use some other external command to do the transformation into a parseable form. However, adding an additional runtime dependency to Terragrunt may not be a path you want to go down.

When your command is producing complex structured data, you’ll need to transform it into something simpler. By using Terraform for expressions you to do some fairly sophisticated transformations to get the data into a simpler form. For example, here at CloudTruth we provide a CLI that allows access to one’s configuration data from the terminal. It provides a natural way to integrate into tools like Terragrunt that can call out to an external program. However, the json it produces is more complex than we’d like for use within terragrunt:

$ cloudtruth --env staging --project terragrunt-hello param list --values --secrets --format json
{
  "parameter": [
    {
      "Description": "This is the message",
      "Name": "message",
      "Param Type": "string",
      "Rules": "0",
      "Secret": "false",
      "Source": "default",
      "Type": "static",
      "Value": "This is the staging message from CloudTruth!"
    }
  ]
}

As a result, we need some extra transformation to make it useable within Terragrunt. We can do this with an extra line in our locals block that uses a for expression to map it into a simpler HCL map of our parameter Name/Value pairs.

terragrunt.hcl

terraform {
  extra_arguments "cloudtruth_env" {
    commands = ["plan", "apply", "destroy"]
    env_vars = {for k,v in local.myconfig : "TF_VAR_${k}" => v }
  }
}

locals {
  environment = "staging"
  cloudtruth_params = jsondecode(run_cmd("--terragrunt-quiet", "cloudtruth", "--env", local.environment, "--project", "terragrunt-hello", "param", "list", "--values", "--secrets", "--format", "json"))
  myconfig = {for p in local.cloudtruth_params.parameter : p.Name => p.Value }
}

Terragrunt Output with variables configured from the run_cmd

$ terragrunt apply -auto-approve
...<snip>...
hello = "Hello This is the staging message from CloudTruth!"
$

Since we have our externally generated config in a HCL native form, we can reuse it for the input blocks to modules in the same way as we do for env_vars. We can also reference specific variables like local.myconfig.message anywhere in Terragrunt configuration that allows one to reference a local variable. Using an external source to configure Terragrunt variables makes for a truly DRY config that is easy to manage across multiple environments at scale