CloudTruth & Kustomize Kubernetes Deployments

There is a better way to Kustomize Kubernetes deployments for multiple environments by using a centralized parameter and secret store with templating.

Prerequisites

To follow along with this blog post, you should have the following:

• A CloudTruth account. If you don’t have one, you can sign up for free here.
• A code editor, like VS Code

What is Kustomize?

In many organizations, you’ll see multiple environments, sometimes including:

• Development
• Staging (or UAT)
• Production

Each of those environments has configurations that need to be created, managed, and maintained. It can be a hassle as it’s more deployment configurations, parameters, config data, and code to manage.

Kustomize gives the ability to segregate environments (like dev, staging, and prod) and have configuration files that are specific to each environment. The configuration files are called kustomization.yaml. Then, the Kustomize configuration files point to a “base” configuration, which is essentially a bunch of Kubernetes Manifests. Kustomize then changes the “base” configuration of the Kubernetes Manifests with the values from the Kustomize configuration files.

Below are two screenshots for a visual representation.

How To Use Kustomize

To use Kustomize, you need three things:

• A folder hierarchy (like in the screenshot above)
• Kubernetes Manifest(s)
• One or multiple (depending on how many environments you have) kustomization.yaml files

First, create three directories called base, overlays, and dev

Next, inside of the base directory, create two files:

• deployment.yaml
• kustomization.yaml

Inside of the deployment.yaml, add the following Kubernetes Manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
spec:
  selector:
    matchLabels:
      app: nginxdeployment
  replicas: 2
  template:
    metadata:
      labels:
        app: nginxdeployment
    spec:
      containers:
      - name: nginxdeployment
        image: nginx:latest
        ports:
        - containerPort: 80

Inside of the kustomization.yaml, add the following Kustomize configuration file:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml

You’ve created two things; the Kubernetes Manifest that Kustomize will look for and the Kustomize config file that tells Kustomize what Kubernetes Manifests you want to use.

Next, inside of the dev directory, create a new file called kustomization.yaml and add the following code:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base/

replicas:
- name: nginx-deployment
  count: 1

What the above kustomization.yaml file is doing is pointing to the base directory, which is where the Kuberentes Manifest and Kustomize config base files live. Then, it’s specifying in the configuration that for the nginx-deployment app, only add one replica.

Now, you can run kustomize build and see the output, specifying only one replica.

How about Configuration Data?

Here’s what Kustomize doesn’t solve:

• You still have multiple environment/parameter/config data files
• You have to have multiple directory structures for different stages (dev, staging, prod, etc.)
• There’s no central way to pass in config data/parameters from one centralized location
• Kustomize focuses on replacing values instead of managing values for each environment/stage

Although the above is great and Kustomize is an amazing tool, you still need a way to centrally manage the config data that you’re passing in. You can use both Kustomize and CloudTruth to segregate environments, config files, and manage multiple parameters.

Taking a look at the screenshot below, you can see that:

• There’s a project called kustomize
• There’s a parameter called replicaCount that has a value of 1 Kubernetes replica

This means that using CloudTruth, Kustomize will look at CloudTruth to retrieve the replicaCountvalue for the k8s manifest.

Inside of the kustomization.yaml file, you’d specify the CloudTruth parameter name replicaCount in two curly brackets, as shown in the code snippet below:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ../../base/

replicas:
- name: nginx-deployment
  count: 

The last step is to run the CloudTruth command below to:

• Specify what project you want to pull a parameter from
• The environment
• Specify the command you want to run. In this case, it’s the kustomize build command

cloudtruth --project "kustomize" --env development run -- kustomize build

The output would be a Kubernetes Manifest that contains the 1 replica.

Wrapping Up

In the DevOps world today, there are a million tools to choose from. Because of that, you may sometimes see that tools are chosen that can do a job, but may not be the right fit. However, combined with other tools and platforms, they become a perfect match.

If you’d like to get started with CloudTruth today, there is a free forever version here.