As an organization scales, its products mature and its operations expand in scope. It’s inevitable that complexity levels increase exponentially. Many companies attempt to address this complexity by decoupling systems, but that can often create new problems. Specifically, decentralization can have a major impact on your ability to coordinate cloud configuration effectively because static, home-grown DIY internal tools for managing these systems are challenging to implement, often creating a long-term management burden.
That’s where CloudTruth comes in. Our simple, plug-and-play system enables DevOps teams to manage their complex infrastructure without the headache of an in-house solution or decentralized approach.
To combat complexity, you must first understand where it comes from. But the tricky thing is, there are a number of interconnected sources of complexity that modern cloud engineering teams must be prepared to address.
First, the process of scaling up means adding more people, processes, and technologies, all of which create potential for coordination challenges. In addition, today’s tech stack involves multiple environments and namespaces, each of which adds more complexity. These systems are not only complex on an individual level, but they also create network effects of complexity when they integrate and interact with one another.
Many teams use different environments for development, QA, staging, and production, necessitating different instances of the system for each stage of the software development lifecycle. In many cases, these environments require further differentiation for different geographical regions, such as “production-US,” “production-EU,” etc. Similarly, most teams build sandbox environments for individual developers (i.e., “dev-matt,” “dev-kevin,” etc.), creating yet more opportunities for confusion. Without a single record of truth describing how the complete system is configured, multiple systems have to be checked to learn the truth, which is both inefficient and error-prone.
Furthermore, within a given environment, many organizations will segment out the configuration settings and variables that affect any one component into their own namespace. While this may seem relatively straightforward, each namespace adds another dimension to the configuration matrix, which can quickly snowball to unmanageable levels of complexity.
For example, consider a parameter like “cache_size.” This parameter will likely have a default value that is used across all components and environments, but a specific component might need a larger cache size in production than in staging. That means you would not only need to set “cache_size = <default>,” but you would also have to set “myComponent.cache_size = <other_value>” in a production context.
So what can we do about all this complexity? Many companies attempt to reduce complexity by decoupling systems, in order to allow teams to work on each tool independently. Through service-oriented architectures such as microservices or event-driven systems, these organizations are able to limit interaction between component tools. This approach offers several benefits:
However, decoupling also creates a lot of new problems.
A decoupled system is a decentralized system. Components can be associated with different apps, services, github repos, IaC systems, etc., all of which have to be managed separately. While that’s not inherently bad, decentralization can make it challenging to understand and manage the system as a whole. This is most evident in the following areas:
Provisioning and Configuration:
For modern cloud stacks that use kubernetes or serverless techniques, some infrastructure is provisioned and managed inline with the application, rather than entirely as a standalone or self-contained unit. This makes it hard to coordinate changes across multiple components, or even figure out where and how to make a change for a single component.
Data Sharing Between Components:
For stateful components that produce shared data (such as Infrastructure as Code (IaC) components, which generate cloud resource IDs and values that are needed by application components), you need a centralized way to share data dynamically between these different components. This applies to services, tools, and applications — decoupling can be problematic for any component that produces shared data.
Faced with this lose-lose tradeoff between complexity and decentralization, some companies attempt to build in-house solutions to manage their cloud infrastructure. (One example of a DIY project is a system to centralize and supply application configuration dynamically at runtime.) This can work for smaller systems — but it’s next to impossible at scale.
Over time, as technology evolves and teams attempt to implement new tools, a large-scale effort to define an “approved” way of doing things leaves organizations sluggish and unable to take advantage of technical advances. Instead of relying on a clunky legacy system, a more practical approach is to use the right tool for each job, and then find a way to coordinate those tools.
If you treat decentralized components like strongly abstracted functions, you can build a centralized system for managing the inputs and outputs for those functions. Such a system would offer the coordination benefits of centralization while preserving components’ decoupled nature (since individual teams would only have to worry about defining the configuration signatures of their specific components). In a nutshell, that’s what CloudTruth provides.
Our initial product is a SaaS tool that helps you easily define, manage, and track the configuration inputs and outputs of your system components in the context of common dimensions such as environments and namespaces. We’re talking about more than just a parameter store for application components. Our solution is designed to provide an all-inclusive experience tailored to meet the specific needs and workflows of modern cloud configuration management.
But this is just the beginning. We’ve got an ambitious product roadmap ahead, and we can’t get there on our own. So give us a try, let us know what you think, and most importantly, let us know what other features you’d like to see going forward. We’re excited to embark on this journey together towards the future of cloud configuration.