This post was inspired by this excellent twitter thread by Charity Majors.
The twelve-factor app is a popular methodology for building software-as-a-service apps. It defines a set of factors that should be applied to the software-building process. Twelve-factor App was initially defined at Heroku in 2011, and nowadays, most cloud providers recommend it as a practical way to create web applications (GCP, AWS, Azure, IBM). 3rd factor is called Config and is defined as:
An app’s config is everything that is likely to vary between deploys (staging, production, developer environments, etc). This includes:
• Resource handles to the database, Memcached, and other backing services
• Credentials to external services such as Amazon S3 or Twitter
• Per-deploy values such as the canonical hostname for the deploy
The twelve-factor app stores config in environment variables.
It made a lot of sense at the time: env variables provide a clean separation from code, and they are language- and OS- agnostic. Changing environment variables usually requires a redeploy and restart, and it’s reflected in the example above: all of them are static values and unlikely to change between deploys.
Modern apps are much more complex and distributed, are deployed to more places, and are updated more frequently (thanks to CI/CD). There is an increased need to make configs vary not just between deploys but between requests from different customers or segments of users.
For example:
- Enable a feature only for some customers
- Use a more expensive LLM model for premium customers
- Manage a custom set of features and limits for an enterprise pricing plan
The examples above are not just more fine-grained than per-deployment configs but much more dynamic: they can change at any time independent from deploys.
Feature flagging is one attempt to solve this. The idea is to separate releases of features from software deployments. Any new feature is hidden behind a feature flag, deployed to production in the disabled state, and then [gradually] enabled. This approach supports having different config values across multiple dimensions: deployments, customer tiers, end-user segments, etc.
Feature flagging is all about releasing new features. What about a more permanent configuration for long-term use? How do we even know whether something is temporary or not? We don’t.
We need a very cheap way to mark some parts of the code as dynamic to group and organize them later. When we are in the flow of writing code, we need tools that minimize decisions that we need to make, like:
- Where to put this config value?
- What is the best size limit or timeout for this operation?
- How do I create a new secret?
Modern product-building practices require non-engineers to be more involved in managing the product in production. Having engineers always in the loop does not scale; letting non-engineers change code configuration directly is risky.
We need to support additional levels of abstraction:
- a human-readable configuration that makes sense to humans managing the product
- low-level configuration used in the code
- mapping and validation between those two
But there is a much more fundamental issue. Configs are code, just with a different editing and deployment pipeline. Changing code is risky; changing configs is even more risky because it happens more frequently. Collectively, through trial and error, we learned how to edit and deploy code safely: types, unit tests, source control, code review, CI/CD, staged deployments, and observability.
Most feature flagging tools sidestep that. They are essentially a glorified key-value store with a GUI. We need a configuration management solution that treats configs exactly as code. Or even better, a solution where configs are an actual code.
This is what we are building at Lekko.
.png)
