Keep your IAC code clean
Jim Brikman is a Terraform (IaC tool) consultant who learned a few interesting things from writing 300,000 lines of IaC code alongside many teams that use the tool.
One of these learnings was that many teams keep their IaC code as a single monolithic file. All the environment configurations (development, QA, testing, staging and production) were stored on one gigantic file. This led to several problems including:
difficulty finding the line of code that needed to be updated
long wait times for the monolith IaC code to execute (in some cases up to 5 minutes)
increased risk of missing red lines e.g. “executing code but the database will be deleted”
everyone needs to be given admin permission to execute the file, so less experienced developers gain admin-level and could unwittingly make changes to production infrastructure
In summary, a monolith IaC file risks a minor change breaking everything.
Now, making a monolithic codebase is a thing that most teams don’t plan to do, but it’s obviously happening if IaC consultants are finding them. Maybe they just finished an SAP integration project (hard to drop the taste for monoliths after doing that for a long time).
Jim recommends the following (and they make logical sense):
Isolate each environment as a separate IaC file or folder
Break down each environment into various components e.g. database, VPC, frontend — VPC changes maybe once or twice a year while frontend might change 10 times a day, so by separating them, you’re reducing the risk of config changes affecting VPC 10x a day
Make all the above components reusable
Modular architecture with smaller components reduces the surface area for attacks and errors