Writing Infrastructure-as-Code (IaC) is relatively new in software development terms. Workflows and best practices are still incubating and vary widely from team to team. This article provides an opinionated approach how to structure IaC in Git and align development workflows using Gitflow in an AWS multi-account environment.
Cloud Architecture
Clouds are diverse by nature and each cloud provider offers different services to implement environments; AWS has accounts, Azure has subscriptions and GCP has projects. A cloud for small to midsized companies basically consists of two groups of components:
- Core / Foundation
DNS, directory service, logs, security management, networking services etc - Workload
Your company’s applications and required services
I strongly recommend reading the AWS Landing Zone introduction article explaining the concepts behind multi-account AWS architectures.
The principle is having dedicated AWS accounts for isolation of environments. For larger companies its feasible to deploy multiple sets of workload accounts and structure them according to business units, projects and/or teams. Keep productivity in mind while designing your cloud; too many accounts are painful to work with, while too few may lead to name clashes and larger incident blast radius.
A potential AWS account structure for a small to midsized company might look as followed:
According to the AWS Landing Zone best practices, the accounts provide the following:
Master (root)
- service control policies
- organizations (sub-accounts)
- consolidated billing
Shared-Services (core)
- DNS (+ delegation)
- AMIs / shared Docker images
- LDAP / Active Directory
- Deployment tools
- Monitoring
Logging (core)
- consolidated workload logs
- security / cloud trail
Security (core)
- users (iam)
- security tools and audit
Networking (core)
- networking services
- AWS Direct Connect
Production / Testing / Sandbox (workload)
- AWS application services
- your applications
This account structure can be represented in Git and deployed using a CI/CD pipeline. Let’s take a look at Gitflow first.
Gitflow Introduction
Many software projects follow an iterative project release cycle. This workflow fits well with a Gitflow based branching model and CI/CD tooling following the release early, release often philosophy.
While working on different software projects using a variety of programming languages and frameworks, Gitflow has proven to be a great fit for small to the medium sized teams I worked with; it’s simple to learn, scales well and encourages collaboration through code reviews whenever a branch is merged.
A common Gitflow workflow looks as followed:
The branches master and develop persist while feature branches are created for improvements or bug fixes and merged back into the parent branch. Each circle represents a commit to the Git repository and triggers a CI/CD pipeline.
This well established workflow is also suitable to manage infrastructure code by mapping Git branches to specific AWS accounts.
Gitflow for Workload Accounts
To use Gitflow with workfload environments, your account structure should match the application development lifecycle which differs from company to company. The basic concepts are often similar to:
- Development / Sandbox (one or multiple)
to prototype and develop new infrastructure components - Staging / Testing (one)
to test and integrate all components - Production (one)
to operate the production workload
If your organization performs UAT (user acceptance testing), an additional stage between Staging and Production might be feasible.
These environments can be mapped to the Gitflow branches as followed:
Note: The names Sandbox and Staging are just personal preference.
Committing to a branch (or pushing or merging a branch) should automatically trigger a CI/CD build in your tool of choice which creates cloud infrastructure, builds and tests your software, and provisions the infrastructure in the according workload environment.
Please refer to my next article for a skeleton CI/CD pipelines using Gitlab.
Gitflow for Core / Foundation Accounts
Automated testing of core / foundation account infrastructure is a bit more difficult. Some resources (AWS Direct Connect peers, DNS domains, 3rd party service accounts etc) might be unique and a test setup therefore impossible, too expensive or too much effort to maintain.
An architecture with test accounts could map to Gitflow as followed:
Its feasible to start without dev accounts for core accounts and introduce them once the requirement arises or the impact of production issues justify the extra effort and cost.
Git Repository Content
A Git repository might combine infrastructure and application code or split it into a dedicated repository. The following variations are to be considered:
- Infrastructure + Application Code
Useful for smaller apps or serverless functions to deploy infrastructure together with the application - Infrastructure Code only
Applicable for infrastructure code without application code, for example to setup the AWS structure explained above or as a dedicated repo for a application infrastructure services - Application Code only
Feasible if the application requires many infrastructure services for which the code is separated into a dedicated Git repository
Personally, I’ve work with all three layouts but generally prefer to split infrastructure and application code as it simplifies the CI/CD pipelines.
Conclusion
The AWS account structure heavily depends on your company’s organization, development lifecycle and available engineers to maintain extend your cloud. Once the architecture is laid out, Gitflow and a powerful Git service like Github or Gitlab with integrated CI/CD running Terraform are the building blocks for a modern cloud.
That’s it folks. For more information on IAC, please read my other articles.
#YouAreAwesome #StayAwesome