Hi family, hope we are doing well. We recently had to transition our infrastructure to Terraform.
Join me through this tutorial to find out how to transition an Ops team seamlessly to IAAC and CI/CD in a single migration step.
“Greenfield” vs “Brownfield” Projects In IAC
According to many authors, software architectures come from two different types of projects. In the first place, there are the “Greenfield” projects. Which are brand new projects; this means the software engineering team starts implementing the architecture without any antecedent. Then, there are the “Brownfield” projects. These are projects that already have a history. The architecture was thought differently and the implementations follow accordingly. Brownfield’s meaning is that the team needs to figure out how to transition to the new architecture. In summary, there is no need to start from scratch, but the challenge is to switch to the desired architecture.
My experience in infrastructure is analogous to what I’ve mentioned in the previous paragraph. There are projects that start with IAC and others that don’t. On one hand, Greenfield projects use IAC, the team has the required coding skills, and a CI/CD like workflow is already implemented; at least partially. On the other hand, Brownfield projects require many extra steps and involve more complexity. The migration requires time for adoption and a well thought and planned pipeline.
In this article, I’ll sum up my experience with a Brownfield project in IAC.
Import Your Existing Infrastructure to Terraform
The first step requires the team to Terraform the Cloud Infrastructure. Every resource used in the Cloud Provider console must be converted to code. There are many ways to accomplish the import.
One way would be to take a look at the resources and start coding them. Despite how logical this sounds, this is a bad option. It would take the developer days to code everything, without even mentioning the mess to sync the configuration code with the state code. A better approach is to use Terraform itself. Start by importing the state code using Terraform import. After that, use the Terraform show command to get the configuration code. This option looks very reasonable. Nonetheless, there is even a better way. The Terraformer tool recursively imports both the configuration and state code automatically.
Use Terraformer to import the cloud infrastructure. Here is a sample of how to use it. It pretty much explains itself.
Terraformer exports a folder with the name of the project and subfolders that refer to the regions, resources, providers, outputs, and Terraform states. Despite this looks complete, the Terraform states are not always useful. For instance, I started my Brownfield project with Terraformer 0.13.4. But Terraformer imports the states with 0.12. The incompatibility makes me unable to merge states. In this sense, I planned a different partial approach.
I merged the configuration code from Terraformer and imported the state code using Terraform import. I made an inventory of the resource names scrapping the .tf files generated by Terraformer. Once I figured the approach I coded a Python script to automate my import Operations.
The scripts requires as input several parameters and a plain file. The parameters refer to the project id, the zone, the target module name and the resource name. The plain file contains several lines which refer to the resource name and its zone. Both divided by a single space. Running the script imports the configured resources.
In order to test the success of the script, run Terraform plan. The resources that were imported shouldn’t present any change.
Continuous Integration and Continous Infrastructure Deployment
Once the infrastructure had been Terraformed, I proceeded to build a CI/CD workflow. Without it, operations in IAC would be unimaginably chaotic. For instance, DevOps would be able to apply changes locally in their machines without even committing the code. As a consequence, state files and configuration files would constantly fail for the members of the team (which would cause an eventual big fail of the project). In summary, if there is no CI/CD, then the project will be a mess.
Continuous Integration (CI)
Continuous Integration is defined as the pipeline that builds the project, runs syntax, acceptance, unit tests, and packages it for its deployment. In Terraform, validating the acceptance of the code would require the construction of new infrastructure and the deployment of a dummy service which could be used for validation. Even though this idea sounds good, it is not something smart. Imagine your code is used to build a 9 node Kafka/Zookeeper Cluster. You would need to create these servers for each commit which makes this expensive.
In contrast to all mentioned, I decided to stay on syntax checking. Thus, I built a Python script that would be used to iterate all the subfolders with a Terraform project. Each iteration starts terraform, authenticates to the remote backend, and validates the code syntax. Here is a sample of my script.
The script executes tests based on the unit test package by Python. Everything is packaged inside a Docker container which is run by a CI tool. I used Drone CI.
Continuous Infrastructure Deployment (CID)
The next step is a tricky one. Continuous Deployment is so famous, yet not often implemented due to its complexity. In the Terraform scope, I chose to use Terraform Cloud. This tool is offered by Hashicorp, is free up to 5 users, handles state files by Git commits, allows notifications (e.g.: Slack) and offers an awesome UI.
Each project handled by Terraform has its own workspace on Terraform Cloud. Each workspace listens to a specific integration branch (only committable by pull requests) and has notifications to Slack for every plan, success, failure, application of the plan, creation, etc.
Terraform Cloud Console
Apply the terraform plan
Updated Pull-Request Workflow
The following picture depicts the workflow DevOps engineers must follow in order to manage infrastructure with CI/CD.
The image is pretty much self-explanatory. But this is something that can be applied as a single step once it has been tested and implemented. As an addition, Terraform Cloud forces DevOps to only run Terraform plans and applies from the console. Thus, avoiding any local change.
Conclusion
This article presented an approach to apply IAC and CI/CID to a “Brownfield” project in Cloud Operations. It started by describing a way to import infrastructure from a cloud provider. Then, depicted how to synchronize the state and configuration Terraform code. After that, it presented an implementation of Continuous Integration. Then it continued presenting how to do Continuous Infrastructure Deployment.
Wrap-up
Here we go folks. Hope this whole tutorial helps you. For any question please leave a comment below.
Stay tuned and follow me for more updates. Don’t forget to give us your 👏 if you enjoy reading the article as a support to your author!!
