Gitlab CI/CD using Terraform

Gitlab CI/CD using Terraform

Lionel Tchami's photo
Lionel Tchami
·

8 min read

Table of contents

Terraform and Gitlab really do make an excellent combination in developing and deploying any immutable infrastructure as code. This article explains an opinionated DevOps workflow and also provides a CI/CD pipeline template using Gitlab and Terraform to deploy multiple cloud environments.

Introduction

If you have not read my previous post about Gitflow and working with multiple AWS accounts, please take look as this blog post builds on the concepts explained there.

Why Gitlab?

If you don’t know Gitlab or don’t have an account yet, please check it out here. The hosted Gitlab SaaS is free to use and it offers a wide collection of seamlessly integrated services which really enable a highly productive DevOps workflow. Some relevant topics for this article are:

For the past years, Gitlab has proven itself of inestimable value for my professional work in maintaining repos in Kotlin, Python, Go, Typescript, Terraform and much more.

Example Pipeline #1: Local File (stateless, simple)

To make it simpler, we’ll use the local_file Terraform resource from the Local Provider and create a file containing the name of the environment we’re running in.

The Code

main.tf in the root directory contains the following:

// main.tf  
resource "local_file" "foo" {  
    content  = var.environment  
    filename = "/tmp/bar"  
}  

variable "environment" {}

We will keep the infrastructure code DRY and also create configuration files in a the subdirectory config/; one file per environment:

// config/production.tfvars  
environment = "production"

// config/staging.tfvars  
environment = "staging"

// config/sandbox.tfvars  
environment = "sandbox"

And finally the Gitlab CI/CD pipeline .gitlab-ci.yml in the root directory is as below:

# .gitlab-ci.yml  
stages:  
  - infrastructure  

.terraform:  
  stage: infrastructure  
  image:  
    name: hashicorp/terraform:light  
    entrypoint:  
      - '/usr/bin/env'  
      - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'      
  before_script:  
    - terraform init  

# ~~~~~~~~~~~~~~~~ Apply ~~~~~~~~~~~~~~~~  
.terraform_apply:  
  extends: .terraform  
  script:  
    - terraform apply -auto-approve -var-file=config/${ENVIRONMENT}.tfvars  

terraform_apply_production:  
  extends: .terraform_apply  
  variables:  
    ENVIRONMENT: production  
  only:  
    refs:  
      - master  

terraform_apply_staging:  
  extends: .terraform_apply  
  variables:  
    ENVIRONMENT: staging  
  only:  
    refs:  
      - develop  

terraform_apply_sandbox:  
  extends: .terraform_apply  
  variables:  
    ENVIRONMENT: sandbox  
  except:  
    refs:  
      - master  
      - develop

Some Explanation

This example pipeline defines an abstract (or not executed) job .terraform from which all other jobs extend. The before_script block executes the compulsary terraform init to download providers and initialize the state.

.terraform_apply defines another abstract job, from which all environment specific jobs will inherit and set the config file to use via gitlab variables.

The Gitlab strategies defined with only and except constrain or limit the Git branches, for which the specific job is executed.

Following our Gitflow workflow of merging branches, the assigned environment is now automatically deployed.

Example Pipeline #2: AWS Docker Registry (advanced)

As our next step, we will create one AWS Docker registry (ECR) per environment in a single AWS account using S3 + DynamoDB to store and lock the Terraform backend state.

Preparation — Terraform state file

If you don’t have an S3 bucket and/or DynamoDB table yet, please visit this article or use the following Terraform snippet with the randomized bucket and table names:

resource "aws_s3_bucket" "terraform_state" {  
    bucket = "terraform-state-8ab67ec2"  

    versioning {  
      enabled = true  
    }  

    lifecycle {  
      prevent_destroy = true  
    }      
}  

resource "aws_dynamodb_table" "terraform_lock" {  
  name = "terraform-state-8ab67ec2-lock"  
  hash_key = "LockID"  
  read_capacity = 5  
  write_capacity = 5  

  attribute {  
    name = "LockID"  
    type = "S"  
  }  
}

Preparation — Access key

Our Terraform AWS provider requires AWS access and secret keys to modify AWS infrastructure on behalf of the user the whome access keys belong to.

You can either create access keys for an existing user or create a dedicated terraform IAM user. On later option, select programmatic access on creation, under permissions select Attach existing policies directly and choose a policy which has the rights to create the infrastructure you intend to maintain. Note that, the policy AdministratorAccess has full access to all resources.

Preparation — Environment Variables

The Terraform AWS provider can be configured via HCL arguments or environment variables. To avoid storing credentials in the Git repository, we set the following pipeline environment variables for the gitlab-runner:

  • AWS_DEFAULT_REGION

  • AWS_ACCESS_KEY_ID

  • AWS_SECRET_ACCESS_KEY

These variables can be set in the Gitlab UI at: Settings -> CI / CD -> Variables

Code

main.tf in the root directory contains the following:

// main.tf  
provider "aws" { }  

resource "aws_ecr_repository" "nginx" {  
  name = "${var.environment}-nginx"  
}

For us to comply with Terraform best-practices, we create variables.tf in the root directory:

// variables.tf  
variable "environment" {}

The Terraform backend state is kept in a separate HCL file in the root directory:

// backend.tf  
terraform {  
  backend "s3" {  
    encrypt = true  
  }  
}

A config per environment in the sub-directory config/:

// config/production.tfvars  
environment = "production"

// config/staging.tfvars  
environment = "staging"

// config/sandbox.tfvars  
environment = "sandbox"

A backend config per environment in the sub-directory config/:

// config/production_backend.tfvars  
region         = "us-east-1"  
bucket         = "terraform-state-8ab67ec2"  
dynamodb_table = "terraform-state-8ab67ec2-lock"  
key            = "production/nginx.state"

// config/staging_backend.tfvars  
region         = "us-east-1"  
bucket         = "terraform-state-8ab67ec2"  
dynamodb_table = "terraform-state-8ab67ec2-lock"  
key            = "staging/nginx.state"

// config/sandbox_backend.tfvars  
region         = "us-east-1"  
bucket         = "terraform-state-8ab67ec2"  
dynamodb_table = "terraform-state-8ab67ec2-lock"  
key            = "sandbox/nginx.state"

And finally the Gitlab CI/CD pipeline .gitlab-ci.yml in the root directory:

# .gitlab-ci.yml  
stages:  
  - infrastructure  

.terraform:  
  stage: infrastructure  
  image:  
    name: hashicorp/terraform:light  
    entrypoint:  
      - '/usr/bin/env'  
      - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'      
  before_script:  
    - terraform init -backend-config=config/${ENVIRONMENT}_backend.tfvars  

# ~~~~~~~~~~~~~~~~ Apply ~~~~~~~~~~~~~~~~  
.terraform_apply:  
  extends: .terraform  
  script:  
    - terraform apply -auto-approve -var-file=config/${ENVIRONMENT}.tfvars  

terraform_apply_production:  
  extends: .terraform_apply  
  variables:  
    ENVIRONMENT: production  
  only:  
    refs:  
      - master  

terraform_apply_staging:  
  extends: .terraform_apply  
  variables:  
    ENVIRONMENT: staging  
  only:  
    refs:  
      - develop  

terraform_apply_sandbox:  
  extends: .terraform_apply  
  variables:  
    ENVIRONMENT: sandbox  
  except:  
    refs:  
      - master  
      - develop  

# ~~~~~~~~~~~~~~~~ Destroy ~~~~~~~~~~~~~~~~  
terraform_destroy:  
  extends: .terraform  
  script:  
    - terraform destroy -auto-approve -var-file=config/${ENVIRONMENT}.tfvars  
  only:  
    variables:  
      - $ENVIRONMENT  
      - $DESTROY

Some further Explanation

The command terraform init is executed with the backend config for the environment.

An additional job terraform_destroy allows removing the infrastructure created by Terraform. To enable this job, the pipeline variable ENVIRONMENT must be set to the environment name and DESTROY to true, manually via Gitlab UI at: CI / CD -> Run Pipeline

Note:

This pipeline expects to only ever have a single feature branch for the given Git repository! In my experience that’s good practice for infrastructure as code.

Example Pipeline #3: AWS multiple accounts (final)

Our previous pipeline deploys all Docker registries in the same AWS account, which is feasible for small setups. Larger cloud architectures should contain multiple environments in dedicated AWS accounts.

So for the third and final pipeline, we create one AWS Docker registry (ECR) per environment, each in a separate AWS sub-account.

Preparation — AWS accounts

If you don’t have sub-accounts for your workloads environments yet, create them using the AWS Organization in the AWS console or deploy the following Terraform snippet in your AWS master account:

resource "aws_organizations_account" "production" {  
  name  = "production"  
  email = "production+admin@foobar.io"  
}  

resource "aws_organizations_account" "staging" {  
  name  = "staging"  
  email = "staging+admin@foobar.io"  
}  

resource "aws_organizations_account" "sandbox" {  
  name  = "sandobx"  
  email = "sandbox+admin@foobar.io"  
}

Note: Replace the e-mail addresses with your own and use plus addressing in case your mail service supports it.

Make sure the AWS user (to which the access keys belong) has the rights to assume the role OrganizationAccountAccessRole in the target account. This is given if the policy AdministratorAccess was attached. The AWS docs provide more information how to access sub-accounts.

Code

main.tf in the root directory contains the following:

// main.tf  
provider "aws" {   
  assume_role {  
    role_arn = "arn:aws:iam::${var.account}:role/OrganizationAccountAccessRole"  
  }  
}  

resource "aws_ecr_repository" "nginx_hello" {  
  name = "${var.environment}-nginx-hello"  
}

Now, similar to Pipeline #2, we create variables.tf in the root directory:

// variables.tf  
variable "environment" {}  
variable "account" {}

A config per environment with the new account variable in the sub-directory config/:

// config/production.tfvars  
environment = "production"  
account     = "123456789"

// config/staging.tfvars  
environment = "staging"  
account     = "987654321"

// config/sandbox.tfvars  
environment = "sandbox"  
account     = "313374985"

The backend config in the sub-directory config/and Gitlab CI/CD pipeline .gitlab-ci.yml in the root directory are identical to the ones in Pipeline #2.

Further Explanation

The assume_role attribute of the AWS provider takes a role_arn which is assumed on access to your AWS account. We assemble the role_arn with account id provided via config variable ${var.account} and the role OrganizationAccountAccessRole which is created by default in all sub-accounts (but missing in the master account).

Setting role_arn effects in switching to the target account before creating the AWS resources.

The result of running this pipeline are three ECR registries, one in each sub-account production, staging and sandbox.

Optimization: Deploy production manually

The Gitlab pipeline explained above follows the continuous deployment principle and automatically deploys infrastructure changes to production on merge to the master branch. But for some use-cases or setups, an additional verification step is desired.

The following .gitlab-ci.yml jobs execute terraform plan for production and turn terraform apply into a manual job:

terraform_plan_production:  
  extends: .terraform  
  variables:  
    ENVIRONMENT: production  
  script:  
    - terraform plan -var-file=config/${ENVIRONMENT}.tfvars  
  except:  
    variables:  
      - $DESTROY  
  only:  
    refs:  
      - master  


terraform_apply_production:  
  extends: .terraform_apply  
  when: manual  
  variables:  
    ENVIRONMENT: production  
  only:  
    refs:  
      - master

The Conclusion

Applying the Gitflow concept to Terraform on Gitlab is a powerful and highly productive workflow to enable CI/CD for instrastructure-as-code and deploy immutable cloud infrastructure.

FAQ

  1. Why are the steps **terraform validate** and **terraform plan** skipped?
    validate and plan are executed as part of apply.

  2. There is a lot of redundant code in the pipeline. Why not use YAML anchors?
    YAML anchor decrease readability and make it harder for other engineers to follow the workflow.

That’s it folks, creating our Gitlab CI/CD pipeline with Terraform. For more on IAC, please read my other articles.

#YouAreAwesome #StayAwesome

Devopsci-cdGitLabTerraformInfrastructure as code