How to Track Multiple Git Branches in AWS CodePipeline

CodePipeline is Amazon Web Services’s (AWS’s) Continuous Integration / Continuous Deployment (CI/CD) pipeline service. In a nutshell, that means it orchestrates all of the necessary build, test, approval and deployment steps necessary to take code from source to production. While not as mature as some of its competitors, such as Jenkins, GitLab and Travis CI (to name a few), it still has many redeeming qualities due to its tight integration with other services in the AWS ecosystem.

NEW RESEARCH: LEARN HOW DECISION-MAKERS ARE PRIORITIZING DIGITAL INITIATIVES IN 2024.

Like most AWS services, CodePipeline can take a bit of configuration and tinkering to get working properly. One challenge that I ran into recently was setting up CodePipeline to work with multiple git branches. I tend to follow the GitFlow branching model and was scratching my head on how to get CodePipeline to work on all of my feature and release branches.

The way to make it work depends on your git repository. If you are using GitHub or Bitbucket, you will have to create an AWS CodeBuild project to watch the repository, build a zip, and upload the artifact to Simple Storage Service (S3). If you are using AWS CodeCommit, you will have to create a Lambda trigger that creates a new pipeline for each branch.

GitHub or Bitbucket Approach

Even though CodePipeline supports GitHub as a source, it only works for one specific branch.

A CodePipeline source is something that starts the pipeline automagically and does not require an input artifact. At the time of writing this article, the only options for a source are: AWS CodeCommit (for one specific branch), AWS ECR, AWS S3, and GitHub (for one specific branch).

Fortunately, there is an alternative method to make it work:

1. Create an S3 bucket
2. Use S3 as the CodePipeline source
3. Connect AWS to GitHub or Bitbucket
4. Create a CodeBuild project to watch your repository

1. Create an S3 Bucket

Step one is to create an S3 bucket to store your repository source artifacts.

Directions

1. 1. Save template.s3.yml to a working directory
      • Note: template.s3.yml includes:
        • Versioning – Versioning is mandatory for a CodePipeline S3 source
      • Source: https://github.com/theneverstill/blog-post-how-to-track-multiple-branches-in-aws-codepipeline/blob/master/github-or-bitbucket-approach/1-create-an-s3-bucket/template.s3.yml
   2. Execute the following Terminal command in your working directory

aws cloudformation deploy \
  --template-file template.s3.yml \
  --stack-name example-s3 \
  --parameter-overrides S3BucketNameArtifacts=artifacts.example.com

You are encouraged to change the Terminal command --stack-name value (example-s3) and the --parameter-overrides value for S3BucketNameArtifacts (artifacts.example.com). However, ensure that you use the same parameter overrides in all of the steps.

2. Use S3 as the CodePipeline Source

Step two is to use the S3 bucket you created in step one as the source for your pipeline.

Directions

1. 1. Save template.codepipeline.yml to a working directory
      • Note: template.codepipeline.yml includes:
        • A manual approval stage (and associated action)
          • Pipelines must contain at least two stages
        • An IAM role for CodePipeline
          • You should use the minimum required permissions for your pipeline (i.e not administrator)
      • Source: https://github.com/theneverstill/blog-post-how-to-track-multiple-branches-in-aws-codepipeline/blob/master/github-or-bitbucket-approach/2-use-s3-as-the-codepipeline-source/template.codepipeline.yml
   2. Execute the following Terminal command in your working directory

aws cloudformation deploy \
  --template-file template.codepipeline.yml \
  --stack-name example-codepipeline \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides S3BucketNameArtifacts=artifacts.example.com S3KeyNameSource=source.zip

You are encouraged to change the Terminal command --stack-name value (example-codepipeline) and the --parameter-overrides values for S3BucketNameArtifacts (artifacts.example.com) and S3KeyNameSource (source.zip). However, ensure that you use the same parameter overrides in all of the steps.

3. Connect AWS To GitHub or Bitbucket

Step three is to connect AWS to GitHub or Bitbucket via OAuth.

Directions

1. Sign into the AWS web console
2. Navigate to CodeBuild (New Console Experience | Old Console Experience)
3. Click the button to start creating a new build project
4. Scroll down to the Source section
5. Select [GitHub | Bitbucket] from the Source Provider dropdown
6. Click the button to Connect to [GitHub | Bitbucket]
7. Authenticate in the newly opened window
8. Once authenticated, you can abandon the new build project

4. Create a CodeBuild Project

Step four is to create a CodeBuild project to watch your GitHub or Bitbucket repository.

Directions

1. 1. Save template.codebuild.yml to a working directory
      • Note: template.codepipeline.yml includes:
        • An Ubuntu 14.04 base image
          • Any AWS managed Ubuntu image should work
        • An IAM role for CodeBuild
          • You should use the minimum required permissions for your pipeline (i.e not administrator)
      • Source: https://github.com/theneverstill/blog-post-how-to-track-multiple-branches-in-aws-codepipeline/blob/master/github-or-bitbucket-approach/4-create-a-codebuild-project/template.codebuild.yml
   2. Execute the following Terminal command in your working directory

aws cloudformation deploy \
  --template-file template.codebuild.yml \
  --stack-name example-codebuild \
  --capabilities CAPABILITY_IAM \
  --parameter-overrides S3BucketNameArtifacts=artifacts.example.com S3KeyNameSource=source.zip RepositoryType=GITHUB RepositoryLocation=https://github.com/example/example.git

1. Save, stage and commit buildspec.yml to your GitHub or Bitbucket repository
   • Source: https://github.com/theneverstill/blog-post-how-to-track-multiple-branches-in-aws-codepipeline/blob/master/github-or-bitbucket-approach/4-create-a-codebuild-project/buildspec.yml

The artifacts section of codespec.yml controls which files will be zipped into the source artifact.

You are encouraged to change the Terminal command --stack-name value (example-codebuild) and the --parameter-overrides values for S3BucketNameArtifacts (artifacts.example.com), S3KeyNameSource (source.zip), RepositoryType (GITHUB) and RepositoryLocation (https://github.com/example/example.git). However, ensure that you use the same parameter overrides in all of the steps.

After template.codebuild.yml has been deployed, every push to your repository on any branch (that contains codespec.yml) will trigger a build.

Gotchas

CodePipeline environment variables do not contain any information from git (e.g. the branch name or commit hash) since the source is S3. You will have to preserve this information on your own (e.g. via exporting necessary git information to a file during the source build).

You must have the requisite permissions to the repository in order to create a webhook (i.e. it will not work for public repositories for which you lack the requisite permissions).

CodeCommit Approach

Using CodeCommit as your git repository affords you with a couple of options for configuring CodePipeline to work with multiple branches:

1. Use an approach similar to the GitHub or Bitbucket flow detailed above; however, you create a Lambda trigger (instead of a webhook) on CodeCommit to start the CodeBuild project
   • This approach is out of scope for this article, but you can read the following approach to learn how to setup Lambda triggers
2. Create a Lambda trigger that will create a new pipeline whenever you create a new branch (and subsequently delete the pipeline when you delete the branch)
   • This approach creates multiple pipelines. While this does provide increased visibility and greater granularity, it also means higher complexity.
   • You could make this approach work for GitHub or Bitbucket if you wanted a new pipeline for each branch. You would have to follow the GitHub or Bitbucket flow detailed above, but deploy the pipeline CloudFormation stack in the webhook-triggered CodeBuild project instead of generating a source artifact.

Directions

1. 1. Save lambda.js, package.json, template.base.yml and deploy.sh to a working directory
      • Note: the files include:
        • IAM roles
          • You should use the minimum required permissions for your pipeline (i.e not administrator)
      • • CloudFormation stack names and parameter override values
          • You are encouraged to change these; however, ensure that you use the same parameter override values across all of the files
      • lambda.js source: https://github.com/theneverstill/blog-post-how-to-track-multiple-branches-in-aws-codepipeline/blob/master/codecommit-approach/lambda.js
      • package.json source: https://github.com/theneverstill/blog-post-how-to-track-multiple-branches-in-aws-codepipeline/blob/master/codecommit-approach/package.json
      • template.base.yml source: https://github.com/theneverstill/blog-post-how-to-track-multiple-branches-in-aws-codepipeline/blob/master/codecommit-approach/template.base.yml
      • deploy.sh source: https://github.com/theneverstill/blog-post-how-to-track-multiple-branches-in-aws-codepipeline/blob/master/codecommit-approach/deploy.sh
   2. Execute the following Terminal command in your working directory

bash ./deploy.sh

1. Save, stage and commit template.pipeline.yml to the newly created CodeCommit repository (or your own CodeCommit repository if you updated the files accordingly)
   • template.pipeline.yml source: https://github.com/theneverstill/blog-post-how-to-track-multiple-branches-in-aws-codepipeline/blob/master/codecommit-approach/template.pipeline.yml
2. Push a new branch to your repository to trigger the creation of a new pipeline.

When you run the Terminal command a new repository named example-repository will be created. You are encouraged to update the files accordingly if you want to use an existing CodeCommit repository.

When a branch is created, Lambda will deploy the version of template.pipeline.yml in that branch. A pipeline will not be deployed if this file is missing from the branch.

If any branches already exist in the repository when you create the trigger, you must manually deploy the template.pipeline.yml CloudFormation stacks for those branches.

Gotchas

At the time of writing this article, the version of the JavaScript AWS SDK bundled with the Lambda Node.js 8.10 runtime is 2.290.0. CodeCommit.getFile() was added in version 2.324.0. Therefore, if you want to use that function (and many others), you must bundle a later version of the JavaScript AWS SDK into your Lambda function package.

At the time of writing this article, some events in the CodeCommit web console do not trigger Lambda (e.g. creating and deleting branches). You should only create and delete git branches remotely (e.g. via the terminal).