Toad World Blog

Continuous Integration from AWS Code Commit to Amazon ECR with AWS CodeBuild

Jul 19, 2017 11:00:00 PM by Deepak Vohra

Introduction

In an earlier article, Continuous Integration from AWS CodeCommit to Docker Hub with AWS CodeBuild, we discussed how Jenkins has some limitations as a build tool and how AWS CodeBuild overcomes those limitations. We discussed creating a Continuous Integration (CI) pipeline to build, package, and deliver a Docker image to Docker Hub, starting with source code in AWS CodeCommit.  In this article we shall discuss creating a CI pipeline to build, package, and deliver a Docker image to Amazon ECR (EC2 Container Registry) starting with source code on AWS CodeCommit.  This article has the following sections:

  • Setting the Environment
  • Creating an Amazon ECR Repository
  • Configuring buildspec.yaml
  • Creating a CodeCommit Project
  • Creating a CodeBuild Project
  • Running the CodeBuild
  • Deleting an ECR Repository

 

Setting the Environment

An AWS account is required to use AWS CodeCommit and AWS CodeBuild, and may be created at https://aws.amazon.com/. An Amazon ECR repo is also required to upload a built Docker image; we shall create a repo in this article.

 

Creating an Amazon ECR Repository

To create an EC2 Container Service click on Services>EC2 Container Service as shown in Figure 1.

2235.ec1

Figure 1. Services>EC2 Container Service

In Amazon ECS the Repositories navigation node is selected when the page gets navigated to. Click on Create repository as shown in Figure 2.

5050.ec2

Figure 2. Create repository

Specify a Repository name (nodeserver for example) as shown in Figure 3. Click on Next step.

3302.ec3

Figure 3. Configuring Repository

An Amazon ECR repository gets created, as indicated by the message in Figure 4.  The Docker tasks that may be subsequently performed and the commands for each of the tasks get listed. The first Docker task listed is to get the login command that can be used by a Docker client to authenticate to the registry.  The second task listed is to login to the registry using the command returned in 1. The third Docker command listed is to build a Docker image; all as shown below.

1731.ec4

Figure 4. EC2 Container Registry Created

The next Docker command listed is to tag the Docker image generated. The fifth Docker command listed is to push the Docker image to the ECR repo. We shall use these command when we configure a build spec file to run a CodeBuild. Click on Done as shown in Figure 5.

3806.ec5

Figure 5. Done

The repository created, nodeserver, gets listed in the Repositories page, as shown in Figure 6. Click on the repo link to configure permissions and list images in the repo.

6521.ec6

Figure 6. Repositories

The All repositories>nodeserver page gets displayed, as shown in Figure 7. The Images tab is selected by default and no images are listed initially.

4863.ec7

Figure 7. All repositories>nodeserver

Select the Permissions tab as shown in Figure 8.

8686.ec8

Figure 8. Permissions Tab

By default, only the user creating the repo has access to the repo. A policy document could be added to allow other users to access the repo. Click on Add to add a permission statement, as shown in Figure 9.

0753.ec9

Figure 9. Permissions>Add

Specify a Sid (uploadDockerImage for example) and select Principal as Everybody. The Effect is selected as Allow by default. In Actions select All actions. Click on Save all as shown in Figure 10.

5543.ec10

Figure 10. Saving Permission Statement

The repository policy gets updated and a Policy Document gets displayed, as shown in Figure 11.

 0257.ec11

Figure 11. Policy Document

 

Configuring buildspec.yml

A build spec (buildspec.yml file) is required for CodeBuild to convert input source code on CodeCommit into a build output, which would be a Docker image on Amazon ECR for this article.  A build spec consists of a collection of build commands and settings in YAML syntax, as was introduced in my earlier article, Continous Integration from AWS CodeCommit to Docker Hub with AWS CodeBuild . We shall use three phases: pre_build, build, and post_build. The tasks performed in each phase are shown in Figure 12.

0675.ec12-1

Figure 12. Build Phases

In the pre_build phase login to the Amazon ECR. In the build phase build the Docker image and tag the Docker image. In the post_build phase, push (upload) the Docker image to Amazon ECR. The buildspec.yml file used in this article is listed:

phases:
  build:
    commands:
      - "echo Build started on `date`"
      - "echo Building the Docker image..."
      - "docker build -t nodeserver ."
      - "docker tag nodeserver:latest 672593526685.dkr.ecr.us-east-1.amazonaws.com/nodeserver:latest"
  post_build:
    commands:
      - "echo Build completed on `date`"
      - "echo Pushing the Docker image..."
      - "docker push 672593526685.dkr.ecr.us-east-1.amazonaws.com/nodeserver:latest"
  pre_build:
    commands:
      - "echo Logging in to Amazon ECR..."
      - "$(aws ecr get-login --region us-east-1)"
version: 0.1

 

Creating a CodeCommit  Project

Create a CodeCommit repository (Node-Server) and set Up SSH Connections to AWS CodeCommit Repository as discussed in the Using the AWS CodeCommit Repository article. An IAM user must be created, which is also discussed. To be able to use AWS CodeCommit with AWS CodeBuild add the AWSCodeBuildAdminAccess permission to the IAM user.  If the CodeCommit repository Node-Server already exists delete the repository and create a new repository.

We shall use code for a Docker image from the GitHub repo for a Node server Docker image, which was introduced in the Using the AWS CodeCommit Repository article. Migrating the GitHub repo code to CodeCommit is a two-phase process.

  1. Clone the GitHub repo to a local repo.
  2. Push the local repo code to CodeCommit.

First, add the buildspec.yml file listed in the previous section of this article to the GitHub repo. The GitHub repo with the three files required to build a Docker image and upload the Docker image to EC2 Container Registry are shown in Figure 13. The YAML file should be of valid YAML syntax, which may be verified with YAML Lint (http://www.yamllint.com/ ).

3480.ec13

Figure 13. GitHub Repo

As discussed in Using the AWS CodeCommit Repository, run the following command to clone the GitHub repo to the local directory.

C:\DockerDevOps\AWS CodeCommit>git clone --mirror https://github.com/dvohra/docker-node-server.git  aws-codecommit-docker-node-server

As the output in Figure 14 indicates, the GitHub repo gets cloned to local directory.

5556.ec14

Figure 14. Cloning GitHub Repo to Local Repo

Using the CodeCommit repo clone url ssh://git-codecommit.us-east-1.amazonaws.com/v1/repos/Node-Server run the following git pushcommand to upload local repo code to CodeCommit repo.

git push ssh://APKAJQW7CCERK6FNZQOA@git-codecommit.us-east-1.amazonaws.com/v1/repos/Node-Server --all

The local repo gets migrated to CodeCommit repo, as shown in Figure 15.

8270.ec15

Figure 15. Migrating Local Repo to CodeCommit Repo

The three files buildspec.yml, Dockerfile, and server.js get added to the CodeCommit repo Node-Server, as shown in Figure 16.

2804.ec16

Figure 16. CodeCommit Repo with Source code files

Click on the buildspec.yml file and the build spec gets displayed, as shown in Figure 17.

4061.ec17

Figure 17. buildspec.yml

 

Creating a CodeBuild Project

In this section we shall create an AWS CodeBuild project to build the source code in the CodeCommit into a Docker image, and subsequently upload the Docker image to Amazon ECR. Click on Services>Developer Tools>CodeBuild as shown in Figure 18.

3312.ec18

Figure 18. Services>Developer Tools>CodeBuild

In the AWS CodeBuild>Build projects page click on Create project as shown in Figure 19.

0741.ec19

Figure 19. AWS CodeBuild>Create project

Specify a Project name (Node-Server) and in Source: What to build select AWS CodeCommit as shown in Figure 20. The other options are Amazon S3and GitHub.

8203.ec20

Figure 20. Configuring CodeBuild Project

Select the Node-Server repository in CodeCommit as shown in Figure 21.

5531.ec21

Figure 21. Selecting AWS CodeCommit Repository

In Environment image keep the default setting of Use an image managed by AWS CodeBuild. Select Operating system as Ubuntu and Runtime as Docker. Select Docker Version as 1.11.2 or 1.12. Keep the default setting for Build specification, which is Use the buildspec.yml in the source code root directory. In Artifact type select No artifacts as shown in Figure 22. The artifact type does not generate any artifact and is used to upload a Docker image to Amazon ECR or Docker Hub. The other option is Amazon S3.

4073.ec22

Figure 22. Selecting Artifact type

In Service role select Create a service role in your account as shown in Figure 23. The role name is auto-generated as in the Role name field. Click on Show advanced settings.

5140.ec23

Figure 23. Selecting Service Role Option to create a Role

The advanced settings are optional to configure and the default should suffice for most builds. Compute type could be increased if required; the default being 3 GB memory and 2 vCPU, as shown in Figure 24.

0763.ec24

Figure 24. Setting Advanced Settings

Click on Continue as shown in Figure 25.

1830.ec25

Figure 25. Continue

The Summary of CodeBuild gets displayed. Click on Save as shown in Figure 26.

5554.ec26

Figure 26. Saving CodeBuild Configuration

A new CodeBuild called Node-Server gets generated, as shown in Figure 27.

3225.ec27

Figure 27. CodeBuild Project Node-Server

We need to modify the default service role to add permission to access ECR. Select Services>IAM. Select Roles. Click on the codebuild-Node-Server-service-role as shown in Figure 28.

6840.ec28

Figure 28. codebuild-Node-Server-service-role Role

To add a permission click on Attach Policy as shown in Figure 29.

4382.ec29

Figure 29. Attach Policy

In the Policy Type Filter specify “AmazonEC2ContainerRegistryFullAccess” as shown in Figure 30. The AmazonEC2ContainerRegistryFullAccess policy gets displayed. Select the policy.

2620.ec30

Figure 30. Filtering and Selecting AmazonEC2ContainerRegistryFullAccess policy

The policy gets added to the service role, as shown in Figure 31.

0172.ec31

Figure 31. AmazonEC2ContainerRegistryFullAccess policy

If required the CodeBuild project may be updated. Select the project in the Build projects list and select Actions>Update as shown in Figure 32.

7510.ec32

Figure 32. Actions>Update

Click on Update after modifying any settings as shown in Figure 33.

5852.ec33

Figure 33. Update

 

Running the CodeBuild

In this section we shall run the CodeBuild project. Click on Start build to start a build as shown in Figure 34.

0576.ec34

Figure 34. Start build

The start new build page gets displayed with a summary of the build, as shown in Figure 35. The build has not started yet.

7028.ec35

Figure 35. Summary of CodeBuild

Scroll down and click on Start build to start the build as shown in Figure 36.

4466.ec36

Figure 36. Start build

The build gets started and an InProgress message gets displayed as shown in Figure 37.

3718.ec37

Figure 37. Build In Progress

Though not required, click on the refresh icon button to refresh the build as shown in Figure 38.  The Phase Details section lists the status of the phases. SUBMITTED is listed as Succeeded and Provisioning is listed as In Progress.

6433.ec38

Figure 38. Phase Details

When the complete build succeeds the message Succeeded gets displayed for the build as shown in Figure 39.

0247.ec39

Figure 39. Build Succeeded

The Phase Details should list all phases as Succeeded as shown in Figure 40.

2313.ec440

Figure 40. All Phases Succeeded

The Build logs display the logs for the different phases, as shown in Figure 41.

1665.ec41

Figure 41. Build Logs

After a build has completed successfully a Docker image gets uploaded to Amazon ECR. The Docker image uploaded most recently has the “latest” tag, as shown in Figure 42.

4370.ec42

Figure 42. Docker Image with tag “latest”

If the build is run again another Docker image gets added and gets the “latest” tag. Any Docker image/s added previously get listed without a tag, as shown in Figure 43.

7103.ec43

Figure 43. Listing Docker Images in ECR Repository

If an error occurs during a build the Build logs display the error message. as shown in Figure 44.

0160.ec44

Figure 44. Error Message in Logs

The build status gets listed as Failed, as shown in Figure 45.

3884.ec45

Figure 45. Build Status Failed

If  a CodeBuild project build fails the project may be modified with Edit project as shown in Figure 46, and a build started again.

2235.ec46

Figure 46. Edit build

 

Deleting an ECR Repository

To delete the Amazon ECR repository click on Delete repository as shown in Figure 47.

3302.ec47

Figure 47. Repositories>Delete Repository

In the confirmation dialog click on Delete as shown in Figure 48.

7026.ec48

Figure 48. Delete repository

The repository gets deleted, as shown in Figure 49.

8182.ec49

Figure 49. ECR Repository Deleted

 

Summary

In this article we discussed creating a CodeBuild project to build a Docker image from source code in CodeCommit. The build spec file buildspec.ymlspecified the commands to run and is included with the source code in the CodeCommit. The Docker image generated is uploaded to Amazon ECR by the CodeBuild.

Tags: MySQL

Deepak Vohra

Written by Deepak Vohra

Deepak Vohra is an Oracle Certified Associate, Oracle Database 10g, and Sun Certified Java Programmer. Deepak has published on OTN and in Oracle Magazine.