Building an IaC pipeline on AWS with fully integrated security

Translation of the article was prepared on the eve of the start of the course “Infrastructure as a code in Ansible”…

We also want to recommend you super-intensive “IaC Ansible” in 5 days of which you will learn how to make rolling out the code a simple and pleasant process in the spirit of “press one button”, make the setup of a new machine fully automated, automatically keep up-to-date all checks and triggers in monitoring and learn about problems even before they come, and much other.

In my last article (link) We looked at the Infrastructure as Code (IaC) governance model, its importance, why many companies use it, and how you can improve the security of the IaC pipeline.

It is known that developers of cloud systems often work in conditions of strong pressure, when the deadlines are tight and the result must be delivered quickly. Because of this, they often do not pay enough attention to coding and configuration. And lack of proper configurations is a source of security holes. It follows that it is imperative to rely on design best practices throughout the entire architecture development process, even when deadlines are very tight.

Now let’s look at a real-world example of how you can automatically validate the correctness of development practices in IaC pipelines for AWS through hundreds of rule-compliance checks. AWS Well-Architected Framework (safety, cost optimization, productivity, high professional standards and reliability) and other standards.

Here’s a general idea of ​​what we’ll be working on:

Initial Requirements

To form the environment we need, you will need the following elements:

• Wednesday VSCode IDEinstalled on the computer → download link…

• Installed plug-in Cloud One – Conformity in VSCode environment -> link…

• Cloud One Account – Conformity → register and get a free trial…

• AWS account → register if you don’t have it…

Let’s get started building your first Infrastructure as Code (IaC) pipeline, checking out the right development practices!

Phase 1: Installing the security plug-in in the programming environment and obtaining an API token for scanning CloudFormation templates

After installing the VSCode IDE, remember to also install the Cloud Conformity Security Plugin as shown here:

• VSCode Marketplace – Cloud Conformity Plugin – LINK

Create your account with Cloud One – Conformity using the link provided here (Create an account), enter it and generate an API token.

In Cloud Conformity: click Username top right and select → User Settings → API Keys → New API Keyto create an API key to be used in the VSCode plugin. Be sure to copy the key and keep it in a safe place. It is impossible to get it again.

Copy your API key and return to VSCode

1. Click the extensions icon (left) and click Extension Settings ⚙️ to write the Cloud Conformity Template Scanner.

2. In the Cloud Conformity environment, select Edit in settings.json… Go to the ApiKey section.

3. Enter the API key generated in the previous step and save the changes.

CloudFormation templates can now be scanned using hundreds of rule compliance checks AWS Well-Architected Framework and other standards, guaranteeing the superior quality of the developed cloud infrastructure.

Here’s an example of a CloudFormation template that doesn’t follow development best practices that you can test:

To test the extension in action, open the above CloudFormation template with VSCode, then open the command palette by clicking:

• macOS: ⇧ + ⌘ + P

• Windows / Linux: Ctrl + Shift + P

Find Cloud One Conformity: Scan the Current Open Template and press , which will automatically scan this CloudFormation template:

The scan result will appear on a second tab titled Scan Result, as shown in the image below. You can also use cloud knowledge baseTo help you better understand the recommended practices violations found and how to fix them in your CloudFormation template or production environments:

Excellent, the first phase of IaC security automation is complete.

Step 2. Build a CI / CD pipeline using AWS and then integrate the Conformity pattern scanner into it

In the first step, we showed how you can perform a scan before pushing a new version of the code to the repository. However, sometimes a developer may forget to do this and save a CFT template that has problems. We will now create a Security Gateway to help prevent new resources from being created that do not match your company’s recommended development practices.

With this gateway, you can track and audit all changes in real time and only then deploy to AWS.

In this section, we will show you how to use a pattern scanner in a CI / CD pipeline with AWS CodeCommit, AWS CodeBuild and AWS CodePipeline… Let’s get started.

CodeCommit will be used as a repository to host our code. Many people and companies around the world use GitHub, GitLab and BitBucket for this. For your VSCode to be able to move code to CodeCommit, you need to tweak it.

Creating a Git Repository on AWS CodeCommit

• Create an IAM user with permission to use CodeCommit and access only with an SSH key. (For Git permission information and guidelines, see this link from AWS.)

Go to Security credentials → HTTPS Git credentials for AWS CodeCommit to generate credentials.

Download the credentials and save them to a safe place.

Learn more about this procedure on the AWS website → LINK…

• Create a repository AWS CodeCommit…

You can run the git clone command on your computer and easily transfer git config to it.

git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/{ИМЯ ВАШЕГО РЕПОЗИТОРИЯ}

You can now use your own CloudFormation template, or take the bad CloudFormation template I provided earlier to submit to AWS CodeCommit.

git add .

git commit -m "First Commit"

git push

So your first save to AWS CodeCommit is complete. Now let’s move on to AWS CodeBuild and AWS CodePipeline.

Building CodeBuild to Automatically Scan CloudFormation Templates

AWS CodeBuild’s solution is very similar to GitHub Actions, Azure DevOps, and GitLab. It is a CI / CD technology with which you can create new projects, automate processes, and deploy new applications or infrastructures.

We will use CodeBuild with a specific container image to run the Conformity template scanner to identify potential issues before deploying a new IaC to production.

• Creating a Build Project in AWS CodeBuild

I’ll be using the standard AWS image, but you can take a different management image or create your own for this type of automation.

Here is the configuration for the image below:

Environment image: Managed Image
Operationg System: Amazon Linux 2
Runtime: StandardImage: aws/codebuild/amazonlinux2-x8664-standard:3.0
Image version: Always use the latest
Environment type: Linux
Service Role: New
Role Name: <NEW ROLE NAME>

Here is the configuration for the image below:

Environment Variables
CC_API_KEY = <YOUR API KEY FROM CLOUD ONE - CONFORMITY>
CC_REGION = <REGION SELECTED TO CREATE YOUR CONFORMITY TENANT>
CC_RISK_LEVEL = <RISK LEVEL NOT BE ACCEPT>
CFN_TEMPLATE_FILE_LOCATION = <YOUR CLOUDFORMATION TEMPLATE PATH>
STACK_NAME = <THE CLOUDFORMATION STACK NAME

Here is the configuration for the image below:

Build specifications: Insert build commands
Build Commands:version: 0.2phases:
    install:
        runtime-versions:
            python: 3.7
    pre_build:
        commands:
            - pip3 install awscli --upgrade --user
    build:
        commands:
            - pip3 install -r https://raw.githubusercontent.com/OzNetNerd/Cloud-Conformity-Pipeline-Scanner/master/requirements.txt
            - wget https://raw.githubusercontent.com/OzNetNerd/Cloud-Conformity-Pipeline-Scanner/master/src/scanner.py
            - CC_API_KEY=`jq -r '.CC_API_KEY' <<< $CC_API_KEY`
            - python3 scanner.py
            
    post_build:
        commands:
            - aws cloudformation deploy --template-file $CFN_TEMPLATE_FILE_LOCATION --stack-name $STACK_NAME --no-fail-on-empty-changese

Link to Buildspec.yml on GitHub – https://raw.githubusercontent.com/fernandostc/IaC-Security-Automation/master/buildspec.yml

Note… The template scanner I’m using in this example was created by Will Robison, a solution architect at Trend Micro – here’s a link to this project on GitHub: https://github.com/OzNetNerd/Cloud-Conformity-Pipeline-Scanner

Now click Create build project to complete the process.

Create a private key in Secret Manager to store the Cloud One API key – Conformity

Now click Store to complete the process.

Important! Don’t forget to create a policy to give your Role Service in CodeBuild the following permission, with which CodeBuild can get a value from SecretManager:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "secretsmanager:GetSecretValue",
            "Resource": "arn:aws:secretsmanager:us-east-1:<AWS Account ID>:secret:<Secret Key ARN>"
        }
    ]
}

Build AWS CodePipeline to Automate a Template Scan Trigger

AWS CodePipeline can help automate the process of running the template scanner after every IaC code update. You can create multiple stages in CodePipeline, but our example is simple, one stage is enough for it. It is very easy to understand.

Let’s create a new pipeline in AWS CodePipeline

You can name the pipeline and create a new service role if you have not had any service roles before.

Now you need to set the following parameters, taking into account the previously created CodeCommit:

Source Provider (Поставщик исходного кода): <AWS CodeCommit>Repository Name (Имя репозитория): <CloudOneConformity или заданное вами ранее имя репозитория>Branch Name (Имя ветви): <master>

Now you need to set information about the build process:

Build provider (Поставщик построения): <AWS CodeBuild>Project Name (Имя проекта): <IaC-Security-Automation или заданное вами ранее имя репозитория>

In this case, we do not plan to use the deployment commands, since we are deploying inside CodeBuild within one of the stages. You can click Skip the deploy stage.

You can view the configuration and click Create pipeline.

Automated pipeline testing

1. Create your own CloudFormation template or use this one: link…

2. Scan the CloudFormation template using the Cloud One – Conformity plug-in and check the result.

Make sure there are some issues that need to be resolved before saving to production.

3. The template can be submitted to the repository to test its functionality in the pipeline. This is a kind of test before fixing all the problems found in the CloudFormation template.

4. Save the new code to CodeCommit:

git add .
git commit -m "Test Automation"
git push

5. Saving the new code to CodeCommit will launch CodePipeline.

Once the automation process is complete, you will see both steps completed successfully.

NOTE. If you have any problems, check the build process. Go to the latest build and view its logs. They are very useful for debugging in case of difficulty.

Conclusion

So, you have a fully automated IaC pipeline with AWS CodeCommit, AWS CodeBuild, AWS CodePipeline, and Cloud One – Conformity, allowing you to analyze all kinds of configuration deviations during the CloudFormation templating process.

Hopefully, this article will help you and your team stay on top of architectural best practices during development and effectively work on innovative cloud solutions under tight deadlines. If you need more information, be sure to contact me!

Acknowledgments

I want to say a BIG THANKS to some of the people who helped me make this article better with their amazing reviews:

• Raphael Bottino (Raphael Bottino)

• Melissa Clow (Melissa Clow)

If this article was useful to you, press the up arrow and write comments.

– Learn more about the course“Infrastructure as a code in Ansible”…

– To sign up for super-intensive “IaC Ansible”