Run the Job By the Help Devtron 😎 (Job 👉 Testing and Deployment the CloudFormation Script)

Test and Stimulate the Cloud-formation template with Minimal Cost

Problem Statement👀

CF template has a major dependency on AWS cloud for testing, to test CF Cloud engineer launches the CF stack, which results in high cloud cost. We are looking for a solution that can optimize the run/test for the CF template.

(Use EKS open-source cloud-formation template and implement a solution to test the CF template without launching the EKS cluster)

There are a Number of Ways to optimize the cost for testing of the Cloud-Formation Script working perfectly or not: -

→ LocalStack (Local AWS Cloud Stack)

LocalStack

LocalStack provides an easy-to-use test/mocking framework for developing Cloud applications. It spins up a testing environment on your local machine that provides the same functionality and APIs as the real AWS cloud environment.

→ Syntax or Typos Validation of Cloud-Formation
There are two tools that I know can be used to check the template or Syntax for YAML or JSON.

1. Validate-template (aws-cli)
   Cloud Formation first checks if the template is valid JSON. If it isn’t, Then Cloud Formation checks if the template is valid YAML. If both these checks fail, Then Cloud Formation returns a template validation error.
2. AWS CloudFormation Linter

cfn-lint

Validate AWS CloudFormation YAML/JSON templates against the AWS CloudFormation Resource Specification and additional checks. Includes checking valid values for resource properties and best practices

Practical Implementation

So, Firstly I am Going to Download the EKS cloud-formation scripts from the given Github link then I will start testing it out with the tools I mentioned above where I will go with step by step link from installation to the use of that tool.

Here For CI/CD I am going to use the Devtron.

At last, I will show How we can also able to integrate the CI/CD in such a way that the Cloud Engineer will first write the Cloud Formation script and then push it on the test branch of the repository from where the Jenkins gets the update afterward Jenkins will test Cloud-Formation template. If it is correct then it will be pushed to the main branch which is directly deploying this CloudFormation Script and if that script has some mistake then it directly notifies the Cloud Engineer Email.

Started Testing manually

→ Let’s Download the Templates First
Basically, For this I am going to create a python script through which I will download selected Cloud-Formation files So, I need do not need to give commands every time.

Validate-template

Let’s Start With Our First Tool i.e. Validate Template, basically, it’s come with aws cli So, we need to first configure the aws-cli then we can able to perform the task of validation.

Configure aws-cli

Now , Configure aws-cli by

$ aws configure

Then start using it.

Let’s Try to Validate a Template

As You Know we are using the validating template that will come under the cloud Formation service because it is made for that only to validate the cloud Formation templates.

What are the options? (How we can be able to pass our file? )
Take the help of command only (manual about that feature):-

So, with the help of — template-body, we can able to pass our template file which is locally present in our machine.

→template-body file://yourTemplateFileName.yml

If our template resides in an Amazon Simple Storage Service Bucket, then by the help of –template-url we can able to give the complete path.

→ template-url https://s3.amazonaws.com/cloudformation-templates-ap-south-1/yourTemplateFileName.yml

What is a Formats type of Output it will provide?

Let's try to test it out with the help of a template
I am going to make a mistake in the syntax of that YAML file.

I did a mistake around 37 line numbers.

Let’s Resolve that issue and then again try to validate it.

AWS CloudFormation Linter

Installation
Clone Repository

Now, We are able to successfully configure the cfn-lint.

So, with the — help we got to know about the more options by which we can able to validate our cloud-Formation template.

Here, we can also able to pass multiple templates to validate all in a single go. Like in some regions some services are not available so bypassing the region we can able validate our template such that is this template will work in that particular region or not.

Let’s try test it on our template which have some mistakes;

After Correction

LocalStack

It allows you to develop and test our application locally. Basically, with the help of this tool we can able to set up the whole-complete big architecture of AWS in our Local Machine by the help of APIs of LocalStack we can dev/test our application.
So, as I mentioned that the whole Architecture in your Local machine, then services are also available which states that CloudFormation service is also provided them and even according to our we are testing the CloudFormation Templates of EKS. So, EKS service is also available here.
Perfect, then we can test it in such a way that it will not only validate the syntax CloudFormation can able to deploy the resources actually on our local machine so that we can able to test and stimulate the Cloudformation template with zero cost.

But it has Pricing Model where; 
→ They provide service free (Open Source — Community Edition)  But Only Core Services this edition. 
→ Pro Edition (which cost around € 15/Month and in this edition they also provide 14 days of free trials. 
→ Enterprise Edition (For Which You need to contact them for more information about this Edition)

→ Before Starting with testing let's first look at the services which we are going in and which pricing they lie.

So, As you can see that CloudFormation Comes under the Community Addition (so its free)

But EKS comes under the Pro Edition. 😩
But No issue it provides the 14 days free trial let’s use it.

(Note:- According to the testing of these templates we launch the Cloud Formation stack which results in a high cost in comparison to that this € 15 / month is negligible we can able to perform a number of tests and try to simulate the environment this not only can be used for this cloud Formation testing it can help us to a number of different testing like Application Deployment, etc.)

For Getting used the API of other extended AWS features let's signup and first get the API key:-

→ https://app.localstack.cloud/sign-up

→ Then we will directly land on the dashboard — https://app.localstack.cloud/dashboard

→ As You Can see it doesn’t have the service EKS because its a Community Edition

Configuration Part

There are many ways to setup or configure it :-

→ LocalStack CLI

The easiest way to start and manage LocalStack — either on your machine, in a Docker container on your machine, or even on a remote Docker host.

→ LocalStack Cockpit

Get a desktop experience and work with your local LocalStack instance via the UI.

→ Docker

Use the docker CLI to manually start the LocalStack Docker container

→ Docker-Compose

Use docker-compose to configure and start your LocalStack Docker container

→ Helm

Use helm to create a LocalStack deployment in a Kubernetes cluster.

Let’s Configure it by the help of LocalStack CLI

First, install all these requirements

Now configure the Docker because all the APIs are available in that docker image running the container required the container engine so let’s set up the docker:-

Give the Complete Power of Docker Command to Ubuntu (So, that LocalStack can able to run containers by their image on behalf of Ubuntu)

Now Start the Service of Docker

The environment for the Local Stack is Ready lets setup the LocalStack 

Finally, The LocalStack Installation is Completed.

Before Starting the LocalStart let's also provide the pro edition access with the help of the API key so that we can able to perform our task.

Start the LocalStack

As you can see a container also started where its API is exposed at 4566 port number

Now, SetUp is Completed But the Question is;

How we can able to make requests to these APIs so that we can able to perform the tasks there should be some interfaces?
Yes, there the two ways through which we can able to perform tasks or who can able to make requests to those APIs
— Either we can use aws-cli or was local

Let's Start we aws-cli

Before using any command need to configure the aws-cli So, configure it with the help of some dummy Access Key and Secret Key (Because we are not actually to talk with aws API)

For using this LocalStack API you always need to provide — endpoint-url (i.e. http://localhost:4566) (I already told you that these API’s are in those containers)

→ Whenever you make a request to these APIs they send those data to that dashboard which is provided by the localstack after signup

List the Stack or Any Kinesis service (is there any stack present or created recently)

Now let’s try to check it with the help of a CloudFormation Template.

Try with the wrong template

So, that stack is also not created (Only that last correct deployment is no new deployment is created)

→ We can also able to set up the awslocal So, that we need not specify –endpoint-url every time.

Dashboard

Now From this practical, I got an idea of How I can able to create a Pipeline! Let’s Try To Implement this Interesting Concept;

Creating CI/CD pipeline for Testing and Deploying the Cloud-Formation Template

→ For the Configuration part of Devtron you can refer to this blog.

Now, in case you can use any one of the tools I have already given a detailed introduction about them with detailed implementation.
But in my case, I am going to use my favorite tool i.e. LocalStack

— — Let’s First Try Write the Steps which we need to perform with Devtron So, that it will make our Task Easy — —

Step 1: Setup the GitOps Part (WebHook → trigger → automatic)
Step 2: Whenever there is any commit in the “ test ” branch  Clone that Repo (also makes it important that who so ever push the code must mention their email inside a file and then make the push request to the “test” branch. So, that we can able to send the email if there is an error)
Step 3: Start testing and redirected the output of that command in a file. Step 4:  If error then sends the mail to that cloud engineer and abort the further steps.  If no error  then send a mail to that cloud engineer that your cloud formation template is perfectly fine now pushing it to the main/master branch ( then the job of the main/master branch will start)

Note: FOR STEP 4 WE WILL CREATE A PYTHON SCRIPT
For Script you can refer to my GitHub Repo.

Now, For Complete Practical You Can Watch it on Youtube.

References :
Documentation: https://docs.devtron.ai/
Githu Repo: https://github.com/Shashwatsingh22/CloudFormation-Script-CI-CD/tree/test-by-dev

If this blog helps you gain some knowledge then Please Clap 👏 for it.

and Follow me for such Content 👋

Support me if you like my work! Buy me a coffee

Shashwat Singh is creating the educational content On DevOps and Backend Tools.

— — — — — — — * — — — — — — -* — — — — -* — — — — —* — — — — -

The above blog is submitted under ‘Blogathon Contest 2022’ conducted by
Devtron (Website Hyperlink here: https://devtron.ai/)

Check out their Github repo: here(Github Hyperlink here: https://github.com/devtron-labs/devtron)

Follow them on: LinkedIn(Hyperlink here: https://www.linkedin.com/company/devtron-labs/),

Twitter(Hyperlink here: https://twitter.com/DevtronL)