Run the Job By the Help Devtron ๐ (Job ๐ Testing and Deployment the CloudFormation Script)
Test and Stimulate the Cloud-formation template with Minimal Cost
Problem Statement๐
CF template has a major dependency on AWS cloud for testing, to test CF Cloud engineer launches the CF stack, which results in high cloud cost. We are looking for a solution that can optimize the run/test for the CF template.
(Use EKS open-source cloud-formation template and implement a solution to test the CF template without launching the EKS cluster)
There are a Number of Ways to optimize the cost for testing of the Cloud-Formation Script working perfectly or not: -
โ LocalStack (Local AWS Cloud Stack)
LocalStack provides an easy-to-use test/mocking framework for developing Cloud applications. It spins up a testing environment on your local machine that provides the same functionality and APIs as the real AWS cloud environment.
โ Syntax or Typos Validation of Cloud-Formation
There are two tools that I know can be used to check the template or Syntax for YAML or JSON.
- Validate-template (aws-cli)
Cloud Formation first checks if the template is valid JSON. If it isnโt, Then Cloud Formation checks if the template is valid YAML. If both these checks fail, Then Cloud Formation returns a template validation error. - AWS CloudFormation Linter
Validate AWS CloudFormation YAML/JSON templates against the AWS CloudFormation Resource Specification and additional checks. Includes checking valid values for resource properties and best practices
Practical Implementation
So, Firstly I am Going to Download the EKS cloud-formation scripts from the given Github link then I will start testing it out with the tools I mentioned above where I will go with step by step link from installation to the use of that tool.
Here For CI/CD I am going to use the Devtron.
At last, I will show How we can also able to integrate the CI/CD in such a way that the Cloud Engineer will first write the Cloud Formation script and then push it on the test branch of the repository from where the Jenkins gets the update afterward Jenkins will test Cloud-Formation template. If it is correct then it will be pushed to the main branch which is directly deploying this CloudFormation Script and if that script has some mistake then it directly notifies the Cloud Engineer Email.
Started Testing manually
โ Letโs Download the Templates First
Basically, For this I am going to create a python script through which I will download selected Cloud-Formation files So, I need do not need to give commands every time.
Validate-template
Letโs Start With Our First Tool i.e. Validate Template, basically, itโs come with aws cli So, we need to first configure the aws-cli then we can able to perform the task of validation.
Configure aws-cli
Now , Configure aws-cli by
$ aws configure
Then start using it.
Letโs Try to Validate a Template
As You Know we are using the validating template that will come under the cloud Formation service because it is made for that only to validate the cloud Formation templates.
What are the options? (How we can be able to pass our file? )
Take the help of command only (manual about that feature):-
So, with the help of โ template-body, we can able to pass our template file which is locally present in our machine.
โtemplate-body file://yourTemplateFileName.yml
If our template resides in an Amazon Simple Storage Service Bucket, then by the help of โtemplate-url we can able to give the complete path.
โ template-url https://s3.amazonaws.com/cloudformation-templates-ap-south-1/yourTemplateFileName.yml
What is a Formats type of Output it will provide?
Let's try to test it out with the help of a template
I am going to make a mistake in the syntax of that YAML file.
I did a mistake around 37 line numbers.
Letโs Resolve that issue and then again try to validate it.
AWS CloudFormation Linter
Installation
Clone Repository
Now, We are able to successfully configure the cfn-lint.
So, with the โ help we got to know about the more options by which we can able to validate our cloud-Formation template.
Here, we can also able to pass multiple templates to validate all in a single go. Like in some regions some services are not available so bypassing the region we can able validate our template such that is this template will work in that particular region or not.
Letโs try test it on our template which have some mistakes;
After Correction
LocalStack
It allows you to develop and test our application locally. Basically, with the help of this tool we can able to set up the whole-complete big architecture of AWS in our Local Machine by the help of APIs of LocalStack we can dev/test our application.
So, as I mentioned that the whole Architecture in your Local machine, then services are also available which states that CloudFormation service is also provided them and even according to our we are testing the CloudFormation Templates of EKS. So, EKS service is also available here.
Perfect, then we can test it in such a way that it will not only validate the syntax CloudFormation can able to deploy the resources actually on our local machine so that we can able to test and stimulate the Cloudformation template with zero cost.
But it has Pricing Model where; ๏ท
โ They provide service free (Open Source โ Community Edition) ๏ But Only Core Services this edition. ๏ท
โ Pro Edition (which cost around โฌ 15/Month and in this edition they also provide 14 days of free trials. ๏ท
โ Enterprise Edition (For Which You need to contact them for more information about this Edition)
โ Before Starting with testing let's first look at the services which we are going in and which pricing they lie.
So, As you can see that CloudFormation Comes under the Community Addition (so its free)
But EKS comes under the Pro Edition. ๐ฉ
But No issue it provides the 14 days free trial letโs use it.
(Note:- According to the testing of these templates we launch the Cloud Formation stack which results in a high cost in comparison to that this โฌ 15 / month is negligible we can able to perform a number of tests and try to simulate the environment this not only can be used for this cloud Formation testing it can help us to a number of different testing like Application Deployment, etc.)
For Getting used the API of other extended AWS features let's signup and first get the API key:-
โ https://app.localstack.cloud/sign-up
โ Then we will directly land on the dashboard โ https://app.localstack.cloud/dashboard
โ As You Can see it doesnโt have the service EKS because its a Community Edition
Configuration Part
There are many ways to setup or configure it :-
โ๏ LocalStack CLI
The easiest way to start and manage LocalStack โ either on your machine, in a Docker container on your machine, or even on a remote Docker host.
โ LocalStack Cockpit
Get a desktop experience and work with your local LocalStack instance via the UI.
โ Docker
Use the docker CLI to manually start the LocalStack Docker container
โ Docker-Compose
Use docker-compose to configure and start your LocalStack Docker container
โ ๏ Helm
Use helm to create a LocalStack deployment in a Kubernetes cluster.
Letโs Configure it by the help of LocalStack CLI
First, install all these requirements
Now configure the Docker because all the APIs are available in that docker image running the container required the container engine so letโs set up the docker:-
Give the Complete Power of Docker Command to Ubuntu (So, that LocalStack can able to run containers by their image on behalf of Ubuntu)
Now Start the Service of Docker
The environment for the Local Stack is Ready lets setup the LocalStack ๏
Finally, The LocalStack Installation is Completed.
Before Starting the LocalStart let's also provide the pro edition access with the help of the API key so that we can able to perform our task.
Start the LocalStack
As you can see a container also started where its API is exposed at 4566 port number
Now, SetUp is Completed But the Question is;
How we can able to make requests to these APIs so that we can able to perform the tasks there should be some interfaces?
Yes, there the two ways through which we can able to perform tasks or who can able to make requests to those APIs
โ Either we can use aws-cli or was local
Let's Start we aws-cli
Before using any command need to configure the aws-cli So, configure it with the help of some dummy Access Key and Secret Key (Because we are not actually to talk with aws API)
For using this LocalStack API you always need to provide โ endpoint-url (i.e. http://localhost:4566) (I already told you that these APIโs are in those containers)
โ Whenever you make a request to these APIs they send those data to that dashboard which is provided by the localstack after signup
List the Stack or Any Kinesis service (is there any stack present or created recently)
Now letโs try to check it with the help of a CloudFormation Template.
Try with the wrong template
So, that stack is also not created (Only that last correct deployment is no new deployment is created)
โ We can also able to set up the awslocal So, that we need not specify โendpoint-url every time.
Dashboard
Now From this practical, I got an idea of How I can able to create a Pipeline! Letโs Try To Implement this Interesting Concept;
Creating CI/CD pipeline for Testing and Deploying the Cloud-Formation Template
โ For the Configuration part of Devtron you can refer to this blog.
Now, in case you can use any one of the tools I have already given a detailed introduction about them with detailed implementation.
But in my case, I am going to use my favorite tool i.e. LocalStack
โ โ Letโs First Try Write the Steps which we need to perform with Devtron So, that it will make our Task Easy โ โ
Step 1: Setup the GitOps Part (WebHook โ trigger โ automatic)
Step 2: Whenever there is any commit in the โ test โ branch ๏ Clone that Repo (also makes it important that who so ever push the code must mention their email inside a file and then make the push request to the โtestโ branch. So, that we can able to send the email if there is an error)
Step 3: Start testing and redirected the output of that command in a file. Step 4: ๏จ If error ๏ then sends the mail to that cloud engineer and abort the further steps. ๏จ If no error ๏ then send a mail to that cloud engineer that your cloud formation template is perfectly fine now pushing it to the main/master branch ( then the job of the main/master branch will start)
Note: FOR STEP 4 WE WILL CREATE A PYTHON SCRIPT
For Script you can refer to my GitHub Repo.
Now, For Complete Practical You Can Watch it on Youtube.
References :
Documentation: https://docs.devtron.ai/
Githu Repo: https://github.com/Shashwatsingh22/CloudFormation-Script-CI-CD/tree/test-by-dev
If this blog helps you gain some knowledge then Please Clap ๐ for it.
and Follow me for such Content ๐
Support me if you like my work! Buy me a coffee
โ โ โ โ โ โ โ * โ โ โ โ โ โ -* โ โ โ โ -* โ โ โ โ โ* โ โ โ โ -
The above blog is submitted under โBlogathon Contest 2022โ conducted by
Devtron (Website Hyperlink here: https://devtron.ai/)
Check out their Github repo: here(Github Hyperlink here: https://github.com/devtron-labs/devtron)
Follow them on: LinkedIn(Hyperlink here: https://www.linkedin.com/company/devtron-labs/),
Twitter(Hyperlink here: https://twitter.com/DevtronL)