A Guide for Dummies: Creating DevOps Chains with Open Source Tools
Creating the first DevOps chain in five steps for newbies.
DevOps has become a panacea for too slow, disconnected and other problematic development processes. But we need minimal knowledge in DevOps. It will cover concepts such as the DevOps chain and how to create it in five steps. This is not a complete guide, but only a “fish” that can be expanded. Let's start with the story.
My acquaintance with DevOps
I once worked with clouds in the Citi Group and developed an IaaS web application to manage Citi's cloud infrastructure, but I was always wondering how to optimize the development chain and improve the culture among developers. Greg Lavender, our technical director for cloud architecture and infrastructure, advised me about the book Project Phoenix. She perfectly explains the principles of DevOps, while being read like a novel.
The table on the back shows how often companies are rolling out new versions:
How do Amazon, Google and Netflix manage to roll out so much? And everything is simple: they understood how to create an almost perfect DevOps chain.
In Citi, everything was completely different, until we switched to DevOps. Then my team had different environments, but we did the delivery to the development server manually. All developers had access to only one development server based on IBM WebSphere Application Server Community Edition. When trying to deliver at the same time, the server “fell”, and we had to “negotiate” each time “painfully”. We also had insufficient code coverage with tests, a laborious manual delivery process and no possibility to track the delivery of a code with the help of some task or customer requirement.
It was clear that an urgent need to do something, and I found a like-minded colleague. We decided to create the first DevOps chain together – he set up a virtual machine and Tomcat application server, and I started Jenkins, integrating with Atlassian Jira and BitBucket, as well as covering the code with tests. The project was successful: we fully automated the development chain, achieved almost 100% uninterrupted operation of the development server, could monitor and improve code coverage with tests, and the Git branch could be tied to the Jira delivery and task. And almost all the tools with which we built the DevOps chain were open source.
In fact, the chain was simplified, because we did not even use advanced configurations using Jenkins or Ansible. But we did it. Perhaps this is a consequence of the Pareto principle (aka the 80/20 rule).
Brief description of the DevOps and CI / CD chains
DevOps has different definitions. DevOps, like Agile, includes various disciplines. But most would agree with the following definition: DevOps is a method, or life cycle, of software development, the main principle of which is to create a culture where developers and other employees are “on the same wavelength”, manual labor is automated, everyone is doing what he can best the frequency of deliveries increases, productivity increases, and flexibility increases.
And although the tools alone are not enough to create the DevOps environment, you cannot do without them. The most important of these is continuous integration and continuous delivery (CI / CD). There are different stages for each environment in the chain (for example, DEV (development), INT (integration), TST (testing), QA (quality control), UAT (acceptance testing by users), STG (preparation), PROD (use)), manual tasks are automated, developers can make high-quality code, make its delivery, and can easily be rebuilt.
This note describes how to create a DevOps chain in five steps, as shown in the image below, using open source tools.
Let's get down to business.
Step 1: CI / CD Platform
The first thing you need is a CI / CD tool. Jenkins is an open source CI / CD tool written in Java with the MIT license, which began popularizing the DevOps movement and which de facto became the standard for CI CD.
What is Jenkins? Imagine that you have a magic control panel for a variety of services and tools. The CI / CD tool, such as Jenkins, is useless by itself, but with different tools and services it becomes omnipotent.
In addition to Jenkins there are many other open tools, choose any.
Here’s what the DevOps process looks like with the CI / CD tool.
You have a CI / CD tool in localhost, but for now there is nothing special to do. Let's go to the next step.
Step 2: Versioning
The best (and, perhaps, the easiest) way to check the magic of the CI / CD tool is to integrate it with the source control management (SCM) tool. Why do I need version control? Let's say you make an application. You write it in Java, Python, C ++, Go, Ruby, JavaScript, or in any other language, of which the car and a small cart. What you write is called source code. At first, especially if you are working alone, you can save everything to a local directory. But when the project grows and more people join it, you need a way to share code changes, but avoid conflicts when merging changes. And you need to somehow restore previous versions without using backups and using the copy-paste method for files with code.
And here without SCM anywhere. SCM stores the code in repositories, manages its versions and coordinates it among developers.
There are a lot of SCM tools, but Git has deservedly become the standard. I advise you to use it, but there are other options.
This is what the DevOps pipeline looks like after adding SCM.
The CI / CD tool can automate the loading and unloading of source code and teamwork. Not bad? But now how to make this a working application, beloved by billions of users?
Step 3: build automation tool
Everything goes as it should. You can upload code and commit changes to the version control system, as well as invite friends to work with you. But you have no application yet. For this to be a web application, it must be compiled and placed in a package for delivery or run as an executable file. (An interpreted programming language, such as JavaScript or PHP, is not necessary to compile.)
Use an assembly automation tool. Whatever tool you choose, it will assemble the code in the right format and automate the cleanup, compilation, testing, and delivery. Build tools vary by language, but the following open source options are commonly used.
Excellent! Now insert the build automation tool configuration files into the version control system so that the CI / CD tool assembles them.
It feels good. But where now to roll it all out?
Step 4: Web Application Server
So, you have a packed file that you can execute or roll out. In order for an application to be beneficial, it must have some kind of service or interface, but you need to put it all somewhere.
A web application can be hosted on a web application server. The application server provides an environment where you can execute the program logic from the package, draw the interface, and open web services through a socket. You need an HTTP server and several other environments (virtual machine, for example) to install an application server. For now, let's imagine that you are dealing with all of this in the process (although I will talk about containers below).
There are several open web application servers.
We have already turned out almost a working DevOps chain. Great job!
In principle, you can stop here, then you can do it yourself, but you should also talk about the quality of the code.
Step 5: Test Coverage
Testing takes a lot of time and effort, but it is better to immediately find errors and improve the code in order to please the end users. For this purpose, there are many open tools that not only test the code, but also advise how to improve it. Most CI / CD tools can connect to these tools and automate the process.
Testing is divided into two parts: testing frameworks for writing and executing tests, and tools with tips for improving code quality.
Testing frameworks
Tools with quality tips
Most of these tools and frameworks are written for Java, Python, and JavaScript, because C ++ and C # are proprietary (although GCC has open source).
We applied the test coverage tools, and now the DevOps pipeline should look like the figure at the beginning of the tutorial.
Additional steps
Containers
As I said, the application server can be placed on a virtual machine or server, but containers are more popular.
What are containers? In short, in the virtual machine, the operating system most often takes up more space than the application, and a container usually needs several libraries and configurations. In some cases, virtual machines are indispensable, but the container holds the application along with the server at no extra cost.
Docker and Kubernetes are usually taken for containers, although there are other options.
Check out the Docker and Kubernetes articles on Opensource.com:
- What is a docker?
- Introduction to Docker
- What is Kubernetes?
- Kubernetes from scratch
Middleware Automation Tools
Our DevOps chain is focused on building and delivering the application together, but you can do other interesting things with the DevOps tools. For example, use the Infrastructure as Code (IaC) tools, which are also called middleware automation tools. These tools help automate installation, management, and other tasks for middleware. For example, an automation tool can take applications (web application server, database, monitoring tools) with the correct configurations and roll them out to the application server.
Here are some options for open source middleware automation tools:
Details in articles on Opensource.com:
- Ansible Getting Started Guide
- Ansible deployment automation strategies
- Top 5 Configuration Management Tools
And now what?
This is just the tip of the iceberg. The DevOps chain can do much more. Start with the CI / CD tool and find out what else you can automate to simplify your work. Don't forget about open communication tools for effective collaboration.
Here are some more good DevOps articles for beginners:
- What is DevOps?
- 5 DevOps Developer Skills
- DevOps for everyone
- Introduction to Predictive Analytics in DevOps
And you can integrate DevOps with open tools for agile:
- What is agile?
- 4 steps to mastering agile
