Learn to deploy Java (and other) applications for free (Docker, CircleCI, Google Cloud)
Hello everyone. Lately, I’ve been thinking more and more often about what other useful article I could write. In parallel to this, I constantly saw advertisements on the Internet about “cool” courses in IT, they promise to make you Java, Python and any developer in six months/year, and okay with it, let’s say they teach you something in a year and, perhaps, where – then you will get a free internship (which is also uncertain, given the current market). But when they advertise “cool DevOps courses,” I already begin to wonder how one can become a DevOps without having programming experience, without having tried the entire development cycle in at least some language, without trying out various application build settings, without trying Linux , with all its utilities, docker, cuber, git, etc. in work, but just “learn” this in isolation from everything and expect that they will take you somewhere, and if they take you, then they will let you do something serious? I still couldn’t find the answer to this question in my head.
However, why am I doing this, and besides, I tried all this back in my student years, and still, until recently, I didn’t really get involved in development, simply because there were people who were separately involved in these things. But recently I had the opportunity to help one company with setting up the deployment of some of their products, and overcoming my uncertainty, I finally decided, something worked out and I decided to share my experience (and replenish the knowledge base of numerous AI chatbots that constantly surf the Internet in search of new information). Of course, this is far from an “advanced” level and, most likely, in serious large applications you will use something more serious, like Gitlab Ci, Kuber, etc., but usually only devops have direct access to these things, leaving out a couple of files for developers, which they set up, like a ci file or helm charts, but that’s not the point of the article, the point is that this is a simple guide for developers like me, who haven’t done full deployment setup on their own before, but I don’t mind trying, to expand my experience, you can consider this a continuation of my series of articles about pet projects (first we learned how to better design it, now we’re trying to deploy it).
So let’s get started. Firstly, I will say that for deployment we will use the free CircleCi service. Yes, perhaps this is not the most obvious option, because there are GitHub Actions, there is Gitlab Ci, there is a lot more, but CircleCi seemed quite simple to me, which means it is quite suitable for beginners and those who just want to try. To get started, you just need to register. Previously, this could be done through Github, but for some reason this functionality was disabled at the end of September, so register by mail and connect Github, I think you will find how to do it. Next, if you have access to several projects from different users, select your profile and open the list of projects in the left menu. In the list of projects you will see a button “Set up project”, after clicking which it will offer you several options to choose from. The easiest one is to use the settings from the project. Create a .circleci folder with a config.yml file in your project folder, and after setting up this file, circleci automatically uses it for deployment.
So, we have decided on the CD service (continuous delivery and continuous deployment). But where to deploy, you ask me, because for this you need a machine. If you have a local machine that can be accessed from outside, you can easily use it for testing, I tried this option in this combination, everything also works. However, because Most do not have such servers, we will also need some kind of service that provides Linux servers. The most obvious choice is Amazon’s AWS. A great service that provides a free year (!) of server use. However, they may have problems validating your card, because they even rejected my work visa card and then forced me to send a bunch of documents to confirm it, and it still didn’t help. Therefore, our choice for now falls on Google Cloud. Unfortunately, they provide only 3 free months and $300 credit, however, it’s quite suitable for testing and trying, and there were no problems with linking the card. I think how to find a service, log in to your Google account, register and enter your credit data, I don’t need to explain, so let’s move on to creating an instance and setting it up. Having entered the console, scroll down a little and select “Create a VM”, in the page that opens click “Create instance”. Next, we configure our instance.
You can choose the region and name as you wish, as well as the configuration of the machine, depending on your needs, the standard E2 is quite suitable for us. And in general, most of the settings can be left default, except for those that I will show. Be sure to allow HTTP/HTTPS traffic and select your Linux distribution if you wish; in my case, I chose Ubuntu. Honestly, we will add everything else later, so click next and create your instance.
After creation, go to instance -> metadata and add your public ssh key to connect to the server. If for some reason you still don’t have it, and, for example, you use Git using your login and password, then what are you waiting for, run and generate a pair for yourself right now (ssh-keygen -t ed25519 -C "your_email@example.com")
For now, we are done with setting up the instance. Next, we will still need to open the ports we need, but more on that later. Let’s move directly to setting up the application build and Ci file. And so, each language and type of application has its own way of building it, but I will tell you using the example of my Spring application in Java. I’ll say right away that it didn’t turn out entirely successful, because… it is microservice and initially I only counted on launching it locally, and accordingly I configured all the communication between services directly in the docker compose file, without using nginx or anything similar, a rookie mistake, but quite suitable for demonstrating deployment. So, first create a Dockerfile of your application. In my case, everything turned out a little complicated, first I assemble the entire project using a Maven dockerfile and copy the generated jars to the project folder, because I used a multi-module Maven project structure. If you have only one service in the project, then your dockerfile will look quite simple:
FROM openjdk:17.0.2-jdk-slim
COPY target/shows-migrations-service-0.0.1-SNAPSHOT.jar .
CMD ["java", "-jar", "shows-migrations-service-0.0.1-SNAPSHOT.jar"]Next, write a compose file in which you will collect all the services you need, namely your service, database, and anything else as needed. In my case it was 3 services, postgres and keycloak:
version: "3.8"
services:
postgres:
image: 'postgres:14-alpine'
container_name: postgres
restart: always
ports:
- "5432:5432"
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
POSTGRES_DB: postgres
volumes:
- ./imports/init.sql:/docker-entrypoint-initdb.d/init.sql
networks:
- auth
keycloak:
image: quay.io/keycloak/keycloak:20.0.0
container_name: keycloak
restart: always
volumes:
- ./imports:/opt/keycloak/data/import
command: ['start-dev --import-realm --http-relative-path=/auth']
environment:
KC_DB: postgres
KC_DB_URL_HOST: postgres
KC_DB_URL_PORT: 5432
KC_DB_URL_DATABASE: postgres
KC_DB_USERNAME: postgres
KC_DB_PASSWORD: postgres
KEYCLOAK_ADMIN: admin
KC_DB_SCHEMA: public
KEYCLOAK_ADMIN_PASSWORD: admin
KEYCLOAK_FRONTEND_URL: http://localhost:8484/auth
ports:
- "8484:8080"
depends_on:
- postgres
links:
- "postgres:postgres"
networks:
- auth
...
shows-data:
image: shows-data-service
container_name: shows-data
restart: always
depends_on:
- postgres
- keycloak
- migrations
network_mode: host
environment:
POSTGRES_JDBC_URL: jdbc:postgresql://localhost:5432/postgres
KEYCLOAK_URL: http://localhost:8484/auth
SERVER_PORT: 8080
extra_hosts:
- "host.docker.internal:host-gateway"
...
networks:
auth:
driver: bridgeAs you can see, there are enough settings, and I also omitted 2 services. In short, I import data into postgres, import realm into keycloak, and configure services. Services work on the “host” network, which means that they work on the instance network, in fact this is not necessary, because in the docker compose file you can forward the ports, and when you open the ports in the settings of our instance, everything will work, but for some reason I needed this, I don’t remember why, in general, it depends on the needs of your application. The next step is to configure the CircleCi file itself. Because it turns out to be quite long, I’ll remove repeating things and give them in parts, with an explanation:
version: 2.1
jobs:
deploy-to-development:
docker:
- image: docker:20.10.9
steps:
- checkout
- setup_remote_docker
- run:
name: Build Docker Image
command: |
chmod +x ./build.sh
./build.sh
docker build -f Dockerfile-migrations -t migrations-service:latest .
- run:
name: Compress Docker Image
command: |
docker save migrations-service:latest | gzip > migrations-service.tar.gz
- run: ls -lh
- persist_to_workspace:
root: .
paths:
- migrations-service.tar.gz
- ./docker/Here we are using a Docker image to use its commands. I run my sh file to build the application (the one that collects and copies jars, you most likely will not have this), then the docker image of your application is built, be sure to give it a tag. Next, we save this image and compress it. And save the resulting file and the desired folder with docker compose files (in my case this is a folder, you can just have 1 file).
transfer-and-run:
machine:
image: ubuntu-2004:202010-01
steps:
- attach_workspace:
at: .
- run:
name: Install SSH And Configure
command: |
echo $SSH_PRIVATE_KEY | base64 --decode > ./id_rsa
chmod 400 id_rsa
- run:
name: Stop Remote Docker-Compose
command: |
ssh -o "StrictHostKeyChecking=no" -i ./id_rsa $USER@$HOST '
if [ -f compose.yml ]; then
sudo docker-compose -f docker/compose.yml down --rmi all
sudo rm compose.yml
else
echo "compose.yml not found"
fi
'
- run:
name: Transfer Files
command: |
scp -o "StrictHostKeyChecking=no" -i ./id_rsa ./shows-frontend-service.tar.gz $USER@$HOST:~/
scp -o "StrictHostKeyChecking=no" -i ./id_rsa -r ./docker $USER@$HOST:~/
- run:
name: Decompress Docker Image | Run Compose
command: |
ssh -o "StrictHostKeyChecking=no" -i ./id_rsa $USER@$HOST '
sudo gunzip -c ./shows-frontend-service.tar.gz | sudo docker load
sudo rm ./*.tar.gz
cd docker
sudo docker compose -f compose.yml up -d
'
workflows:
deploy-to-dev:
jobs:
- deploy-to-development:
filters:
branches:
only:
- deploy
- transfer-and-run:
requires:
- deploy-to-development
filters:
branches:
only:
- deployAnd so, here, as we see, we define the second part of our deployment, namely the transfer of files and their launch. We do this using the Ubuntu image, not Docker. Here we use the files from the previous step that we saved (workspace). Next, we take SSH_KEY, which we previously saved in the ENV project variables in Circleci in base64 form, and save it to a file, giving it the necessary access. Circleci stores the variables themselves in encrypted form and seems to be safe. An alternative: do the same thing, but on the host machine and give some kind of key to access Git, clone the Git project and do everything on the host machine, but this doesn’t seem to be much better. Next, we use this key to connect via ssh to the host, also using the $USER and $HOST variables. Stop the docker file and delete unnecessary images. We copy the files we need to the host machine (I remind you that I omitted some for the sake of brevity, so don’t be surprised that the names are jumping around), and load the images we compressed into Docker, then delete the compressed files. Then we run the compose file, and all services should start. Also below you see the settings for which branch to launch push jobs from, in my case it is deploy.
After that, you will see in the CircleCi panel that all your steps have been completed and marked with a checkmark, after which you can connect to the host machine and check manually that everything works (though not in my case, because I messed up a little with the structure of the application itself, but everything started to deploy and start, so that’s another conversation). And before I forget, I’ll show you how to add ENV variables to the project: go to Settings Project and add them to Environment Variables (by the way, I just noticed that it turns out that you can directly add SSH keys there, it’s probably better to do this in the future, but since I wrote a guide, I’ll leave it like this):
And how to open ports in the google console: go to the page of your project (it’s important, not the instance, but the project), go to the left menu in “VPC Network -> Firewall”, click on the top “Create firewall rule”, give a name as desired, be sure to put some tag and allow the ports you need (for example, 8081). Then go to your instance, click “Edit” and in “Network tags” add the tag that you set for the rule, and your port will open.
The full code of the project and the project itself are available on my git, if someone doesn’t understand something and needs the source (I myself perfectly understand this feeling when something is not clear in the guide, but there is no link to the source, I hope I didn’t burn any unnecessary data in the article and the git). In any case, I think this guide will be useful to someone, after all, it is a useful experience for developers to try to deploy their pet projects on their own, especially since if you manage to register on Amazon successfully, then you will have a whole year of free hosting, which is quite good, if you deploy some of your pet projects and indicate a link to it in your git/resume, you will find a job within a year (although in this economy this is not a fact, of course).
Oh, and I almost forgot to say, CircleCi also has a time limit for a build used per month, so I recommend not to abuse it too much and only deploy from some stable branch. And, of course, now it’s fashionable to use Kuber everywhere, and almost no one deploys with bare Docker, but for a simple application it’s fine, and even if the application crashes, the “restart: always” property will restart it, so there’s some kind of stability -is still present. In any case, if you liked the article, you will find links to me in my profile, thank you all for your attention.
