Docker Compose: From Development to Production
Translation of podcast transcriptions prepared in advance of the start of the course “Linux Administrator”
Docker Compose is an amazing tool for building a desktop
environment for the stack used in your application. It allows you to define
every component of your application, following the clear and simple syntax in YAML-
files…
With the advent docker compose v3 these YAML files can be used directly in the production environment when working with
cluster Docker swarm…
But does this mean that you can use the same docker-compose file in
development process and production environment? Or use the same file for
staging? Well, in general – yes, but for such functionality we need the following:
- Variable interpolation: using environment variables for some
values that change in each environment. - Configuration override: the ability to define a second (or any
another follow-up) docker-compose file which will change something relatively
first, and docker compose will take care of merging both files.
Differences between development and production files
During development, you will most likely want to check for code changes in
in real time. To do this, usually the source volume is mounted in
the container that contains the runtime for your application. But for the production environment
this method is not suitable.
In production you have a cluster with many nodes and the volume is local to the software
relative to the node that your container (or service) is running on, so you don’t
you can mount the source code without complicated operations that include
sync code, signals, etc.
Instead, we usually want to create an image with a specific version of your code.
It is customary to mark it with the appropriate tag (you can use the semantic
versioning or another system of your choice).
Overriding configuration
Considering the differences and that your dependencies may differ in scenarios
development and production, it is clear that we will need different config files.
Docker compose supports combining different compose files for
obtaining the final configuration. How it works can be seen in an example:
$ cat docker-compose.yml
version: "3.2"
services:
whale:
image: docker/whalesay
command: ["cowsay", "hello!"]
$ docker-compose up
Creating network "composeconfigs_default" with the default driver
Starting composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1 | ________
whale_1 | < hello! >
whale_1 | --------
whale_1 |
whale_1 |
whale_1 |
whale_1 | ## .
whale_1 | ## ## ## ==
whale_1 | ## ## ## ## ===
whale_1 | /""""""""""""""""___/ ===
whale_1 | ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
whale_1 | ______ o __/
whale_1 | __/
whale_1 | __________/
composeconfigs_whale_1 exited with code 0
As said, docker compose supports combining multiple compose-
files, this allows you to override various options in the second file. For example:
$ cat docker-compose.second.yml
version: "3.2"
services:
whale:
command: ["cowsay", "bye!"]
$ docker-compose -f docker-compose.yml -f docker-compose.second.yml up
Creating composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1 | ______
whale_1 | < bye! >
whale_1 | ------
whale_1 |
whale_1 |
whale_1 |
whale_1 | ## .
whale_1 | ## ## ## ==
whale_1 | ## ## ## ## ===
whale_1 | /""""""""""""""""___/ ===
whale_1 | ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
whale_1 | ______ o __/
whale_1 | __/
whale_1 | __________/
composeconfigs_whale_1 exited with code 0
This syntax is not very convenient during development, when the command
will need to be done many times.
Fortunately, docker compose automatically looks for a special file named
docker-compose.override.yml to override values docker-compose.yml… If a
rename the second file, you get the same result, just using the original command:
$ mv docker-compose.second.yml docker-compose.override.yml
$ docker-compose up
Starting composeconfigs_whale_1
Attaching to composeconfigs_whale_1
whale_1 | ______
whale_1 | < bye! >
whale_1 | ------
whale_1 |
whale_1 |
whale_1 |
whale_1 | ## .
whale_1 | ## ## ## ==
whale_1 | ## ## ## ## ===
whale_1 | /""""""""""""""""___/ ===
whale_1 | ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
whale_1 | ______ o __/
whale_1 | __/
whale_1 | __________/
composeconfigs_whale_1 exited with code 0
Okay, that’s easier to remember.
Interpolating variables
Configuration files support interpolation
variables and default values. That is, you can do the following:
services:
my-service:
build:
context: .
image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
...
And if you do docker-compose build (or push) no environment variable
$ MY_SERVICE_VERSION, the value latestbut if you install
pre-build environment variable value, it will be used during build or push
to the register private.registry.mine…
My principles
Approaches that are convenient for me may be useful for you too. I follow this
simple rules:
- All my stacks for production, development (or other environments) are defined via
docker-compose files. - The configuration files needed to cover all my environments as much as possible
avoid duplication. - I need one simple command to work in each environment.
- The main configuration is defined in the file docker-compose.yml…
- Environment variables are used to define tags for images or other
variables that can change from environment to environment (staging, integration,
production). - Variable values for production are used as values for
by default, this minimizes the risk of running the stack in production without
set environment variable. - To start a service in a production environment, use the command docker stack deploy – compose-file docker-compose.yml –with-registry-auth my-stack-name…
- The working environment is started using the command docker-compose up -d…
Let’s take a look at a simple example.
# docker-compose.yml
...
services:
my-service:
build:
context: .
image: private.registry.mine/my-stack/my-service:${MY_SERVICE_VERSION:-latest}
environment:
API_ENDPOINT: ${API_ENDPOINT:-https://production.my-api.com}
...
AND
# docker-compose.override.yml
...
services:
my-service:
ports: # This is needed for development!
- 80:80
environment:
API_ENDPOINT: https://devel.my-api.com
volumes:
- ./:/project/src
...
I can use docker-compose (docker-compose up)to start the stack in
development mode with source code mounted in / project / src…
I can use these same files in production! And I could use exactly
the same file docker-compose.yml for staging. To expand this on
production, I just need to build and send an image with a predefined tag
at the CI stage:
export MY_SERVICE_VERSION=1.2.3
docker-compose -f docker-compose.yml build
docker-compose -f docker-compose.yml push
In production, this can be run with the following commands:
export MY_SERVICE_VERSION=1.2.3
docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth
And if you want to do the same at the stage, you just need to define
necessary environment variables for working in a staging environment:
export MY_SERVICE_VERSION=1.2.3
export API_ENDPOINT=http://staging.my-api.com
docker stack deploy my-stack --compose-file docker-compose.yml --with-registry-auth
As a result, we used two different docker-compose files, which without
duplicate configurations can be used for any of your environments!
Learn more about the course “Linux Administrator”