building a web server

Foreword

It so happened that I have to work on a large number of sites, the tasks to solve are also different – from server settings to “make up a form”. And on one of the projects, a task arose – to upgrade to the current version of php (8.1 at the time of writing), upgrade to the current version of CMS (1C Bitrix), and, in general, “bring it to mind”.
Since the project has acquired a significant amount of functionality that is not directly related to the site (incremental and full scheduled backups with uploading to the cloud, compilation of dictionaries, synchronization with different providers), and work is carried out in 3 environments (locally, a test site and a production site), then I decided that this would be a good opportunity to move the entire infrastructure to Docker containers.
Since the technology is already established, it was expected that there would be a ready-made server template “out of the box” that would suit our needs. But having searched, it was not possible to find a complete solution – everywhere there were some nuances, because of which the solution did not fit. As a result, our own server was assembled for the site on 1C Bitrix. After that, everything related to this CMS was cut from the server and now it can be used for other projects without restrictions.

The code is available at github.

Server Components

For the full operation of the server, we need the following components:

database (MySQL);
PHP
NGINX;
proxy for sending mail (msmtp);
composer;
letsencrypt certificates;
backup and recovery;
optionally – a cloud for storing backups.

We also need to schedule different actions to run. For this, crontab will be used on hostand not in containers.

Before starting work

On the server, we need docker-compose. Instructions:

We will also need access to the smtp mail service and s3 storage for backups (optional).

About gmail smtp

Google informed, which from June 2022 suspends access to insecure applications (with authorization only by account password). To be able to use gmail smtp, you need to enable two-factor authentication in your account settings, create a separate authorization password for our site and use it. Detailed instructions are sufficient.

Services and environments

For flexibility in setting up the server, we create 4 separate compose.yml files:

compose-app.yml – the main services of our application (database, php, nginx, composer);
compose-https.yml – for site operation via https protocol. Includes certbot, as well as http to https redirect rules for nginx;
compose-cloud.yml – for storing backups in cold storage;
compose-production.yml – overrides restart rules for all containers.

compose-app.yml

version: '3'
services:
  db:
    image: mysql
    container_name: database
    restart: unless-stopped
    tty: true
    environment:
      MYSQL_DATABASE: ${DB_DATABASE}
      MYSQL_ROOT_PASSWORD: ${DB_ROOT_PASSWORD}
      MYSQL_USER: ${DB_USER}
      MYSQL_PASSWORD: ${DB_USER_PASSWORD}
    volumes:
      - ./.backups:/var/www/.backups
      - ./.docker/mysql/my.cnf:/etc/mysql/my.cnf
      - database:/var/lib/mysql
    networks:
      - backend

  app:
    image: php:8.1-fpm
    container_name: application
    build:
      context: .
      dockerfile: Dockerfile
      args:
        GID: ${SYSTEM_GROUP_ID}
        UID: ${SYSTEM_USER_ID}
        SMTP_HOST: ${MAIL_SMTP_HOST}
        SMTP_PORT: ${MAIL_SMTP_PORT}
        SMTP_EMAIL: ${MAIL_SMTP_USER}
        SMTP_PASSWORD: ${MAIL_SMTP_PASSWORD}
    restart: unless-stopped
    tty: true
    working_dir: /var/www/app
    volumes:
      - ./app:/var/www/app
      - ./log:/var/www/log
      - ./.docker/php/local.ini:/usr/local/etc/php/conf.d/local.ini
    networks:
      - backend
    links:
      - "webserver:${APP_NAME}"

  composer:
    build:
      context: .
    image: composer
    container_name: composer
    working_dir: /var/www/app
    command: "composer install"
    restart: "no"
    depends_on:
      - app
    volumes:
      - ./app:/var/www/app

  webserver:
    image: nginx:stable-alpine
    container_name: webserver
    restart: unless-stopped
    tty: true
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./app/public:/var/www/app/public
      - ./log:/var/www/log
      - ./.docker/nginx/default.conf:/etc/nginx/includes/default.conf
      - ./.docker/nginx/templates/http.conf.template:/etc/nginx/templates/website.conf.template
    environment:
      - APP_NAME=${APP_NAME}
    networks:
      - frontend
      - backend

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge

volumes:
  database:

compose-https.yml

version: '3'
services:
  webserver:
    volumes:
      - ./.docker/certbot/conf:/etc/letsencrypt
      - ./.docker/certbot/www:/var/www/.docker/certbot/www
      - ./.docker/nginx/templates/https.conf.template:/etc/nginx/templates/website.conf.template

  certbot:
    image: certbot/certbot
    container_name: certbot
    restart: "no"
    volumes:
      - ./log/letsencrypt:/var/www/log/letsencrypt
      - ./.docker/certbot/conf:/etc/letsencrypt
      - ./.docker/certbot/www:/var/www/.docker/certbot/www

compose-cloud.yml

version: '3'
services:
  cloudStorage:
    image: efrecon/s3fs
    container_name: cloudStorage
    restart: unless-stopped
    cap_add:
      - SYS_ADMIN
    security_opt:
      - 'apparmor:unconfined'
    devices:
      - /dev/fuse
    environment:
      AWS_S3_BUCKET: ${AWS_S3_BUCKET}
      AWS_S3_ACCESS_KEY_ID: ${AWS_S3_ACCESS_KEY_ID}
      AWS_S3_SECRET_ACCESS_KEY: ${AWS_S3_SECRET_ACCESS_KEY}
      AWS_S3_URL: ${AWS_S3_URL}
      AWS_S3_MOUNT: '/opt/s3fs/bucket'
      S3FS_ARGS: -o use_path_request_style
      GID: ${SYSTEM_GROUP_ID}
      UID: ${SYSTEM_USER_ID}
    volumes:
      - ${AWS_S3_LOCAL_MOUNT_POINT}:/opt/s3fs/bucket:rshared

compose-production.yml

version: '3'
services:
  db:
    restart: always

  app:
    restart: always

  webserver:
    restart: always

  cloudStorage:
    restart: always

And define the environment settings in the file .env

.env

COMPOSE_FILE=compose-app.yml:compose-cloud.yml:compose-https.yml:compose-production.yml
SYSTEM_GROUP_ID=1000
SYSTEM_USER_ID=1000

APP_NAME=example.local
ADMINISTRATOR_EMAIL=example@gmail.com

DB_HOST=db
DB_DATABASE=example_db
DB_USER=example
DB_USER_PASSWORD=example
DB_ROOT_PASSWORD=example

AWS_S3_URL=http://storage.example.net
AWS_S3_BUCKET=storage
AWS_S3_ACCESS_KEY_ID=#YOU_KEY#
AWS_S3_SECRET_ACCESS_KEY=#YOU_KEY_SECRET#
AWS_S3_LOCAL_MOUNT_POINT=/mnt/s3backups

MAIL_SMTP_HOST=smtp.gmail.com
MAIL_SMTP_PORT=587
MAIL_SMTP_USER=example@gmail.com
MAIL_SMTP_PASSWORD=example

Depending on what set of services we need in a particular environment, we specify in the variable COMPOSE_FILE set of compose-*.yml files

In catalog .docker/ we store settings for all services that are used in the application. There are 2 worth noting here:

For nginx we use a rules file .docker/nginx/default.conf and two patterns (.docker/nginx/templates/http.conf.template and .docker/nginx/templates/https.conf.template). Depending on which protocol we are working on, the appropriate nginx settings will be used. Templates are detailed on the image page. nginx;
For msmtp in a file .docker/msmtp/msmtp we specify patches of the form #PASSWORD#which will be replaced when the image is built.

.docker/msmtp/msmtprc

# Set default values for all following accounts.
defaults
auth           on
tls            on
logfile        /var/www/log/msmtp/msmtp.log
timeout 5

account        docker
host           #HOST#
port           #PORT#
from           #EMAIL#
user           #EMAIL#
password       #PASSWORD#

# Set a default account
account default : docker

Create a file Dockerfilein which we indicate the features of the assembly and, as mentioned earlier, for msmtp we set connection parameters from environment variables:

Dockerfile

FROM php:8.1-fpm

ARG GID
ARG UID
ARG SMTP_HOST
ARG SMTP_PORT
ARG SMTP_EMAIL
ARG SMTP_PASSWORD

USER root

WORKDIR /var/www

RUN apt-get update -y \
    && apt-get autoremove -y \
    && apt-get -y --no-install-recommends \
    msmtp \
    zip \
    unzip \
    && rm -rf /var/lib/apt/lists/*

COPY ./.docker/msmtp/msmtprc /etc/msmtprc

RUN sed -i "s/#HOST#/$SMTP_HOST/" /etc/msmtprc \
        && sed -i "s/#PORT#/$SMTP_PORT/" /etc/msmtprc \
        && sed -i "s/#EMAIL#/$SMTP_EMAIL/" /etc/msmtprc \
        && sed -i "s/#PASSWORD#/$SMTP_PASSWORD/" /etc/msmtprc

COPY --from=composer /usr/bin/composer /usr/bin/composer

RUN getent group www || groupadd -g $GID www \
    && getent passwd $UID || useradd -u $UID -m -s /bin/bash -g www www

USER www

EXPOSE 9000

CMD ["php-fpm"]

Backup

The backup consists of two parts: an archive with files and a database dump. We can store them locally or send them to the cloud. We use a script to generate cgi-bin/create-backup.sh.
Recovery – cgi-bin/restore-backup.sh respectively. If we have a cloud storage connected, then we will offer to restore from it:

create-backup.sh

#!/bin/bash

BASEDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../" &> /dev/null && pwd)

source "$BASEDIR/.env"

cd "$BASEDIR/"

# If run script with --local, then don't send backup to remote storage
moveToCloud="Y"
while [ $# -gt 0 ] ; do
    case $1 in
        --local) moveToCloud="N";;
    esac
    shift
done

# If backups storage is not mounted, then anyway store backups local
if ! [[ $COMPOSE_FILE == *"compose-cloud.yml"* ]]; then
    moveToCloud="N"
fi

# Current date, 2022-01-25_16-10
timestamp=`date +"%Y-%m-%d_%H-%M"`
backups_local_folder="$BASEDIR/.backups/local"
backups_cloud_folder="$AWS_S3_LOCAL_MOUNT_POINT"

# Creating local folder for backups
mkdir -p "$backups_local_folder"

# Creating backup of application
tar \
	--exclude="vendor" \
    -czvf $backups_local_folder/"$timestamp"_app.tar.gz \
	-C $BASEDIR "app"

# Creating backup of database
docker exec database sh -c "exec mysqldump -u root -h $DB_HOST -p$DB_ROOT_PASSWORD $DB_DATABASE" > $backups_local_folder/"$timestamp"_database.sql
gzip $backups_local_folder/"$timestamp"_database.sql

# If required, then move current backup to cloud storage
if [ $moveToCloud == "Y" ]; then
    mv $backups_local_folder/"$timestamp"_database.sql.gz $backups_cloud_folder/"$timestamp"_database.sql.gz
    mv $backups_local_folder/"$timestamp"_app.tar.gz $backups_cloud_folder/"$timestamp"_app.tar.gz
fi

# If we already moved backup to cloud, then remove old backups (older than 30 days) from cloud storage
if [ $moveToCloud == "Y" ]; then
    /usr/bin/find $backups_cloud_folder/ -type f -mtime +30 -exec rm {} \;
fi

restore-backup.sh

#!/bin/bash

BASEDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../" &> /dev/null && pwd)

source "$BASEDIR/.env"

cd "$BASEDIR/"

backupsDestination="$BASEDIR/.backups/local"

# If backups storage is mounted, ask, from where will restore backups
if [[ $COMPOSE_FILE == *"compose-cloud.yml"* ]]; then
    while true
    do
        reset
        echo "Select backups destination:"
        echo "1. Local;"
        echo "2. Cloud;"
        echo "---------"
        echo "0. Exit"

        read -r choice

        case $choice in
            "0")
                exit
                ;;
            "1")
                break
                ;;
            "2")
                backupsDestination="$AWS_S3_LOCAL_MOUNT_POINT"
                break
                ;;
            *)
                ;;
        esac
    done
fi
reset

# Select backup for restore
echo "Available backups:"
find "$backupsDestination"/*.gz  -printf "%f\n"
echo "------------"
echo "Enter backup path:"

read -i "$backupsDestination"/ -e backup_name

if ! [ -f "$backup_name" ]; then
    echo "Wrong backup path."
    exit 1
fi


backup_mode="unknown"
if [[ $backup_name == *"app.tar.gz"* ]]; then
    backup_mode="app"
elif [[ $backup_name == *"database.sql.gz"* ]]; then
    backup_mode="database"
fi

if [ $backup_mode == "unknown" ]; then
    echo "Unknown backup type"
    exit 1
fi

reset

if [ $backup_mode == "app" ]; then
    mkdir -p "$BASEDIR"/.backups/tmp
    cp "$backup_name" "$BASEDIR"/.backups/tmp/app_tmp.tar.gz

    tar -xvf "$BASEDIR"/.backups/tmp/app_tmp.tar.gz -C "$BASEDIR"

    rm -rf "$BASEDIR"/.backups/tmp
fi

if [ $backup_mode == "database" ]; then
    mkdir -p "$BASEDIR"/.backups/tmp
    cp "$backup_name" "$BASEDIR"/.backups/tmp/database_tmp.sql.gz

    gunzip "$BASEDIR"/.backups/tmp/database_tmp.sql.gz

    if ! [ -f "$BASEDIR"/.backups/tmp/database_tmp.sql ]; then
        echo "Error in database unpack process"
        exit 1
    fi

    docker-compose exec db bash -c "exec mysql -u root -p$DB_ROOT_PASSWORD $DB_DATABASE < /var/www/.backups/tmp/database_tmp.sql"

    rm -rf "$BASEDIR"/.backups/tmp
fi

Crontab

Scheduled launch is done on the host side. The file is used for initialization. cgi-bin/prepare-crontab.sh. During execution, the script collects all files from the directory .crontabreplaces the path to the application in them #APP_PATH# to the current one, and adds them to the crontab on the host.

prepare-crontab.sh

#!/bin/bash

BASEDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../" &> /dev/null && pwd)

# Load environment variables
source "$BASEDIR"/.env

# Create temporary directory
mkdir -p "$BASEDIR"/.crontab_tmp/

# Copy all crontab files to temporary directory
cp "$BASEDIR"/.crontab/* "$BASEDIR"/.crontab_tmp/

# Set actual app path in crontab files
find "$BASEDIR"/.crontab_tmp/ -name "*.cron" -exec sed -i "s|#APP_PATH#|$BASEDIR|g" {} +

# Set crontab
if [[ $COMPOSE_FILE == *"compose-https.yml"* ]]; then
    find "$BASEDIR"/.crontab_tmp/ -name '*.cron' -exec cat {} \; | crontab -
else
    find "$BASEDIR"/.crontab_tmp/ -name '*.cron' -not -name 'certbot-renew.cron' -exec cat {} \; | crontab -
fi

# Remove temporary directory
rm -rf "$BASEDIR"/.crontab_tmp/

Certbot

If https is not needed within this environment, then we skip this step.
We use certbot to get ssl certificates. But there is one feature here – to confirm ownership of the domain, we need to start nginx, but it will not start without certificates. It turns out a vicious circle. We use a script to solve cgi-bin/prepare-certbot.shwhich creates stub certificates, starts nginx, requests up-to-date certificates, installs them, and restarts nginx.
To update the certificates, create a file cgi-bin/certbot-renew.shwhich we will launch according to the schedule.

prepare-certbot.sh

#!/bin/bash

BASEDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../" &> /dev/null && pwd)

source "$BASEDIR/.env"

cd "$BASEDIR/"

if ! [ -x "$(command -v docker-compose)" ]; then
  echo 'Error: docker-compose is not installed.' >&2
  exit 1
fi

domains=($APP_NAME www.$APP_NAME)
rsa_key_size=4096
data_path="$BASEDIR/.docker/certbot"
email=$ADMINISTRATOR_EMAIL
staging=0 # Set to 1 if you're testing your setup to avoid hitting request limits

if [ -d "$data_path" ]; then
  read -p "Existing data found for $domains. Continue and replace existing certificate? (y/N) " decision
  if [ "$decision" != "Y" ] && [ "$decision" != "y" ]; then
    exit
  fi
fi


if [ ! -e "$data_path/conf/options-ssl-nginx.conf" ] || [ ! -e "$data_path/conf/ssl-dhparams.pem" ]; then
  echo "### Downloading recommended TLS parameters ..."
  mkdir -p "$data_path/conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf > "$data_path/conf/options-ssl-nginx.conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot/certbot/ssl-dhparams.pem > "$data_path/conf/ssl-dhparams.pem"
  echo
fi

echo "### Creating dummy certificate for $domains ..."
path="/etc/letsencrypt/live/$domains"
mkdir -p "$data_path/conf/live/$domains"
docker-compose run --rm --entrypoint "\
  openssl req -x509 -nodes -newkey rsa:$rsa_key_size -days 1\
    -keyout '$path/privkey.pem' \
    -out '$path/fullchain.pem' \
    -subj '/CN=localhost'" certbot
echo


echo "### Starting nginx ..."
docker-compose up --force-recreate -d webserver
echo

echo "### Deleting dummy certificate for $domains ..."
docker-compose run --rm --entrypoint "\
  rm -Rf /etc/letsencrypt/live/$domains && \
  rm -Rf /etc/letsencrypt/archive/$domains && \
  rm -Rf /etc/letsencrypt/renewal/$domains.conf" certbot
echo


echo "### Requesting Let's Encrypt certificate for $domains ..."
domain_args=""
for domain in "${domains[@]}"; do
  domain_args="$domain_args -d $domain"
done

case "$email" in
  "") email_arg="--register-unsafely-without-email" ;;
  *) email_arg="--email $email" ;;
esac

if [ $staging != "0" ]; then staging_arg="--staging"; fi

docker-compose run --rm --entrypoint "\
  certbot certonly --webroot -w /var/www/.docker/certbot/www \
    $staging_arg \
    $email_arg \
    $domain_args \
    --rsa-key-size $rsa_key_size \
    --agree-tos \
    --force-renewal" certbot
echo

echo "### Reloading nginx ..."
docker-compose exec webserver nginx -s reload

certbot-renew.sh

#!/bin/bash

BASEDIR=$(cd "$(dirname "${BASH_SOURCE[0]}")/../" &> /dev/null && pwd)

cd "$BASEDIR/"

docker-compose run --rm certbot renew && ocker-compose kill -s SIGHUP webserver
docker system prune -af

At this stage, the site is available and you can continue working with it.

A step-by-step installation process and a description of the variables are available at github.

building a web server

Foreword

Server Components

Before starting work

Services and environments

Backup

Crontab

Certbot

The beginning of the end of Freedom of choice in industrial automation

Feature Request Management

Linux file permissions

How to make CI on Github for a modern frontend

Programmer career after 30

Why WSL 2 is 13x Faster Than WSL: Insider Preview Impressions

Leave a Reply Cancel reply

Foreword

Server Components

Before starting work

Services and environments

Backup

Crontab

Certbot

Similar Posts

Leave a Reply Cancel reply