How to Set Up a Staged Pipeline in GitLab CI

In GitLab CI, pipelines play a key role in automating CI/CD processes. They allow you to break down the entire build, test, and deploy process into separate, logically related tasks — or “jobs“. These jobs are structured into stages, each of which represents a specific stage of work, such as build, test, or deployment. This division allows for faster development and minimizes errors when delivering code to production.

In this article, we'll look at how to set up a staged pipeline in GitLab CI.

Pipeline setup

File .gitlab-ci.yml — This is where GitLab CI starts working. It is where we describe stages, jobs, and their dependencies. This file is not just a configuration file, it is the entity that defines the behavior of all your CI/CD processes.

Each file .gitlab-ci.yml starts with defining the stages, then defining the jobs that will be performed at these stages. Minimal example:

stages:
  - build
  - test
  - deploy

This code defines three stages – build, test and deploy. It sounds simple, but as soon as we start adding jobs, it gets interesting.

Once the stages are defined, tasks need to be assigned to them. Each task executes a set of commands:

build-job:
  stage: build
  script:
    - echo "Building the project..."
    - make build

test-job:
  stage: test
  script:
    - echo "Running tests..."
    - make test

deploy-job:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - make deploy

In this example, we have three jobs: one for each stage. Each job executes its command at the stage it is attached to. Each job runs independently and can have its own environment variables, dependencies, and startup conditions.

The main feature of dividing the pipeline into stages is the ability to perform different stages depending on the status of the previous ones. For example, testing will not start until the build is complete, and deployment will only occur after successful testing.

An example of a more complex pipeline configuration:

stages:
  - build
  - test
  - deploy
  - cleanup

# Джоба для сборки
build-job:
  stage: build
  script:
    - echo "Building project..."
    - make build
  artifacts:
    paths:
      - build/

# Джоба для тестирования
test-job:
  stage: test
  script:
    - echo "Running tests..."
    - make test
  dependencies:
    - build-job
  artifacts:
    paths:
      - test-results/

# Джоба для деплоя
deploy-job:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - make deploy
  when: manual
  environment: production

# Джоба для очистки временных файлов
cleanup-job:
  stage: cleanup
  script:
    - echo "Cleaning up..."
    - rm -rf build/ test-results/
  only:
    - master

This pipeline is already more complex and interesting. Here's what's important about it:

Artifacts: We save build and test artifacts for subsequent stages. This makes the pipeline independent of re-executing previous stages.
Dependencies: specifying dependencies dependencieswe guarantee that the task test-job uses the results of the assembly.
Manual deployment: deployment is set to manual start when: manual.
Cleaning: the last stage is cleaning up temporary files. It is performed only for the branch masterso as not to clutter the repository with temporary artifacts.

Let's imagine that you need to build a Docker container, test it, and then deploy it to production. This is what it might look like:

stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  image: docker:stable
  services:
    - docker:dind
  script:
    - docker build -t myapp:$CI_COMMIT_SHA .
    - docker save myapp:$CI_COMMIT_SHA | gzip > myapp.tar.gz
  artifacts:
    paths:
      - myapp.tar.gz

test-job:
  stage: test
  script:
    - docker load -i myapp.tar.gz
    - docker run myapp:$CI_COMMIT_SHA ./run-tests.sh
  dependencies:
    - build-job

deploy-job:
  stage: deploy
  script:
    - echo "Deploying to production..."
    - docker load -i myapp.tar.gz
    - docker tag myapp:$CI_COMMIT_SHA myrepo/myapp:latest
    - docker push myrepo/myapp:latest
  when: manual
  environment: production

This pipeline performs the following steps:

Building a Docker Image at the stage build.
Testing the image at the stage test.
Deploy to production when started manually at stage deploy.

Parallel execution of stages

When tasks in a pipeline are independent of each other or require execution after other tasks, the directive needs becomes an excellent solution for acceleration. It allows tasks to be launched not sequentially by stages, but in parallel, as soon as the necessary tasks are completed, which significantly reduces the overall execution time.

Example of configuration using the directive needs:

stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  script:
    - make build

test-job:
  stage: test
  needs: ["build-job"]
  script:
    - make test

deploy-job:
  stage: deploy
  needs: ["test-job"]
  script:
    - make deploy

Here are the tasks test-job And deploy-job will start executing immediately after their dependencies have successfully completed, which speeds up execution compared to the classic sequential launch of stages.

GitLab CI also has a directive dependencieswhich controls which artifacts can be passed between tasks at different stages. This is a must-have when one task needs the results of another.

Example:

stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  script:
    - make build
  artifacts:
    paths:
      - build/

test-job:
  stage: test
  script:
    - make test
  dependencies:
    - build-job

deploy-job:
  stage: deploy
  script:
    - make deploy
  dependencies:
    - test-job

Here are the artifacts created at the stage buildare transferred to the task test-joband then in deploy-job.

One effective method for speeding up pipelines is caching. GitLab allows you to cache dependencies such as libraries and packages so that you don’t have to download them again each time you run the pipeline. For example:

build-job:
  stage: build
  cache:
    paths:
      - node_modules/
  script:
    - npm install
    - npm run build

Cached here node_moduleswhich allows you to avoid installing them again at each assembly step.

Directive parallel allows you to run multiple instances of the same task in parallel.

Example of parallel testing with different Python versions:

test-job:
  stage: test
  image: python:$VERSION
  script:
    - pytest
  parallel:
    matrix:
      - VERSION: ['3.8', '3.9', '3.10', '3.12']

This pipeline will run four tasks in parallel, each with a different version of Python.

In conclusion, I would like to remind you about the open lesson on using gitlab-ci to work with ansible, which will take place on September 18.
In this lesson, you'll learn how to use GitLab CI to automate processes with Ansible. You'll learn the basics of working with GitLab CI and Ansible, and learn how to create pipelines for testing playbooks and managing infrastructure.
You can sign up for a lesson for free on the course page “CI/CD based on GitLab”.