Playwright and GitLab CI Integration

The approach to maintaining test documentation and the tools chosen for it are an important part of the development process, which directly affects the quality of the product. It is especially important to keep test documentation up to date. Qase can be one of the tools suitable for this. In addition, it helps to unite the world of manual testing with automated testing, and the described test cases with their execution.

In this article, we will look at the implementation of the Qase bundle with Playwright and GitLab CI, which we use in SmartHead: from creating a project to receiving automated testing reports.

Benefits of Using Qase, Playwright and GitLab CI

Qase offers extensive functionality for test management, including test cases, checklists, test runs and plans, as well as integrations with bug tracking tools. When combined with Playwright and GitLab CI, this provides the following benefits:

1. Centralized test management: Convenient interface for maintaining all test cases, plans and runs.

2. Automatic creation and synchronization of tests: Ensuring test cases are up-to-date and minimizing the risk of duplication.

3. Ease of integration and use: Playwright makes it easy to write and maintain tests, and GitLab CI automates their execution whenever your code changes.

4. Detailed and visual reports: Qase provides detailed test execution reports, which helps you quickly identify and fix errors.

5. Run tests locally or in CI: Ability to run tests on a local computer or in a CI/CD pipeline with automatic synchronization of information about each run and test case in Qase.

6. Launch pipelines directly from Qase: More integrated and automated testing process. Reports are available in both Qase and GitLab after each run.

7. Attachment support: Ability to attach screenshots, logs, test completion records and other files to reports for detailed analysis of errors and problems.

Let's move on to implementation.

Step 1: Create a project in Qase

Create a new project in Qase

1. Register or log in to your account at Question.

2. Create a new project, specifying its name and other necessary data.

Screenshot of a new repository example in Qase

Setting up automatic test case creation

In the project settings (Settings > Run), enable the option to automatically create new test cases if there are no matches. This will help avoid duplication and keep the database up to date.

When this function is disabled, the name of the test case from the code will be displayed in the test run, but it will not be added to the repository. When the function is enabled, if the test case is not marked as existing (id from Qase is not specified), then the new test case will be added to the project repository automatically.

Create multiple test cases in a project

1. Create 1-2 test cases. We will need them for further steps.

2. Specify basic parameters such as title, description, and test execution steps.

Step 2: Setting up Qase + Playwright integration

Install Playwright and playwright-qase-reporter

1. Go to the root of the project.

2. Follow the instructions to install Playwright.

3. Also, by instructions, install reporter for Qase.

Set up integration in Qase

1. Login to Qase.

2. Go to the “Apps” section.

3. Find and select Playwright.

4. Install the integration and save the created token. You can give it any clear name, it will not be used anywhere, only the token itself.

Playwright activation screen in Qase

Setting up configuration in Playwright

1. Create a .env file. It will store our sensitive data for local launch. We will add the copied token from Qase there.

Screenshot of Qase token in Playwright

2. After installing Playwright, playwright.config.ts will appear in the root of the project. Open the file and add the following configuration:

import { defineConfig, devices } from '@playwright/test';
require('dotenv').config({ path: '.env' }); // Необходим для использования переменных в .env

export default defineConfig({
  testDir: './tests', // Ваш репозиторий с тестами
  fullyParallel: true,
  forbidOnly: !!process.env.CI,
  retries: process.env.CI ? 2 : 0,
  workers: process.env.CI ? 1 : undefined,

  reporter: [
    ['list'],
    ['playwright-qase-reporter',
      {
        testops: {
          project: 'TEST', // Ваш код проекта из Qase
          api: {
            token: process.env.QASE_TOKEN, // Токен из .env
          },
          run: {
            id: process.env.QASE_RUN_ID,
            complete: true,
          },
          uploadAttachments: true,
        },
      }],
  ],

  use: {
    trace: 'on-first-retry',
  },

  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] },
    }
  ],

});

Don't forget to add the .env file to .gitignore to avoid accidentally pushing the token to the repository.

About id in Qase runs

id: process.env.QASE_RUN_ID

Specifying QASE_RUN_ID in the config is necessary to avoid creating different runs in Qase. Without this setting, the automatic run that we create in Qase and the run that starts the pipeline will be assigned different ids. Thus, we clearly indicate: “Go and see if there is already a created QASE_RUN_ID, and if so, put the tests from the run in the pipeline there.”

Write case IDs in tests

To link tests from Playwright with tests from the Qase repository, we need to additionally specify their id in the tests. For convenience, it is better to give names identical to those recorded in Qase. Although one way or another, the test run will indicate the name of the case whose id you specified.

// test.spec.ts
import { test, expect } from '@playwright/test';
import { qase } from 'playwright-qase-reporter';

test('Отображение тайтла главной страницы Playwright', async ({ page }) => {
   qase.id(1); // ID тест-кейса в Qase

   // Step #1
   await page.goto('https://playwright.dev/');
   await expect(page).toHaveURL('https://playwright.dev/');
   // Expected result: Страница открывается

   // Step #2
   const title = await page.getByText('Playwright enables reliable end-to-end testing for modern web apps.');
   await expect(title).toBeVisible();
   // Expected result: Тайтл отображается
});

Also, as an example, we will write a test in which we will not specify qase.id():

test('Тест, которого нет в репозитории', async ({ page }) => {
   // Step #1
   await page.goto('https://playwright.dev/');
   await expect(page).toHaveURL('https://playwright.dev/');
   // Expected result: Страница открывается

   // Step #2
   const title = await page.getByText('Playwright enables reliable end-to-end testing for modern web apps.');
   await expect(title).toBeVisible();
   // Expected result: Тайтл отображается
});

Running tests

Run the tests with the command, specifying your path to the tests:

QASE_MODE=testops npx playwright test tests/test.spec.ts

Checking results in Qase

After successfully passing the tests, you will receive a link to your run in Qase in the console.

Right from the console we can follow the link and see the test run that was just completed.

You can also share it by making the link public or keeping it private. You can also see in detail all the steps of each individual case, the time it took to complete it, follow the link to view the Pipeline, etc.

Screenshot of a completed test run in Qase

The case that was specified with the id will pull up the existing case from Qase. It will pull up the steps you specified and all other information.

A case without specifying an id will be pulled up by default in this state:

A test case that was not there before will also appear in the project repository. It will be created thanks to the setting that we added at the beginning of the article (Setting up automatic test case creation). If you disable this setting, the tests will pass, but new ones will not be added to the repository.

Recommendation: on subsequent launches a new case will not be created, but I would advise writing the id in the code of each case to avoid confusion.

Step 3: Integrate Qase with GitLab CI

Create a project on GitLab if it hasn't been created yet. Also, make sure CI/CD is enabled in “Settings” -> “General” -> “Visibility, project features, permissions”.

Set up variables in GitLab

Thanks to the created .env we will be able to safely store data on the project, but to run tests in CI we need to somehow share this data with GitLab. For this purpose, there are Variables in GitLab.

To add your own variable, you need to:

1. Go to “Settings” -> “CI/CD”. Here we can save information similar to the .env file for future use.

2. Click Add variable and in the window that opens, enter in Key QASE_TOKEN, and in Value — token values.

3. Save the data:)

Screenshot of saved variable in git

Configure the .gitlab-ci.yml file in the project

There are many articles available on CI/CD that will help you choose the most suitable way to use it. I will only show a small part that will help integrate with Qase.

To begin, add a .gitlab-ci.yml file to the root of your project. An example of a minimal .gitlab-ci.yml configuration would include:

1. Grabbing the official Docker image from Playwright.

2. Establishing dependencies.

3. Running tests with sending reports to Qase.

stages:
- test

test:
stage: test
image: mcr.microsoft.com/playwright:v1.42.1-jammy

script:
 - npm ci
 - npx playwright install
 - QASE_MODE=testops npx playwright test tests/test.spec.ts

Don't forget to push the code to your GitLab repository.

After this, the pipeline will be launched and a test completion report similar to the one previously generated will be generated in Qase.

Create an automated test run in Qase

After all the settings have been done, we can run a run in Qase, which will create a pipeline in GitLab and run the tests.

1. Go to Qase.

2. Open the Test Runs section and click Start a new test run

3. In the test run settings, enter the required data, select integration with GitLab, specify the required project, branch, and run the tests.

Screenshot Start new test run

Once the run is complete, you will receive a detailed report on it.

You can also share it, making the link public or not. Also see in detail all the steps of passing each individual case, the time of its completion, follow the link to see the Pipeline, etc.

After the run is completed, a report will also be generated.

So we ran the autotests in three ways: locally, when the code in the repository changed, when creating an automatic run in Qase. In each case, the report was sent to Qase.

Conclusion

It is important to effectively use modern automation and testing tools to build software. The integration of Qase, Playwright and GitLab CI is a great combination that will help improve the development and testing process.

Test automation with Playwright allows you to quickly and reliably check the functionality of applications on different platforms and browsers, reducing testing time. GitLab CI ensures automatic test execution with each code update, which helps to detect and fix errors earlier. And integration with Qase allows you to see the full picture within one system: from creating a test case to its automated execution and receiving a detailed report.