Pytest-check assert method non-blocking errors

On the eve of the start of the course Python QA Engineer for future students and everyone interested in the topic of testing, we prepared a translation of useful material.

We also invite you to see the gift demo lesson on “QA Career”.


There is a lot of debate in the testing community about how many asserts should be in one automated UI test. Some people think that there should be one assert per test, that is, each test should check only one element. Others are quite happy that their test checks several items at once.

Whichever approach you choose, I think it’s safe to say that tests should remain clear, concise, readable, and of course, should be easy to maintain. Personally, I have no problems with multiple asserts in one test, since I focus on one functional area.

For example, let’s take a registration form:

As part of testing the user registration form, I might want to test several things at once. Maybe I want to check that the message – congratulations on registration is displayed correctly, or maybe I will check that the user will be redirected to the login page after registration.

In such a case, I will not verify that the user can successfully log in and leave that for another test.

Problem

The problem with Python’s built-in assert method is that it stops test execution on the first error. In some scenarios, this actually makes sense. That is, there is no point in continuing the test if part of the application is not working. However, it also happens that it makes sense to carry out all the checks regardless of whether an error occurs on one of them or not, especially if all your asserts refer to the same page at the end of the test.

We use this approach when testing web services. When inspecting the response body, we most likely want to check each return value and then print out which ones were not as expected, instead of stopping the test on failure in the first assert.

Solution: Pytest-check plugin

Pytest-check (from Brian Ocken) – Pytest pluginwhich allows you to wrap all test asserts in a single pass / fail result. For example, if you have 3 asserts and the first of them fails, then Pytest-check will continue to execute the remaining 2. Then it will report that the test has failed if one or more checks come with the result fail.

Python OpenSDK TestProject also supports Pytest, so if you already have pytest tests on Selenium it is very easy to convert them to tests TestProject… You can read more about this in my blog article HowQAand see a step-by-step getting started guide here

Let’s take a look at how the Pytest-check plugin works.

Selenium tests

First, we need a test. I will use the registration page of this application: https://docket-test.herokuapp.com/register

import selenium.webdriver as webdriver
from selenium.webdriver.common.by import By
def test_register_user():
    # Arrange
    url = "https://docket-test.herokuapp.com/register"
    # set the driver instance
    driver = webdriver.Chrome()
    # browse to the endpoint
    driver.get(url)
    # maximise the window
    driver.maximize_window()
    # Act
    # Complete registration form
    # enter username value
    driver.find_element(By.ID, "username").send_keys("Ryan")
    # enter email value
    driver.find_element(By.ID, "email").send_keys("Test@email.com")
    # enter password value
    driver.find_element(By.ID, "password").send_keys("12345")
    # enter repeat password value
    driver.find_element(By.ID, "password2").send_keys("12345")
    # click register button
    driver.find_element(By.ID, "submit").click()

We have a test, but we need to write some tests. Let’s do it with the assert method:

# Assert
# confirm registration has been successful
# check if congratulations message contains the correct text
message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text
assert message == "Congratulations, you are now registered"
# check user is routed to login page
current_url = driver.current_url
assert current_url == "https://docket-test.herokuapp.com/login"

If we run them, then everything will be successful:

So far so good, but what happens if both asserts return fail? Let’s change the code to see what happens:

# Assert
# confirm registration has been successful
# check if congratulations message contains the correct text
message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text
assert message == "Well done, You've Registered"
# check user is routed to login page
current_url = driver.current_url
assert current_url == "https://docket-test.herokuapp.com/register"
driver.quit()

So, we changed the message we expect to see and changed the expected URL, so when we run this test again, it will return with a fail result:

As expected, the test failed in assert. Great, our test fulfilled its purpose, we found an error, or something like that …

However, we can see from the error message that the test failed due to the wording of the message, and it did not even check the URL. The test stops running as soon as the first fail appears, so we can’t see the whole picture. In this case, we will need to correct the text of the expected message and repeat the test again. Let’s do it.

We have now changed the expected message back to “Congratulations, you are now registered“, And we can run the test again:

Aha! The test crashed again, this time due to an invalid URL.

I know what you’re thinking about how cool it would be if we caught both of these errors in one test run. Well, you’re in luck, Pytest-check comes into play.

Pytest-Check

Installation

We can install pytest-check via pip install pytest-check… After we have installed pytest-check, it can be imported into our test.

import pytest_check as check

Now that everything is ready, we can tweak our asserts a little. From now on, we will not use the assert statement, instead we will use the pytest-check syntax as follows.

To check the message, use the function check.equal and add the expected and actual text there, for example:

check.equal(message, "Congratulations, you are now registered1")

We can do the same with URL validation, but we’ll do it with a different method, namely check.is_in

check.is_in("login", current_url)

The complete test looks like this:

import selenium.webdriver as webdriver
from selenium.webdriver.common.by import By
import pytest_check as check
def test_register_user():
    # Arrange
    url = "https://docket-test.herokuapp.com/register"
    # set the driver instance
    driver = webdriver.Chrome()
    # browse to the endpoint
    driver.get(url)
    # maximise the window
    driver.maximize_window()
    # Act
    # Complete registration form
    # enter username value
    driver.find_element(By.ID, "username").send_keys("Ryan8")
    # enter email value
    driver.find_element(By.ID, "email").send_keys("Test@email8.com")
    # enter password value
    driver.find_element(By.ID, "password").send_keys("12345")
    # enter repeat password value
    driver.find_element(By.ID, "password2").send_keys("12345")
    # click register button
    driver.find_element(By.ID, "submit").click()
    # Assert
    # confirm registration has been successful
    # check if congratulations message contains the correct text
    message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text
    check.equal(message, "Congratulations, you are now registered")
    # check user is routed to login page
    current_url = driver.current_url
    check.is_in("login", current_url)
    driver.quit()

At this point, we can return all expected values ​​to their original state for the test to complete successfully. So let’s do it.

Fine! Now let’s see what happens if both checks fail. First you need to change the code so that they fail:

# check if congratulations message contains the correct text
message = driver.find_element(By.XPATH, "/html[1]/body[1]/div[1]/div[1]/div[1]/div[1]/form[1]/div[1]").text
check.equal(message, "Congratulations, you are now registered!")
# check user is routed to login page
current_url = driver.current_url
check.is_in("1", current_url)

We launch.

Just like last time, the test failed, as we expected, but now we see that fail returned in two checks: first, the conclusion that the content of the message does not match the expected, and then the result of the URL check. If you approach the question from the point of view of pytest, then we can consider this as one failed test, but now we know that in fact several checks returned fail.

Only if all checks in the test are successful will it be marked as pass.

Here is such a cool Pytest-check. You can find out more about it in documentation


Learn more about the Python QA Engineer course.

Watch a demo lesson on “QA Career”.

GET A DISCOUNT

Similar Posts

Leave a Reply