How to test a website with Django. Part 2. JavaScript and Russian text on English pages

In the last article, we got acquainted with tests for Django and created a personal tester user. It’s time to continue learning about testing the site by writing a test for Russian characters on English pages and parsing tests for JavaScript.

Testing text translation

When a page intended for a foreign user contains Russian text (or vice versa), this is, to put it mildly, not very pleasant. Therefore, it is better to test such situations and get rid of them in time.

Our translation tests are divided into two types:

Checking the absence of Russian characters on English pages;
Checking for errors in django.po files.

Checking the absence of Russian characters on English pages

Let’s move on to the first test. Its full code looks like this:

from requests import get
from bs4 import BeautifulSoup
from bs4.element import Tag

from django.test import TestCase

# А
CODE_OF_FIRST_RUSSIAN_LETTER = 1040

# я
CODE_OF_LAST_RUSSIAN_LETTER = 1103

CODES_OF_RUSSIAN_SYMBOLS = list(
    range(
        CODE_OF_FIRST_RUSSIAN_LETTER,
        CODE_OF_LAST_RUSSIAN_LETTER + 1
    )
) + [1025, 1105] # Ё, ё

ENGLISH_PAGES = (
    'https://pvs-studio.com/ru/',
    'https://pvs-studio.com/ru/blog/posts/',
    'https://pvs-studio.com/ru/for-clients/',
    'https://pvs-studio.com/ru/docs/',
    # ...
)

def is_russian_symbol(symbol: str) -> bool:
    """True if symbol is Russian"""

    return ord(symbol) in CODES_OF_RUSSIAN_SYMBOLS

def get_page_words(content: Tag) -> tuple:
    """Return page's words"""

    page_text = content.get_text()
    page_text_without_extra_spaces = " ".join(page_text.split())
    page_words = page_text_without_extra_spaces.split()

    return tuple(page_words)

class CorrectTranslationTests(TestCase):
    """Test correct translation"""

    def test_russian_symbols_presence(self):
        """Test English pages for presence of Russian symbols"""

        for page in ENGLISH_PAGES:
            page_content = BeautifulSoup(get(page).content, 'html.parser')
            main_div_content = page_content.find(
                'div', {"class": 'b-content'}
            )

            if main_div_content:
                page_words = get_page_words(main_div_content)

                for word in page_words:
                    for symbol in word:
                        error_message = f'n"{symbol}" ({word}) on {page}n'

                        with self.subTest(error_message):
                            self.assertFalse(is_russian_symbol(symbol))

Let’s analyze it in detail.

Imports

To get content from a page, we need to send a GET request to it. The method successfully copes with this get.

from requests import get

To get the content of a particular page block, we need beautiful soup. For type hinting we import Tag.

from bs4 import BeautifulSoup
from bs4.element import Tag

And let’s not forget about test casewhich allows you to create tests.

from django.test import TestCase

Constants

Each character has its numeric code. In Python, this code can be obtained using the function order. Russian character codes range from 1040 (A) to 1103 (I). They also include 1025 (Yo) and 1105 (Yo). It is these codes that we enter into the variable CODES_OF_RUSSIAN_SYMBOLS. With its help, a search for Russian characters will be carried out.

# А
CODE_OF_FIRST_RUSSIAN_LETTER = 1040

# я
CODE_OF_LAST_RUSSIAN_LETTER = 1103

CODES_OF_RUSSIAN_SYMBOLS = list(
    range(
        CODE_OF_FIRST_RUSSIAN_LETTER,
        CODE_OF_LAST_RUSSIAN_LETTER + 1
    )
) + [1025, 1105] # Ё, ё

IN ENGLISH_PAGES we enter the pages that we will check:

ENGLISH_PAGES = (
    'https://pvs-studio.com/ru/',
    'https://pvs-studio.com/ru/blog/posts/',
    'https://pvs-studio.com/ru/for-clients/',
    'https://pvs-studio.com/ru/docs/',
    # ...
)

is_russian_symbol function

The check for a Russian character is performed by the function is_russian_symbol. She receives the code and checks for its presence in the variable CODES_OF_RUSSIAN_SYMBOLS.

def is_russian_symbol(symbol: str) -> bool:
    """True if symbol is Russian"""

    return ord(symbol) in CODES_OF_RUSSIAN_SYMBOLS

get_page_words function

The content of the page contains, in addition to the text, elements that we do not need. For example, tags. To remove them from beautifulsoup there is a cool method get_text. It only extracts text from HTML markup (1). We could just use this method, but the text it produces contains a large number of consecutive spaces. Therefore, we replace them with single ones using split And join (2). Having received a string, we split it into a list of words (3) in order to iterate over it in the future.

def get_page_words(content: Tag) -> tuple:
    """Return page's words"""

    page_text = content.get_text() # (1)
    page_text_without_extra_spaces = " ".join(page_text.split()) # (2)
    page_words = page_text_without_extra_spaces.split() # (3)

    return tuple(page_words)

test function

The test works like this:

Bypasses all English pages;
Extracts all content from each page (a similar example was in the last article);
On our website baboutMost of the text that the user sees is stored in div with class b-content. Therefore, the test extracts content from it using the method find. Other blocks div we test separately;
Gets all the words from the content;
Iterates over each word and each character;
Checks if the character is not Russian.

class CorrectTranslationTests(TestCase):
    """Test correct translation"""

    def test_russian_symbols_presence(self):
        """Test English pages for presence of Russian symbols"""

        for page in ENGLISH_PAGES: # (1)
            # (2)
            page_content = BeautifulSoup(get(page).content, 'html.parser')
            main_div_content = page_content.find(
                'div', {"class": 'b-content'}
            ) # (3)

            if main_div_content:
                page_words = get_page_words(main_div_content) # (4)

                for word in page_words: # (5)
                    for symbol in word: # (5)
                        error_message = f'n"{symbol}" ({word}) on {page}n'

                        with self.subTest(error_message):
                            # (6)
                            self.assertFalse(is_russian_symbol(symbol))

Test run

Running the test now, we won’t see any errors. To make sure it really works, let’s test it for Russian page.

Checking for errors in django.po files

Perhaps when you saw this heading, you wondered: “Why test files django.poif we already directly check the absence of Russian characters on English pages? There are at least 2 reasons for this:

Test django.po helps to catch some of the errors and is fast. This means that it can be carried out after each commit;
If the phrase translation is the empty string (i.e. msgstr stores the empty string), the file test django.po will help to detect this, in contrast to the test written earlier.

In total we will check 3 conditions. Namely, what is in the file django.po must not be:

Rows with empty value msgstr;
A string that starts with “msgid”;
A line that starts with “#, fuzzy”.

Failure to fulfill at least one of these points indicates that the compilation of the translation was completed with errors.

Full test django.po files looks like this:

from django.test import TestCase

MY_PROJECT_LOCALE_PATH = 'my_project/locale/'

DJANGO_PO_PATH = 'LC_MESSAGES/django.po'

RU_DJANGO_PO_PATH = MY_PROJECT_LOCALE_PATH + 'ru/' + DJANGO_PO_PATH

EN_DJANGO_PO_PATH = MY_PROJECT_LOCALE_PATH + 'en/' + DJANGO_PO_PATH

def is_msgstr_empty(line_with_msgstr: str, next_line: str) -> bool:
    """True if msgstr is empty"""

    return 'msgstr ""' in line_with_msgstr and next_line == 'n'

def get_msgstr_errors(file: list) -> tuple:
    """Return numbers of lines where msgstr is empty"""

    errors = []

    for number in range(len(file) - 1):
        line = file[number]
        next_line = file[number + 1]

        if is_msgstr_empty(line, next_line):
            line_number = number + 1

            errors.append(line_number)

    return tuple(errors)

def get_fuzzy_and_msgid_errors(file: list) -> tuple:
    """Return numbers of lines that starts with '#, fuzzy' or #~ msgid"""

    errors = []

    for line_number, line in enumerate(file, 1):
        if line.startswith('#, fuzzy') or line.startswith('#~ msgid'):
            errors.append(line_number)

    return tuple(errors)

def get_locale_files_errors() -> dict:
    """Return errors for ru and en django.po files"""

    with open(RU_DJANGO_PO_PATH, 'r') as file:
        ru_file = list(file).copy()

    with open(EN_DJANGO_PO_PATH, 'r') as file:
        en_file = list(file).copy()

    errors = {
        'ru': get_fuzzy_and_msgid_errors(ru_file) + get_msgstr_errors(ru_file),
        'en': get_fuzzy_and_msgid_errors(en_file) + get_msgstr_errors(en_file),
    }

    return errors

class LocaleTests(TestCase):
    """Tests for locale files"""

    def test_locale_files(self):
        """Test en and ru django.po files"""

        for language, errors in get_locale_files_errors().items():
            with self.subTest(f'Errors for {language}: {sorted(errors)}'):
                self.assertEqual(len(errors), 0)

Let’s see what happens in it.

Constants

In them, we simply form Russian and English paths to django.po files. For example, the first one would be: pvs/locale/en/LC_MESSAGES/django.po.

MY_PROJECT_LOCALE_PATH = 'my_project/locale/'

DJANGO_PO_PATH = 'LC_MESSAGES/django.po'

RU_DJANGO_PO_PATH = MY_PROJECT_LOCALE_PATH + 'ru/' + DJANGO_PO_PATH

EN_DJANGO_PO_PATH = MY_PROJECT_LOCALE_PATH + 'en/' + DJANGO_PO_PATH

is_msgstr_empty function

The first function checks if the string contains msgstr with an empty value.

def is_msgstr_empty(line_with_msgstr: str, next_line: str) -> bool:
    """True if msgstr is empty"""

    return 'msgstr ""' in line_with_msgstr and next_line == 'n'

And here everything seems to be clear, but what is the “and next_line == ‘n’” part for? Everything is simple. If the content of msgstr is very long, Django will wrap it to another line. As a result, the string we are checking will contain the “msgstr “”” part, but it will not be an error, since the translation is given below.

If the translation is small, then it will remain on the same line as msgstr. And the next line will be “n“.

get_msgstr_errors function

Let’s move on to the function get_msgstr_errors. It gives line numbers where msgstr has an empty value. The principle of operation of the function is as follows:

Iterates through the lines of a file;
Stores the current line in a variable line;
Stores the next line in a variable next_line;
Using the previously described function, checks if the current line contains an empty msgstr;
If yes, then enters the line number in the general list.

def get_msgstr_errors(file: list) -> tuple:
    """Return numbers of lines where msgstr is empty"""

    errors = []

    for number in range(len(file) - 1): # (1)
        line = file[number] # (2)
        next_line = file[number + 1] # (3)

        if is_msgstr_empty(line, next_line): # (4)
            line_number = number + 1

            errors.append(line_number) # (5)

    return tuple(errors)

get_fuzzy_and_msgid_errors function

Move on. Function get_fuzzy_and_msgid_errors iterates lines in the file and checks that they don’t start with “#, fuzzy” or “#~ msgid”. If not, adds the line number to the list of errors.

def get_fuzzy_and_msgid_errors(file: list) -> tuple:
    """Return numbers of lines that starts with '#, fuzzy' or #~ msgid"""

    errors = []

    for line_number, line in enumerate(file, 1):
        if line.startswith('#, fuzzy') or line.startswith('#~ msgid'):
            errors.append(line_number)

    return tuple(errors)

get_locale_files_errors function

get_locale_files_errors produces a dictionary with the language of the file django.po and his mistakes.

def get_locale_files_errors() -> dict:
    """Return errors for ru and en django.po files"""

    with open(RU_DJANGO_PO_PATH, 'r') as file:
        ru_file = list(file).copy()

    with open(EN_DJANGO_PO_PATH, 'r') as file:
        en_file = list(file).copy()

    errors = {
        'ru': get_fuzzy_and_msgid_errors(ru_file) + get_msgstr_errors(ru_file),
        'en': get_fuzzy_and_msgid_errors(en_file) + get_msgstr_errors(en_file),
    }

    return errors

test function

Let’s move on to the test itself. Here is what it does:

Goes through the dictionary with language and errors django.po file;

Checks that the number of errors is 0.

class LocaleTests(TestCase):
    """Tests for locale files"""

    def test_locale_files(self):
        """Test en and ru django.po files"""

        for language, errors in get_locale_files_errors().items(): # (1)
            with self.subTest(f'Errors for {language}: {sorted(errors)}'):
                self.assertEqual(len(errors), 0) # (2)

Test run

Let’s check the test for the following django.po file:

# ...

msgid "Фраза с корректным переводом"
msgstr "Phrase with correct translation"

msgid "Для этой фразы нет перевода"
msgstr ""

#, fuzzy
#| msgid "Фраза 1. Раньше она была такой"
msgid "Фраза 2. Теперь она поменялась на такую"
msgstr "Phrase 1. It used to be like this"

#~ msgid "Эта фраза нигде не используется"
#~ msgstr "This phrase is not used anywhere"

The result will be the following:

Testing JavaScript

In this part, using the example of our two tests, I will briefly talk about how to test JS. For more detailed information, I recommend reading this article from learn.javascript.ru.

Let’s start by creating files. We need scripts.js with tested features and tests.htmlwhich will contain and run the tests.

For JS tests, we use a framework mocha. It allows you to run tests like consolesas well as in browser. We will focus on the second option.

Functions tested

For these tests, I took the most simple functions. Here they are:

function getFileExtension(file){
    return file.substr(file.lastIndexOf('.') + 1);
}

function isExtensionValid(extension){
    return extension !== 'exe' && extension !== 'i';
}

The first gives the file extension. The second checks if it is valid. We do not accept files with the extension “i” or “exe”, so such files will be considered invalid. Both functions are used to validate the feedback form.

Full test.html code

Full file code tests.html looks like that:

<!DOCTYPE html>
<html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta http-equiv="X-UA-Compatible" content="IE=edge">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">

        <title>JavaScript tests</title>

        <!-- Our scripts -->
        <script src="https://habr.com/ru/company/pvs-studio/blog/652951/scripts.js"></script>

        <!-- Load Mocha and it's styles-->
        <script 
            src="https://cdnjs.cloudflare.com/ajax/libs/mocha/2.1.0/mocha.js">
        </script>
        <link rel="stylesheet"
            href="https://cdnjs.cloudflare.com/ajax/libs/mocha/2.1.0/mocha.css">

        <!-- Setup mocha-->
        <script>mocha.setup('bdd');</script>

        <!-- Load chai and add assert -->
        <script
            src="https://cdnjs.cloudflare.com/ajax/libs/chai/2.0.0/chai.js">
        </script>
        <script>const assert = chai.assert;</script>
    </head>

    <body>
        <script>
            // Constants for testing

            // For getFileExtension test
            const filesAndExtensions = {
                'file.doc': 'doc',
                'file.pdf': 'pdf',
                'test.i': 'i',
                'test.exe': 'exe',
            };

            // For isExtensionValid test
            const validExtensions = ['doc', 'docx', 'pdf', 'xls',];
            const invalidExtensions = ['i', 'exe'];

            // Tests

            describe("getFileExtension", function() {
                for (let [file, expected_extension] of
                Object.entries(filesAndExtensions)) {
                    it(`${file} has ${expected_extension} extension`,
                    function() {
                            assert.equal(
                            getFileExtension(file), expected_extension
                        );
                    });
                }
            });

            describe("isExtensionValid", function() {
                describe("The extension is not .i or .exe", function() {
                    validExtensions.forEach(extension => {
                        it(`${extension} is valid extension`, function() {
                            assert.isTrue(isExtensionValid(extension));
                        });
                    });
                });

                describe("The extension is .i or .exe", function() {
                    invalidExtensions.forEach(extension => {
                        it(`${extension} is invalid extension`, function() {
                            assert.isFalse(isExtensionValid(extension));
                        });
                    });
                });
            });

        </script>

        <!-- All tests results will be in this div -->
        <div id="mocha"></div>

        <script>mocha.run();</script>
    </body>
</html>

Let’s figure out what’s going on here.

head tag

Let’s start with a tag head. In it, we perform the following actions:

We load the tested functions;
We connect mocha and its styles. They will be used to display tests;
We set up mocha, indicating which testing methodology we will use. BDD Allows the most complete description of what the test does. You can read about all methods here;
Loading the library chai. It contains test validation functions (should, assert And expect). Although more suitable for BDD should And expectwe will use the method familiar to us from Python assert. So we put it in a constant.

<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">

    <title>JavaScript tests</title>

    <!-- (1) -->
    <!-- Our scripts -->
    <script src="https://habr.com/ru/company/pvs-studio/blog/652951/scripts.js"></script>

    <!-- (2) -->
    <!-- Load Mocha and it's styles-->
    <script
        src="https://cdnjs.cloudflare.com/ajax/libs/mocha/2.1.0/mocha.js">
    </script>
    <link rel="stylesheet"
        href="https://cdnjs.cloudflare.com/ajax/libs/mocha/2.1.0/mocha.css">

    <!-- (3) -->
    <!-- Setup mocha-->
    <script>mocha.setup('bdd');</script>

    <!-- (4) -->
    <!-- Load chai and add assert -->
    <script
        src="https://cdnjs.cloudflare.com/ajax/libs/chai/2.0.0/chai.js">
    </script>
    <script>const assert = chai.assert;</script>
</head>

Constants for tests

Let’s move on to body and his tag script. The first thing we do is create constants for tests.

// Constants for testing

// For getFileExtension test
const filesAndExtensions = {
    'file.doc': 'doc',
    'file.pdf': 'pdf',
    'test.i': 'i',
    'test.exe': 'exe',
};

// For isExtensionValid test
const validExtensions = ['doc', 'docx', 'pdf', 'xls',];
const invalidExtensions = ['i', 'exe'];

getFileExtension Function Tests

Function tests come after constants getFileExtension. Here is how they are created:

Create a block with a name getFileExtension.Function describe allows you to separate tests from each other, placing them in different blocks. Those. we can split tests for getFileExtension and for isExtensionValid. Then they will be displayed separately in the browser. First argument describe is the name of the block. It can be anything, as it is only used when displayed in the browser. The second argument is a function that performs tests for this block;
Looping through the elements filesAndExtensions and get the file along with its expected extension;
In function it (it creates one test) we pass the description of the test and the test itself as a function;
At assert-a call the method equal. Its essence of work is the same as for assertEqual in python. In this case, we are comparing two extensions: one obtained with getFileExtension and expected.

describe("getFileExtension", function() { // (1)
    // (2)    
    for (let [file, expected_extension] of Object.entries(filesAndExtensions)) {
        it(`${file} has ${expected_extension} extension`, function() { // (3)
            assert.equal(getFileExtension(file), expected_extension); // (4)
        });
    }
});

The output of these tests will be:

In this case, all 4 tests were successful.

isExtensionValid Function Tests

Now consider the tests for the function isExtensionValid. In them we:

Create describe with the title isExtensionValid;
We share our describe two more blocks, since we have two different types of tests: for valid and not valid extensions. In the first function isExtensionValid must return truein the second false. Therefore, it would be wise to separate these tests. Good describe allows you to create nested blocks;
We go through the valid extensions and create a test for them that checks that isExtensionValid returns true;
We go through not valid extensions, we also create a test for them, but now it checks that isExtensionValid returns false.

describe("isExtensionValid", function() { // (1)
    describe("The extension is not .i or .exe", function() { // (2)
        // (3)
        validExtensions.forEach(extension => {
            it(`${extension} is valid extension`, function() {
                assert.isTrue(isExtensionValid(extension));
            });
        });
    });

    describe("The extension is .i or .exe", function() { // (2)
        // (4)
        invalidExtensions.forEach(extension => {
            it(`${extension} is invalid extension`, function() {
                assert.isFalse(isExtensionValid(extension));
            });
        });
    });
});

The output of these tests will be:

Running Tests

All tests will be displayed in the block div from id mocha. And to run them, you need to call the command run.

<!-- All tests results will be in this div -->
<div id="mocha"></div>

<script>mocha.run();</script>

Having opened tests.html in the browser, you will get this picture:

In addition to our tests, a panel is displayed at the top right. In it you can see:

The number of successfully completed tests;
Number of failed tests;
Runtime of all tests;
How many tests (in percent) have already been run.

You can run a specific test using the button to the right of its name.

If you click on the name of the block, for example on getFileExtension, the result of running the tests of only this block will open. Also, if you click on a specific test, you can see the command to call it.

If the tests fail, you will see exactly what went wrong.

Conclusion

So, today we have analyzed tests for JavaScript and wrote a check for Russian text on English pages. If you haven’t read the first part yet, I recommend that you do. It contains a lot of interesting and useful things. In the next article, we will look at autorun tests and some useful tools. For feedback or criticism, write in the comments or in my instagram. Thank you for your attention and see you in the next article)

112

How to test a website with Django. Part 2. JavaScript and Russian text on English pages

Testing text translation

Checking the absence of Russian characters on English pages

Imports

Constants

is_russian_symbol function

get_page_words function

test function

Test run

Checking for errors in django.po files

Constants

is_msgstr_empty function

get_msgstr_errors function

get_fuzzy_and_msgid_errors function

get_locale_files_errors function

test function

Test run

Testing JavaScript

Functions tested

Full test.html code

head tag

Constants for tests

getFileExtension Function Tests

isExtensionValid Function Tests

Running Tests

Conclusion

Decoding a signal from a 1988 videophone

Digest of the week from Apple Pro Weekly News (04.04 – 10.04.22)

Iceland plans to become a hub for data center construction, and it looks like so far everything is going according to plan

Why I left React Native and switched to Flutter: Part 1

The book “Algorithms informally”

How to batch convert images and videos to gifs with FFmpeg bypassing the CLI

Leave a Reply Cancel reply

Testing text translation

Checking the absence of Russian characters on English pages

Imports

Constants

is_russian_symbol function

get_page_words function

test function

Test run

Checking for errors in django.po files

Constants

is_msgstr_empty function

get_msgstr_errors function

get_fuzzy_and_msgid_errors function

get_locale_files_errors function

test function

Test run

Testing JavaScript

Functions tested

Full test.html code

head tag

Constants for tests

getFileExtension Function Tests

isExtensionValid Function Tests

Running Tests

Conclusion

Share this:

Similar Posts

Leave a Reply Cancel reply