Robot Framework vs Pytest

I am a strong supporter of the Robot Framework. I already wrote on Habré that it can be used to solve almost any test automation problem, especially when development is in Python. In the same article, I mentioned that the company uses Pytest on related projects. I had to get to know this tool quite closely, so now I am ready to make a full-fledged comparison with the Robot Framework, of course, from my personal bell tower.

By the way, the idea of ​​the article was prompted by one of the comments to the previous article, in which the reader compared jUnit and Robot Framework for a Java project. In my opinion, in these conditions, the Robot Framework (based on Python) was initially not very applicable. But I liked the idea of ​​comparing the closest jUnit analogue in the world of Python (the same Pytest) and Robot Framework.

What are these tools?

Pytest

In fact, Pytest is an implementation of the xUnit framework for Python. If you’ve worked with jUnit or nUnit all your life (the two most common representatives of this family for Java and .NET, respectively), then in Pytest you will find exactly the same standards – you will quickly understand what is happening, you will follow the usual approaches to writing unit tests. xUnit frameworks are add-ons over a programming language or libraries in them that are convenient for developers to use. Due to this, they are quite common and popular.

To understand the situation, together with Pytest, as with many xUnit frameworks, it is necessary to use a log generation tool. Allure has traditionally been used in this role. Probably, today it is the standard for autotest logs that business wants to see. But you need to understand that Pytest is developed by its own community, and Allure is developed by its own, and the vectors of these teams may at some point turn in different directions.

Robot Framework

Unlike Pytest, Robot Framework is a domain specific language (DSL) – a language specific to its domain. It’s not Python, although it’s built on top of it. But in Python, you can write anything you want for the Robot Framework. And all the features I’m talking about here (as well as integrations, settings) are available out of the box.

Robot Framework is not that convenient for developers, as it requires a deeper dive. In fact, this is a different approach to writing autotests. Like Cucumber for Java. But the Robot Framework independently generates detailed reports (we will talk about this later), i.e. in fact, it also contains a layer for generating reports, and the product is developed by one community – indivisibly and harmoniously.

By the way, the community is open and quite responsive. He has an open channel in Slack, where anyone can ask a question about the Robot Framework and get an answer to it. I myself sometimes answer there. I even contribute to Robot Framework sometimes.

Arguments in favor of the Robot Framework

No unnecessary restrictions on names

In Pytest and a number of other xUnit frameworks, the test case name must always start with test. It seems logical to me when the test method is used. But otherwise, naming restrictions don’t help the process.

I suppose that these limitations in Pytest came from a desire to separate what is related to test cases and what concerns common methods called from tests. Since Pytest is designed for unit testing, the real and test methods of an application can be in the same class, so there can be confusion.

But in my opinion, the problem here is precisely in the chosen approach, which “drags” along with it the fact that in the 21st century we have to put some kind of restrictions on the names of test cases.

Robot Framework is not in any way geared towards unit tests. It has a Keywords section, and the tests themselves can be called whatever you want. It seems to me that it is more convenient to work this way. Moreover, in the Robot Framework, any entities, including keywords, can be called in Russian. Having chosen the names correctly, the text of the test can be turned into a story (“Open the site, enter your login, make sure that the result is such and such”). This allows you not to write long comments about what is happening there. Any development contributor or manager who has never gone into the depths of testing can open this test and understand it. And time is saved, and understanding of the situation remains even after six months.

Suite setup

It happens that a set of test cases (test suite) requires the same type of settings, for example, the execution of a method that prepares some data, creates entities and stores their identifiers for use in tests. The Robot Framework provides separate suite setup settings that apply to all tests in a suite. There is a separate suite teardown that runs after all the tests in the suite, as well as similar settings for each test case (test setup and test teardown). It is convenient, logical and does not require the invention of bicycles.

The xUnit frameworks have annotations to replace the suite setup, while Pytest has fixtures with scope=”class”…

In Pytest, test cases can be just methods (and then they have no suite setup – i.e. no single preparation). If we need uniform preparation, we can wrap these methods in a class. But if we make a fixture with scope=”class” for this class (i.e., we will try to implement the suite setup), then we will get a separate instance of the class for each test, so that the data from the suite setup will not get into the test cases in any way. The individual instances are probably created with the assumption that data from different test cases should not interfere with each other. But because of this, setting up an environment for running tests is much more difficult than in the Robot Framework, where a suite setup is provided a priori.

Usually this question in Pytest has to be bypassed by creating a separate fixture where the data is loaded. This fixture is subsequently pulled into the tests. Another way is to use Python tools by creating a static field for the class that will be common to all its instances (for example, self.__class__.test_id = 2). But in my opinion, this is also a crutch – you should not access the fields of a class through an underscore from outside.

Logs

As I noted above, Allure is most often used to generate logs in Pytest. It is beautiful and fashionable. But due to the instance feature described above, Allure does not understand what and how is related to the test. The report does not include actions from the suite setup. We have to work around this by writing handlers. And from my point of view, this is no longer just a crutch, but a real crutch.

By the way, other xUnit frameworks have this problem too.

Against the background of Pytest + Allure, the Robot Framework logs are as detailed and even redundant as possible. They even include things that you never think about – Robot writes everything you do. This log makes it much easier to catch floating errors that cannot be easily reproduced. You know exactly where and what value the variable had, what API was called. Often you don’t even need to restart the test to understand what’s going on. For Pytest, in such complex cases, you have to come up with tools that help generate a log, like in the Robot Framework.

Syntactic sugar

In the Robot Framework, there is no need to come up with complicated constructs. There are expressions that save you from unnecessary code.

I like to write keyword wrappers and wrap other keywords in them. For example, in the keyword that takes the ID from the API response, you can slip a keyword that pulls different APIs (if the company has coding standards, then the API responses will be similar – the ID field will most likely be in all).

For example, you can write: “Id from entity creation”. Here “Id from” is a wrapper keyword, “entity creation” is a keyword that calls the API and gives the entire response. Alternatively, you can write “Identifier from table creation”, where “table creation” is another keyword.

For me, writing in this way is more convenient and understandable. And if you change the keyword from “creating an entity” to “creating an entity”, then the construction will be read even from a literary point of view (“Identifier from creating an entity”).

Tagging

In the header of the log, Robot Framework shows statistics for each tag. If you get your tags right from the start, you only need to look at these statistics to see what the problem is without bothering yourself with what’s going on inside. I wrote about this in a previous article. In the same place I mentioned that tags can be bound to bug IDs in Jira or in any other tracker that you use. Just recently I had a case when on one of the supported tests, which I forgot to think about six months ago, old forgotten bugs tied to tags “worked”. And it took several seconds to find them, instead of minutes and hours.

There is tagging in Pytest, but it is not logged. Again, you need some kind of crutches to do it yourself. Therefore, as far as I know, almost no one does this.

By the way, Allure does not provide the ability to display statistics on tags. Probably, if such an opportunity appeared in it, in my eyes it would bring the Pytest + Allure bundle closer to the Robot Framework in functionality. The plus would be that the Pytest + Allure bundle would not require conservative developers to bother with new DSLs. Unfortunately, the emergence of such tools is unlikely due to the fact that Pytest and Allure are developed by different groups.

Arguments for Pytest

There are only two arguments in favor of Pytest on my list. But from a supporter of another instrument, they should sound weighty.

Debugging

One of the readers wrote about this in the comments to the previous article. Unfortunately, you cannot pause execution in the Robot Framework to see the values ​​of variables. This is a fact that can be written into the plus of Pytest.

But, in my opinion, this is not necessary. Everything that could be viewed in this way is available in the logs. It’s just that to work through logs, you need to change your thinking a little, change the developer’s paradigm (accustomed to a certain debugging scenario).

Pytest parametric tests

I found something that Robot Framework does not have – thanks to parameterization, Pytest can create test cases on the fly. In Robot Framework, 10 tests will always be 10 tests, no more, no less. In Pytest, in some cases, a single parametric test can cover a huge number of cases.

On the example of a real project. Suppose we have an API with two parameters, each of which can take several values ​​(for example, one takes 7 values, the other 10). And these meanings do not exclude each other. In accordance with the theory of testing, in this case, it is necessary to select several cases that more or less uniformly cover the grid of 70 “intersections” (pair-wise method). But I, using the product method from the itertools module (which multiplies lists), wrote a test setup that prepares 70 data combinations, and then goes to the API and provides that utopian exhaustive testing. When another option appears in the initial data, I just need to add a line to one of the initial lists.

This cannot be done in the Robot Framework. There you can write a test pattern that takes two values ​​and make 70 calls. As new values ​​appear, the number of calls will have to increase.

In general, there are more advantages for the Robot Framework – I remain an ardent supporter of it. But, of course, this is just my opinion.

The author of the article: Vladimir Vasyaev

PS We publish our articles on several sites on the Runet. Subscribe to our pages in VK, FB, Instagram or Telegram channelto stay informed about all our publications and other Maxilect news.