BDD framework for API automation without code

My name is Victor Myasnikov, I am responsible for product quality at Yulia. I graduated from Baumanka and wanted to build rockets, but it didn’t work out with them, so now I professionally build “bicycles” for QA, and I also like to destroy the routine. I will tell you how we designed the BDD framework and why we needed it at all.

Why is this all?

Yula is an ad service available on iOS, Android, VKontakte and the web. When we made the BDD framework, we had a separate QA team that tested the API. To give you an idea of ​​the situation, I will introduce you to one of the testers of this team. All names are fictitious and coincidences are random, but this will help you understand the features, and somewhere, perhaps, see matches with your situation.

“I tried and it didn’t work”

Let’s get acquainted with Andrey, a manual tester. I’ll make a reservation right away, Andrey is a collective image of a manual tester, whose history may seem familiar to you.

He has been a professional API tester for over two years now. Andrey knows many test design techniques and uses them in his daily work. Naturally, he tried automation. Who among us has not tried it at least once when we first started? Initially, Andrey really wanted to automate, but, unfortunately, something went wrong. Therefore, Andrew concluded: “Automation is not needed and everything only complicates. Better to test yourself“.

Mouse clicks, more clicks!

Every day our Andrey comes to work and tests the tasks. Of course, in all teams and companies, the way to solve problems is different, but any task goes through the main stages, for example, a test on a branch and a test on a release candidate. What does it look like?

Andrey takes Postman and clicks, and clicks, and clicks, and … You get the general idea.
Then, at the next stage, he clicks again, clicks, clicks, and so his working day passes.

Since automation did not work out initially, this is the routine that accumulates and increases. Andrei cannot “digest more”, yet he continues to do this every day.

What’s wrong with that?!

It would seem, well, okay. But this approach will run into several problems.

“Gimme the latest collection”

Each person on our API testing team had their own Postman collection. The testers had to somehow exchange them: someone went on vacation, someone tested someone else’s release and they had to inform the other person about it. As a result, all communication in the team comes down to the phrase: “Give me the latest collection“.

And indeed, everything worked like that: the collections were sent to each other in Slack, Telegram or another messenger. A huge number of them accumulated – version numbers exceeded well over a hundred.

green tick Tested

What happens in Jira with this approach? The following happens. The test report looks like this:

Jira is a mess, and it is not clear what useful information the team can get from this. Probably, every QA left such comments at least once. And more to say, I did the same when I started. These reports are really a problem, because no one can reproduce who did what.

What happened to our Andrew? Poor automation training demotivated him. He was tired of clicking, and he was upset. I sat and thought: “Automation didn’t work out, I don’t need it. And now what i can do?”. Many people face this barrier: once automation “didn’t work”, and there are already too many routines.

Three Important NOTs

Initially, when we designed the BDD framework, we set the task don’t scare our Andrew. He no longer had to fear automation. This was the most important thing.

Second: don’t need to learn program it. Otherwise, we will again come across a barrier: Andrey already once failed.

And the third factor: don’t reinvent the wheel. True, I love them very much, but this time they did without them.

Everything for the team

Since we already had a team, we chose technologies and made a framework for it. After looking at the experience of different companies, we decided to use the BDD approach, and took Cucumber as an implementation. And since the automation team and I already knew Java, the question of choosing tools disappeared by itself.

And now for the cause

Designing the future interaction began with the GET method, as did the study of HTTP request methods. Andrew said:I don’t understand anything, but let’s try“. He wondered what we could offer him. Yes, and fatigue from manual clicking has accumulated.

Part 1. GET

As a basis, we took the Postman already used by the team. The main thing to describe:

• The request type is GET.

• Address bar. Everything is simple here, we indicate it in quotation marks.

• Headings. These are key-value pairs, it is convenient to use a table view: add headers |key|value|.

• Send button.

• Checking the response status code: status code 200.

For bores, those who like to find fault with trifles, or just read more about the BDD approach, you can start with a good article hereor look for something yourself, starting, as expected, with wiki.

As a result, we learned how to describe in the Gherkin language the actions that QA did in Postman. We got a description of the system behavior that we expect.

Naturally, one GET will not be full, so the next request we have is POST.

Part 2. POST

How is POST different from GET? He has a body with which there are problems:

1. It can be arbitrarily large in size, so I didn’t really want to enter it into the test just like that, otherwise it would be unreadable. We decided to put the body in a JSON file, which is located in a certain place in the project and will be sent along with the request. For simplicity, we indicate only his name.

2. The file may contain some dynamic elements, for example, ID. Of course, it can be left in the project in this form, but it will be inconvenient to work. Therefore, we presented the JSON file as a so-called template. This means that at the time of execution, actual values ​​are substituted into all dynamic fields. And when designing a test, they will be stored as a link. Thus, before the test, we can say that we have an ID, which at the time of execution will be replaced by the actual value. Naturally, our tests must be executed in several threads, so we made a thread-safe storage of such variables, where only two operations are performed: put and get.

With this simple manipulation, we can make dynamic elements in our JSON files.

Part 3. Checks

We already have a check on the status code of our requests, but this is not a completely fair check, because we are interested in checking the data itself. Therefore, we took the query language in JSONPath, which allows you to find elements in the structure of a JSON document. We have the document itself, and by $ we can get to its root, execute a query $.id and get the element we need, that is, the ID specifically in this JSON file. So, we can make a check that is easy to implement. You can read more about JSONPath in one of the primary sources here.

This implementation allowed Andrey to write the first set of autotests. This is enough to start covering our API, but the matter was not limited to this – we have several stands.

Part 4. Stands

How did the guys (“Andrey”) do at Postman with stands? They created several environments for themselves, for example, dev, releaseand the necessary data was substituted through the variables.

If you google for the phrase “how to make multi-stand”, you will find the same solution almost everywhere: get two files dev.properties and release. properties.

We did that too, because it’s a simple solution. But when they gave the files to Andrey and the team, they noticed that the following entries began to appear in them:

It turned out that a set of numbers are the secrets of a particular booth. Periodically they changed, we had to go to the repository, find files and change secrets. For other QA in the old branches, this all lost its relevance, they had to constantly update the working branches, which complicated the work. And the more such secrets became, the more difficult it was to maintain. There were, of course, exotic solutions – to change property files on the fly, but we found a more elegant solution – Vault.

Vault is a secure service, a key-value store. In it, we brought as many folders as there were stands (in the example above – dev and release), and brought the keys, hosts, and so on. You can read more about Vault here and here ). This made it possible to remove the secrets from the project, and now they can be changed without our participation. This is important because the secret can be changed, for example, by the operation team for security purposes. And now this team knows where our secrets and theirs lie. Let me emphasize that this safely. Secrets are not stored in the project, they are not in the repository. This way we don’t expose them anywhere and we don’t need to maintain them manually anymore.

Part 5. Still without bicycles

Andrey and his team liked the proposed implementation: it became easier to work, and the guys began to come for new opportunities.

The first request came from another member of the team: to allow hashes to be read. Previously, he did this online with an MD5 generator, which was, firstly, not entirely safe, and secondly, not very convenient. This request was easy to resolve.

Second request: new UUIDs are needed. It turns out that the guys had the UUID generator folder open in Chrome, and each time they generated them manually. A BDD framework can handle this too.

But the third question made me think. I was asked to calculate the amount in the project. The question arose: “Do we want to support all the arithmetic ourselves?” After all, if you need a sum, then soon there will be a difference, multiplication, division, and in general all basic arithmetic. And this is a laborious task. I promised that we would do without bicycles, so I began to look for a solution that would close all the Wishlist at once.

What did we find? JEXL scripting engine that provides ready-made arithmetic and the ability to forward any function in Java (you can read more about JEXL here).

JEXL Integration

The integration is very simple: we added it to our pom file, combined the contexts and called the JEXL engine where we needed. As I already wrote, our context implements two basic functions: get and put. JEXL works in a similar way, it has two methods: get and set.

JexlContext jc = new JexlContext() {
@Override
public Object get(String s) {

return Context.getValue("_" + s);
}
@Override
public void set(String s, Object o) {
Context.put("_" + s, o);
}
@Override
public boolean has(String s) {
return Context.has(s);
}
};

We have merged our context with the JEXL context. So now our initial variables are JEXL variables, and tests can do the following:

* assign
| UUID | #{fun:getUUID()} |
| a | #{1+1} |

We can write a function to generate a UUID, pass it to the test, and the person will simply call it. Arithmetic worked in a similar way: now our tests can perform all four operations without our participation. True, Andrei said that it was very difficult. And it really was. I had to work hard and explain, and then write several articles in Confluence, how it works. But after just a few days, the tests began to acquire such checks and everything went much faster.

It became easier for Andrew to work. He overcame the barrier and felt that he could automate tests that would really help him.

Results

For the first three months, we received the following increase in autotests from the team:

In three months, the number of autotests has quadrupled without damage to the team, the release, and practically with a free retest.

Attentive readers will remember about the collection. The guys finally removed them and started using tests as a source of truth. Want to see how the backend really works? The easiest way to do this is in tests.

As a bonus, we received clear reporting. Naturally, we connected Allure, and now instead of comments – we roll and tested – we have a link to the report. Going through it, everyone in the team can see what exactly is happening there.

This is how we got the basis of our framework and a team that can automate without knowing the code and without harming ourselves and others.