trial by fire on a code refactoring task
Hi all! My name is Sergey Gerasimov, I am a Senior QA Manual Engineer (yes, I’m bragging) at the Petrovich-Tech company.
Imagine the situation: you are a young, handsome and smart tester, sitting quietly, poking buttons, studying the callbacks of your favorite payment services, and then a non-trivial task arises: creating checklists to test all the functionality of placing an order.
It sounds simple, but what lies behind it?
In my past article I talked about payment testing, the test design techniques I used, and in every possible way disavowed pairwise testing. But then evil fate came to me, and today I want to talk about my recent experience of using the “park” in practice.
Conditions and meaning of the task
Let's give a quick background: pairwise testing is a technique test design, which provides complete test coverage. You can read more about it Here. This concludes our description.
Website “Petrovich” is a huge system with a bunch of different products, delivery options for every taste and color in different regions and a wide variety of services, where absolutely every user can choose something for themselves.
And how, if all the conditions and subconditions are present, can you collect the work of the entire order into one checklist?
What already exists: the code has been refactored on the 1C side, which receives information from the site. Now you need to check that all the information that the user enters on his part is transferred to 1C correctly. You need to create a checklist to check all the functionality of the design, without missing out on details.
And yes, perhaps now knowledgeable and wise old-time testers will say: “Seryoga, actually there is a basic principle: redundant testing is impossible, so…”
And I will answer you: “I understand everything, but our task is not to test all possible combinations, but to test all the basic functionality. And based on this, carry out further checks.”
Let's continue. For the task, two test benches with a full set are allocated – a database, space in 1C, etc. There are 4 testers involved in the project: two from the site side, two from 1C.
Planning and distribution of tasks
Your narrator was from the site side. The first thing my colleague and I did was to begin to define the principles by which we would divide areas of responsibility among ourselves. Because “Petrovich” works in a huge number of constituent entities of the Russian Federation, they decided to divide it by city. Why? Because different cities have different conditions, prices, types of goods and other nuances.
I tested Moscow, satellite cities of Moscow, cities of the Russian Federation (where there are no our sales points and partners) and Crafts cities (partners of Petrovich), my friend got St. Petersburg and the entire Northwestern Federal District.
Decomposition of ordering
Next, we proceeded to decompose the checkout. Testers or developers who have encountered the ordering theme in marketplaces understand that almost any order form has hundreds of parameters and thousands of variations. How to organize maximum test coverage?
To begin with, we wrote down all the checkout blocks for which the order is processed. Let's take as an example only the delivery form, although we also have pickup. The checkout blocks included:
Address
Checkbox for arch/underground parking restrictions and the height of this restriction
Availability of object
Availability of the master at the address, his name and telephone number
Comment to the driver
Delivery day
Delivery time (delivery type and time interval)
Additional services (various list)
Promo code for delivery
Payment method
Email
Phone number for calls and SMS, with contact name
Checkbox for operator callback, with the ability to select the day and time of the call
Comments on the order
Availability of free lift
The items on one of the two checkout pages are ready. Now we need to highlight what else can affect our order, since before the checkout there is a basket. Add additional parameters:
City (we divided our areas of responsibility among ourselves)
Type of buyer: individual and legal entity
Loyalty system and its use/non-use
Availability of user authorization
Product type (they are divided)
You feel it, right? I also highlighted and felt the scale of the task.
Most items have their own subconditions: types of delivery and payment, time intervals, spending or accruing points, types of goods and their availability, a bunch of parameters for additional services, etc. To be honest, we were not particularly guided by logic at the beginning and tried to highlight all the parameters for each item. But it was too much. In addition, the task was to create a large checklist in Allura, which was also unclear how to put it together.
At first we estimated the number of cases at 1000 pieces, but later, looking at everything written, we came to the conclusion: we are already talking about tens of thousands of variations.
Of course, this did not suit us! But how then to run such a large volume of checks? This is where we came to pairwise testing.
I really don't like this technique. Using it almost always means a big task with a million parameters that need to be covered. This leads to problems with the tested object. And I'm a lazy person 🙂
Tools: how they were chosen and what they chose
The question of tools arose. Creating cases manually using all the parameters is very time-consuming and tedious. We started studying the “market”. After reading a number of articles about different tools, we decided to settle on two tools: Pairwise and Pict from Microsoft.
Pict is a fairly well-known tool. We paid attention to it when we realized that we had too much input data. Pairwise is the simplest and probably the most popular tool for pairwise testing.
We decided to go with Pairwise. It does not require setup or additional labor costs. Unlike Pict, which needs to be deployed, installed, checked that it works, etc. We could, of course, do this, but we didn’t want to create additional tasks for ourselves.
Problems and solutions: what was abandoned, what was abolished, how the work was structured
So, it was just a matter of making sure that Pairwise digested our data. We won’t go into all the difficulties that caused the instrument to suffer from indigestion, but I’ll give a couple of examples.
In order not to write out a million special conditions, we decided to divide the cities, individuals / legal entities and type of receipt into several separate tables. This will allow us not to touch them when entering data into Pairwise. As a result, we created several tables called like this:
Moscow. Individual Delivery
Moscow. Individual Pickup
Moscow. Entity. Delivery
Moscow. Entity. Pickup
And further on the same principle.
Now more specific. For example, in tables related to delivery, there was initially a “Zone” field. As you understand, any store that provides logistics services has a division of delivery zones. Our “Petrovich” is no exception. And the first thing we encountered was dividing the zone into different parts and testing them.
Since there are 9 delivery zones in St. Petersburg alone (and this is already difficult for Pairwise), we initially decided to reduce the zones to 3: the first, the last and any in between. But in this case, you also need to check negative scenarios, for example, beyond the last zone. We also remembered that there is an option for delivery to a place on the border: some houses may be located at the junction of zones, say, 2 and 3. Screenshot of one of the early tables (something was hidden due to the NDA, but the meaning remains the same):
And now we are back to 5 zones. Which again is hard for Pairwise to digest.
We are faced with the first clarification of requirements. I’m writing to the team lead, hoping that we don’t have to touch every zone:
We agreed that we would leave three zones, and leave two negative ones for exactly two cases per city. And a little later, chasing Excel with cases, we came to the point of abolishing delivery zones altogether. And leave one option “Any” and those same negative options. Why? Because it doesn’t matter to us which delivery zone is signed up: what matters is whether it’s signed up or not. This way we simply excluded an entire column of input data from the Pairwise table.
We did the same with the other columns. For example, “Add. services” we greatly reduced: we left one case with specific services (separately Additional Lifting, separately Unloading, etc.). There were only 9 such cases. We replaced the rest with “Combination of any available or no services.” In order not to upload those 9 cases with different services to Pairwise, we added them manually to our final table:
So we eliminated the remaining columns where there was too little data and which, for example, were limited to the “Yes/No” answer. In the screenshot above you can see examples of such parameters as “Comment to the driver” and “Promo code for delivery”.
We managed to reduce the amount of data entered into Pairwise to several columns: Product type, delivery type, payment and day:
Separately, we have written down the conditions for which there are intersections and restrictions:
Click “Generate Pairwise”:
And we get 40-50 cases instead of the thousand expected for each table:
conclusions
We additionally processed the resulting cases manually. They made rules, thought about additional conditions and added something. For example, negative cases. Thus, we added another plus or minus 3-10 cases to each table. Pairwise testing gave us the opportunity to think about the complex and controversial cases that we discovered in the process of their formation. Some of them were submitted for discussion to analysts.
I took away an important idea from this story: when creating checklists, it is worth taking time to formally double-check them. Often you forget to specify some specific parameter, and a completely crazy case appears.
Despite all the disadvantages, the complexity of processing and the time spent were very pleased with the result. After all, 50 cases for one of 8-9 tables is much better than a thousand 😉
That's all, Senior QA Manual Engineer Sergey Gerasimov was with you. Write your questions about the article in the comments. And by the way, a minute of advertising: come work for us at Petrovich-Tech. We have a lot of interesting tasks, a cool community, and we pay well. So if you love money like me, check out our vacancies herewrite your questions about employment to me in tg – @GersKing, I will answer all questions and promptly pass your resume into the hands of our HR and team leads.
Why am I writing about this here? It's simple – for each IT employee we refer, we are paid a bonus. Did I already mention that I love money? 🙂
