Test Solution Architect: who is it and when is it needed

Every day in the field of IT there are more and more new tasks, including in the field of testing. Previously, a tester just needed to test according to requirements (or without them), but now he needs to first understand how it can be tested at all, what technologies are needed for this, what can be automated, and how to include a release cycle in all this disgrace and etc. Who should answer these questions? Who will communicate with the customer and clarify the requirements? Who will create the approaches and testing architecture, requirements?

Working as a leader and coordinator of testing on projects for large companies and resolving all these issues over the course of three years, I realized that it is important to still attract an individual who will answer the main question: “How to conduct testing?”.
I conducted a small investigation and found that such a role already exists, and it is called Test Solution Architect (TSA), but few people know about it. And the description of TSA vacancies on employers’ websites amazes with their list of duties and skills, but I think that this is more likely due to a lack of understanding of who the TSA is.

Based on my experience in this direction, I decided to show by the example of one of the real projects who TSA is and when it is needed.

Brief Project Description

The objective of the project was to replace one database with another, more precisely, Cassandra and its appendage in the form of FilloDB were replaced by SnowFlake, in the business process of daily, hourly and even minute-by-minute delivery and data processing flows. There were more than 50 such flows, and as planned by the architects, this was supposed to solve a huge number of problems related to performance, data loss, maintenance costs and the purchase of additional licenses for Cassandra, etc. But which of these expectations were met and which are not – this is the topic of another article.

What test roles were on the project?

Historically, the following roles have been distinguished in testing: a manual tester or a functional tester, a testing automation tool (the one who writes the code and tests), and a testing manager (the one who solves all other issues). Our project was no exception. The project involved:

• 1 Test Leader or Test Lead
• 40 manual testers who worked with scrum teams (7 teams)
• 2 developers (this is rather an exception, because developers do not like to work on automation tasks) and 2 test automation
• 2 testers who did stress testing
• 1 functional tester for testing products that was developed by developers and testing automation

Manual testing

The main goal of functional testers was to test the application and say whether it meets the requirements of the customer. For this, testers usually:

1. Create test plans.
2. Create a test strategy.
3. Create test cases with scheduled steps (with the expected and current result).
4. Perform test-cases and conclude on the quality of the application.

We did not have customer requirements or system descriptions. There was only the old system and the new one and the wish: “Make it work, but with a new base.” Therefore, we could only use the like-for-like approach – comparing the results of the old and new systems. Everything was complicated by the fact that we worked with a production system and daily updated data. Unfortunately, we could not start our system at the same time as the production system, and started it later, and this led to the fact that the data in the old and new systems were different.

At this stage, questions began to arise:

1. How to conclude that our system worked correctly if, due to a time shift, we get results that are different from those we get in the old system?
2. Can we get away from data production and work on stable data? What are the risks?
3. And if we nevertheless come to stable data, then how to prove that our system will work correctly with daily updated data?

This is only the tip of the iceberg of the list of similar questions that arose on the project during functional / manual testing.

Test automation

Testing automation engineers (and developers) had the goal of writing applications that could automatically test and / or facilitate the work of functional testers:

• self-tests that would be integrated with CI / CD;
• applications that helped test functional testers;
• and other developments that were needed for the internal needs of the project.

On the project, all applications that were developed for testing should have been a separate product that obeyed all the development rules and would have been delivered to the customer as a result. And, accordingly, questions arose:

1. Who together with the customer will develop non-functional requirements for testing applications? Solution Architect said that this is not their job, as SAs work with the application that was ordered by the customer.
2. Who will develop the functionality requirements for testing applications? There are obvious requirements, but there are non-obvious ones. For example, do we need to store the logs of our applications and analyze them?
3. Who will work out the environment for applications with the client and fix these requirements? For example, specifically on this project there was a restriction on spark 1.4, and this was discovered only after creating the application.

And this is also far from all the issues that arise on the project in terms of test automation.

Test Leader, aka Test Lead

Test Lead should organize the testing processes on the project. Its functions included the distribution of tasks, holding internal meetings with testers, organizing work with the customer (finding out details, reviewing the results), etc. That is, he was focused on the current process and solving daily problems / tasks, and not in working out the tasks of the approach, strategy, testing architecture.

If we are talking about the technical Test Lead, then it was in charge of technical solutions: organizing the development of applications for testing as a process of creating a separate software product that obeys all development rules, for example, creating unit tests, integration with CI / CD processes and etc.

On our project, these roles were played by two people, and each schedule looked like this:

In both cases, Test Lead is busy working on the already chosen testing strategy, but no one is responsible for choosing the strategy itself, justifying this choice to the client, adapting it to changes and other issues. For instance:

1. Development of a plan with the client to enter the stage of acceptance testing by the client (UAT).
2. Studying the requirements with the client when it is considered that the functionality can be transferred to the UAT.
3. Study with the client of the assumptions of testing on different types of environments (a very delicate point). Since usually the test environment does not correspond to production, all questions related to testing on another environment should be worked out with the client.
4. Study with the client of an acceptable discrepancy of metrics in various types of testing. For example, what result does a client expect to see during performance testing? Maybe it will be enough for him that the system works no worse.
5. Collecting all possible metrics in numbers, for example, data discrepancies for this table is permissible by no more than 10%, or the script should run no more than eight hours.

What is wrong here?

The questions above are usually laid on the shoulders of Test Lead, but there is a flawed approach:

1. They are not taught the processes in which Test Lead should work out all such issues with the customer. Most often, an experienced Test Lead understands this and, perhaps, even knows how, but most likely, such an important part as the goal of the business or specific improvements remains outside the scope of its understanding. Accordingly, incorrect priorities may be set.
2. Test Lead usually enters a project when development is already underway or just starting, i.e. he is faced with the conditions that there are already some agreements or even everything has already been fixed. It is lucky if SA is a sufficiently experienced and competent person and goes towards testing – it will help to adapt the initial agreements with the customer to the needs of testing. In another case, the lead has to reinvent the wheel.

And this is not all the problems that fall on Test Lead when there is no TSA.

So who is this TSA and why is it needed?

Test solution architect – This is a person who considers the task from the point of view of business, information and technology, agrees and works out requirements with the customer, draws up a roadmap with deadlines and works out solutions at the interface level.
Projects now dictate other conditions. Projects have become larger, more complicated in technical and process terms, can cover several industries and technologies. Testing has become not only a service, but also a supplier (developer) of software for the customer. Therefore, in testing there is a need to highlight a similar role. Moreover, this person should enter the project together with SA, who is involved in the production application, and begin to work on all issues at the same time.

Specifically, on our project, the role of TSA was played by Test Lead, which joined the project 3 months after the start of development, and this entailed a lot of additional work on harmonizing requirements, environments, figuring out the vision of the final test results from the customer. As a result, the project did not meet the deadlines for most of its life, and the customers were unhappy with the supplies and the test result. Not because the work was done poorly, but because the result produced by the team did not meet the expectations of the customer.

When is TSA not needed?

It is clear that such a role is not needed on all projects. Below I propose the simplest way to determine the need for TSA on a project, depending on how the customer sees the testing approach.

The first type of project – the customer gives the setting of the testing processes at the discretion of the company that he hired.

In this case, the TSA enters the project from the moment it starts working on it, together with SA, develops testing requirements, develops a high level testing scheme, decides what will be automated / developed as separate applications, determines the requirements for these applications and, of course, coordinates with the customer, prioritizes project objectives in accordance with business expectations.

The second type of project – the customer has his own understanding of testing, a clear picture and streamlined processes.

In this case, TSA may not be needed; testing is only required to fit into the existing picture of the world and support it. But if the understanding is superficial, then the TSA needs not only to perform all those actions as for the first type of projects, but also to prove to the customer that this is all necessary.

The third type of project – the customer does not need TSA services.

The reasons may be different:

1. A small and short-term project with simple functionality.
2. The customer has his own TSA.
3. The customer refused to test, etc.

TSA Skills

Solution Architect practice has been successfully applied in development for a very long time. Based on this successful experience, and also taking into account the volume and complexity of projects, issues that exceed the Test Lead area of ​​responsibility, we can say that highlighting the role of TSA is a natural development of events. Moreover, the TSA must be trained in the same processes, techniques and approaches as the SA.
In short, in my opinion, the TSA should have:

1. With technical knowledge, both in general and in the domain area where he works, to know the problems of this area, typical errors, problems and ways to check them.
2. Good communication skills, be able to draw the customer’s attention to testing issues, work out testing requirements, testing approaches, testing reporting and many related issues, such as the testing environment.
3. Leadership skills and be proactive because Testing very often try to leave for later.
4. Good knowledge of testing in general, test documentation, testing processes, testing reporting, approaches, etc .;
5. Good knowledge of software development rules: release policies, versioning policies, understand CI / CD processes, etc.

Based on the foregoing, TSA should have all the same knowledge as SA, but be focused on the tasks of providing the customer with a quality product. The TSA’s work is complicated by the fact that one often has to change the customer’s opinion about testing as exclusively about the internal kitchen of the project.
It was precisely the knowledge that I received at Solution Architect school that allowed me to restore customer confidence on the project that was described above, solve many testing issues and successfully complete the delivery of all functionality almost on time, as well as establish processes and sign contracts for future work.

Conclusion

The IT industry is developing, new tasks appear (challenges if you want), and in the case of testing, the highlighted TSA position has become urgent. Naturally, it all depends on the project, but in projects where a TSA is needed, you need to engage in work at the very initial stage, this will avoid problems in the future and will help the development of the project.