Negative testing. What is it and how to use it? Features of using invalid checks
Negative testing, or error testing, is an important part of the software quality assurance process. It aims to test how a system responds to incorrect, invalid, or unexpected data, which helps identify vulnerabilities and improve the reliability of the product. Unlike positive testing, which checks that the system executes correctly given correct input, negative testing examines scenarios in which a user or external factor enters data that deviates from the norm. This article discusses the key aspects of negative testing, its application in different types of testing, and best practices for using this approach effectively.
The main goal of negative testing is to ensure that the system correctly handles incorrect data and remains stable under any input. This helps prevent potential errors that may arise in real-world conditions, where users may make mistakes or enter data that does not match the system's expectations.
Despite the importance of negative testing, its use is not always appropriate for every type of testing. The main types of testing where negative checks are actively used include:
Functional Testing: This is where negative testing plays a key role, checking how the system reacts to invalid data in each individual component or function.
System testing: At the system level, it is tested how the various components interact when invalid data is entered.
Security Testing: Includes checks for resistance to attacks such as SQL injection, XSS and other types of abuse that can be classified as negative testing.
There are types of testing where negative checks are not used or are less significant:
The number of invalid checks per field being tested is often a subject of discussion in different teams. Typically, the number of such checks depends on several factors, including risk, functionality criticality, system complexity, quality requirements, and testing time. Here are some general recommendations and approaches that are used in different teams:
Minimum coverage: Typically, 3 to 5 invalid checks are added per field in standard cases. This includes:Empty value (e.g. submitting a form without filling in a field).Value that exceeds the allowed limit (e.g. text that is too long or a number that is too large).Invalid data type (e.g. text in a numeric field).Special characters (if they are not supported).Values outside the allowed range (if there are range restrictions).
Maximum coverage: In case of critical fields (e.g. password input field, credit card number, etc.), the number of checks can be increased to 7-10 or even more, including additional checks such as SQL injections, scripts, etc.
4.2. Limitations in Commands
Teams with limited time: Teams with limited testing time may use approaches such as “most important checks”. For example, only the 2-3 most critical invalid checks are checked for each field.
Prioritization: Some teams use prioritization to determine which invalid checks are most important and perform them first. Other tests can be deferred or performed later.
Risk-based testing: With a risk-based approach, the team can limit itself to only those tests that pose the greatest risks to the business.
4.3. Policies and Procedures
Documented Standards: Some teams have documented standards that specify how many invalid checks there should be for each field type. For example, all text fields may require at least 3 checks (empty value, value too long, and invalid characters).
Review and Feedback: Teams can conduct regular test reviews where tests are discussed and revised for necessity and sufficiency.
4.4 Tools and Automation
Automated testing: With automated testing, the number of invalid checks can increase because such tests are easier to maintain and execute. Automation allows you to check a wider range of invalid scenarios without significantly increasing labor costs.
4.5. Practices of other teams
In large companies: In large companies or projects with a high level of criticality, the number of invalid checks may be higher, with mandatory coverage of all possible scenarios. There may also be formal restrictions and procedures for each category of tests.
In startups or small teams: Here they may strive for a minimum set of tests to get the product to market faster, with subsequent addition of tests as the product grows.
5. Negative tests and combinatorics methods
Negative testing and invalid checks can be useful in some combinatorics techniques used in software testing. It is important to understand that such techniques help to identify possible defects in the system when it encounters incorrect or unexpected input data.
Here are some combinatorics methods where negative testing is appropriate:
5.1. Boundary Value Testing
Method Description: This method involves checking the extreme values that are at the boundaries of the acceptable range for each input parameter. The boundaries are the points where errors often occur.
Why it is appropriate to use negative testing: Negative testing in this method involves testing values that are outside the acceptable range. For example, if the acceptable range of values for an input parameter is 1 to 100, you would test the values 0 and 101 to ensure that the system handles invalid data correctly.
Example: If you are testing an age input field that accepts values from 1 to 120, negative checks might include entering -1, 0, 121, or 200 to ensure that the system handles invalid data correctly.
5.2. Testing equivalence classes (Equivalence Partitioning)
Description of the method: In this method, the input data is divided into groups (equivalence classes), and it is assumed that all values within one class are treated equally by the system.
Why Negative Testing Is Appropriate: Within each equivalence class, there may be both valid and invalid subsets of data. Negative testing is aimed at checking those subsets that are out of range or contain invalid data.
Example: If the system expects only numbers to be entered into a phone number field, then tests should include entering letters or special characters (such as “abc” or “#%&”) as invalid data.
5.3. Pairwise Testing Method
Description of the method: This method aims to test all possible combinations of parameter pairs, which helps to minimize the number of tests while maintaining sufficient coverage.
Why negative testing is appropriate: Pairwise testing can create combinations that include invalid or undesirable values for one of the parameters. This helps identify errors that arise as a result of incorrect interactions between parameters.
Example: Suppose a system has two input fields: age and income. Negative testing might include combinations where age is negative and income is above the acceptable limit to test how the system handles invalid combinations.
5.4. Minimum wages and Atomarks
Minimum wages: These are minimal tests that cover the most important and critical cases of using the system functionality. Minimal tests may include negative checks if they relate to critical aspects of the system. For example, checking mandatory form fields may include a test for their non-filling.
Atomarks: Atomic tests are more detailed tests that check specific elements or small sections of functionality. Atomic tests typically focus more on both positive and negative scenarios, including testing for invalid data, such as entering invalid values into input fields.
Using negative checks: In both cases, negative checks are appropriate when it is important to ensure that the system handles errors correctly at a basic level. Example: in an atomic for a text field where a phone number must be entered, there may be a check for entering letters instead of numbers, which is a negative scenario.
5.5. Decision Tables
Method Description: Decision tables are a way to organize and test a set of rules and their consequences. They are especially useful when the system has complex logic with many conditions. Using Negative Checks: In decision tables, negative checks can be included to test the behavior of the system when the input data does not match the rules or when invalid data is entered.
Example: If your decision table has a rule that determines a discount on a product based on the order amount and the presence of a promo code, negative checks might include entering an invalid promo code or an out-of-range amount to ensure that such cases are handled correctly.
5.6 State Transition Diagrams
Description of the method: State transition diagrams are used to model the behavior of a system when transitioning from one state to another depending on the input events or data. Application of negative checks: Negative checks in this method consist of checking for invalid transitions between states or incorrect system response to certain events.
Example: In a hotel booking app where the user must first select dates and then confirm the reservation, a negative check might involve attempting to confirm the reservation without selecting dates to ensure that the system handles this case correctly.
Conclusion
Negative testing in the context of combinatorial testing methods helps to identify defects that may arise when incorrect data is entered. This is especially important for ensuring the stability and reliability of the system, as it helps to prevent failures and errors caused by unexpected input data or their combinations. The use of such combinatorial methods with the inclusion of negative tests allows for more complete and high-quality testing, which is the key to stable operation of the software in real operating conditions.
