What lies under the AV-TEST Certified nameplate?

A lot depends on the effectiveness of the antivirus. After all, even one unrecognized threat can theoretically lead to serious damage to the user. And annoying false positives interfere with the main tasks and sometimes become a real problem for company employees and private users. For us, the presence of antivirus in the data protection system has already become the norm. However, some users want to use a different antivirus. There is no problem here and you can turn off Acronis Malware Removal Tool and use any antivirus instead.

However, people who do not have enough time, competence or experience to choose an antivirus successfully use the built-in solution. And AV-TEST’s comparisons of antivirus engines that can be provided by service providers to their customers showed a very decent level of testing the main threats and scenarios. It is with this test that we want to introduce you in this post.

Testing structure

In the fall of 2021, AV-TEST published the results of an assessment of four security systems suitable for service providers. The experts reviewed products from Acronis (Acronis Cyber ​​Protect 15), Fortinet (Forticlient 7), N-able (Managed Antivirus 6.6) and Webroot (SecureAnywhere Business 9). The tests were carried out for the Microsoft Windows platform, namely, on the basis of Windows 10 Professional 64-bit, with an active Internet connection and with the latest updates installed before the very start of the test.

The solutions were evaluated according to two criteria – the degree of protection (Protection) and usability (Usability). This is an AV-TEST standard certification, which marks the ability of solutions to block the latest cyber threats, as well as avoid detection errors and not interfere with users in doing their work.

Real-world protection

To assess the capabilities of all participants in the test, each product was allocated its own workstation with installed Windows 10. And an attack was organized on each of the test benches using 173 malicious URLs. This is a very effective technique, because many modern threats enter the system precisely after clicking on a questionable link in a browser or from an e-mail message.

In this test, none of the systems showed 100% response. In particular, Acronis achieved 98.84% of the result without completely removing 2 out of 173 threats. And although at the same time there was no danger to the system, the result did not come out 100%. Fortinet and N-able software received 97.11% each, not having completed 5 test cases each. Webroot’s solution only identified 155 out of 173 malicious URLs, with a detection rate of 89.66%.

Malware detection

Scanning on demand or on event is also an important tool. The capabilities of the antivirus in this case determine its potential to detect malicious executable files present in the system, the so-called portable executable (PE). The AV-TEST test uses a pattern when protection systems are asked to check files that are less than 2 weeks old – that is, to find potential threats that can infect a computer right now.

At this stage, the protection systems scanned their workstations, each of which contained 4,577 malicious files. All of them were already known and widespread. The scan was supposed to show the level of static threat detection.

After the static stage, AV-TEST specialists collect the remaining (that is, undetected) files and launch them for execution. In this way, the potential of antiviruses for dynamic detection is checked. This scan is performed several times.

In this test, Acronis and N-able products scored 100%, successfully recognizing all 4,557 files. Fortinet and Webroot also showed good detection rates of 99.98% and 99.82%, respectively.

Minimum number of false positives

The “Usability” parameter was assessed in two stages. The test was divided into two parts – a static assessment of the number of false positives on a previously prepared test set, and a dynamic assessment of the response to user actions that simulate real work on the Internet.

For the static test, the researchers selected three sets of files:

1. Clean Windows and Office files (575 717 files)

2. Clean files of typical business applications (24,912 files)

3. Clean files from third-party developers (318,120 files)

False positives on the first file set are very bad, because they indicate the ability of the solution to disrupt the operation and stability of the system. The second and third sets, of course, are also desirable not to block, but false positives are no longer critically unacceptable here.

Therefore, it is not surprising that neither system found “fake” threats in the first and second set of files, but in the third test each of the systems made its own number of errors. In particular, Acronis’ solution incorrectly marked 1 file, Fortinet and Webroot products gave 5 and 7 false positives, respectively.

The dynamic test meant typical user behavior: downloading clean software from the Internet, installing it and using it. During these actions, cyber defense solutions were turned on, and analysts watched whether there would be false positives, and maybe even blocking legitimate actions.

The first part of the test was labeled “High Priority”. It included copies of widespread software such as Adobe Reader, Google Chrome and Java (19 different programs in total).

The second part of “Standard Priority” included less common software that not all users are familiar with (44 different programs).

I must say that almost all test participants showed excellent results here: Acronis, N-able and Webroot got the maximum without spoiling a single nerve of users. Fortinet’s software only went wrong once, misidentifying a single installer.

Conclusion

In conclusion, I would like to note that the testing was certainly carried out in private and covered only 4 security systems that MSPs can use to provide cybersecurity services to their customers. In practice, however, large-scale AV-TEST tests follow the same pattern. And if your antivirus or protection system has the “AV-TEST Certified” nameplate, then the installed solution has passed similar stages of testing and showed good results.

Similar Posts

Leave a Reply