Jump to content
exile360

Accurate Testing is Hard - Science in practice

Recommended Posts

Throughout the security community, and particularly at Malwarebytes, we've seen discussions involving perceived flaws in the methods used for testing and comparing the efficacy of anti-malware and antivirus applications.  The reason that this is important is because it impacts how a product is perceived, particularly compared to other products in the same space in the market (such as antivirus, anti-malware, anti-spyware, internet security etc.) and many users when seeking protection for their systems will see such tests/comparisons as well as references to them when, often factoring those results heavily in their final decision on what to use to protect their devices.  This places a great burden on anyone trying to test, compare and competently review such applications and it also means that if the methods being used for testing are not only scientifically methodical so that the products are tested fairly with repeatable results, but also truly representative of how users will encounter threats, attacks and malware in the real world, then they are doing a disservice to the public in publishing their results.  It also brings their methods into question by anyone knowledgeable enough to understand the methods being used in their testing as well as having an understanding of how real malware infects devices in the wild meaning that any user who simply goes online to seek an answer as to what product(s) to use to protect their devices, they might get a completely different answer depending on who they ask with no truly authoritative source of information and reference.

You can get further perspective of this issue by reading the article Why Malwarebytes decided to participate in AV testing from Malwarebytes Labs blog.

To give a real-world example of a situation where a product reviewer is trying to do things the right way by extensively analyzing proper testing methodologies and learning the details of how to account for variables and provide accurate information, taking their responsibility as a product reviewer very seriously, look no further than a tech channel known as Gamers Nexus and their recently published video on properly testing CPU coolers; a project they have been working on for several years just to ensure that the results they are providing are accurate and can be counted on as a valid scientific reference:

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...

Important Information

This site uses cookies - We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.