Why do AV products score so highly in professional tests?

[pondus]

Spotlight on security: Why do AV products score so highly in professional tests?

https://www.av-comparatives.org/spotlight-on-security-why-do-av-products-score-so-highly-in-professional-tests/

 

 

[exile360]

I agree wholeheartedly, however I'd also add that even these so-called "professional" tests are often flawed by today's threat landscape's standards on the basis that the vast majority ignore essential steps in the attack chain of real-world threats and infection attack events, especially when it comes to trapping live exploits, which 9 times out of 10 are where virtually all web based attacks/malware infections now begin.  Even if they do try to replicate aspects of the initial attack phase, the methodology is often flawed because they simply download and execute malicious scripts direct from the malicious hosts rather than being served them through malvertisements embedded in otherwise legitimate websites, through compromised legit sites, or through deliberately compromised malicious sites.  Very seldom do any of these tests begin at the same phase as the real attacks where users have become infected.  This is why I really don't trust any of them to provide a comprehensive view of the efficacy of any modern security products because most security products, AVs included, are now layered offerings which include many components to the protection they provide, often with many of the most powerful and effective aspects of protection being bypassed by the poor methods they choose for testing (just like the "YouTube" testers they say so much about in their article).

A true professional test setup must either a start the attack chain from the real beginning through the identification and use of a truly compromised legit site (which unfortunately often rotate ads/malicious content with legit content so replicating it reliably to test each product is virtually impossible and often play host to polymorphic threats that change too rapidly for the test to be the same and fair to all the products being tested), or b to at least set up a testing server to serve the exploit as a virtualized attack which replicates the same circumstances as the real deal where the threat is found in the wild (less realistic but it at least replicates the basic steps and circumstances of the actual attack chain).  The latter is likely what they will start doing soon as they come to realize that this is how most threats/attacks work these days, and I can see signs of it in their evolving testing methodologies, but they're not quite there yet.  They will basically have to record the attack event then replicate the exploit/script execution/downloader etc. steps of the attack, and possibly modify and/or spoof IPs/URLs to make it look the same as the real thing in order for the test to be truly realistic.  It is not easy, but it is possible, however it's likely far more work for each infection/test than any such organization would be willing to put in which is why things are the way they are now.

Edited August 30, 2018 by exile360

[lock]

3 hours ago, exile360 said:

....tests are often flawed by today's threat landscape's standards on the basis that the vast majority ignore essential steps in the attack chain of real-world threats and infection attack events

Are you familiar with crash tests rating in automotive industry? They use a mannequin instead of a human being , which doesn't press the brake before the crash (like in real life) and doesn't try to steer clear before the crash (like in real life) and doesn't oppose any force in arms (like a real person)

Yet , these tests are standard in industry and the rating is accepted by everybody  even though THEY ARE NOT BEING PERFORMED IN REAL LIFE SITUATIONS.

 

Your whole argument is only to justify why MBAM refuses to participate in ANY tests ; among MBAM shields  the Web protection is the weakest one with extremely high rate of FP ( it seems like the developers just wait for people reaction to "remove the block in next update")

So, without the Web shield , MBAM could perfectly fit in AV Test / AV comparative methodology , but when it did it the result was disastrous. So, it seems more profitable to keep alive a myth rather than prove something.

In over 5 years of running MBAM pro and MSE, Malwarebytes never detected anything , on 3 computers, so I do not believe. 

[exile360]

Actually, my points had nothing to do with Malwarebytes' participation in standardized testing.  I've said before, and I still believe that they should participate, regardless of the legitimacy of the testing methods because many users do give some weight to those tests.

My point was entirely about the fact that many essential modules and components in most products (not just Malwarebytes) aren't being tested at all because of the testing methods being used.  Your analogy about the automotive industry doesn't apply because those things have nothing to do with what I'm talking about.  A better analogy, if you want to go there, would be to remove the bumpers from the car and the airbags and the seatbelts and only test the frame to see how a car will do in a crash because that's the equivalent of these testing methods.  They are literally skipping entire phases of the attack chain and thus failing to test several components of protection, not just in Malwarebytes, but in most security products including most of the AVs they test.