First AV Comparatives test for MBAM v3

[Tinstaafl]

On 4/14/2017 at 4:31 PM, dcollins said:

I'd recommend checking out the following thread where @exile360 explains why av-comparative tests need to be taken with a grain of salt. Not that they're entirely false, just that they don't tend to use real world scenarios.

 

Then I suggest that you actually visit the site and learn what is going on there, rather than making unsupported claims.  They have a test section devoted to "Real-World Protection Tests".  The March 2017 test results are posted here: https://www.av-comparatives.org/dynamic-tests/

"This section contains full product long-term dynamic test reports. These tests evaluate the suites “real-world” protection capabilities with default settings (incl. on-execution protection features). It is our aim to do these tests rigorously. Due to that, these tests are time and resource expensive, so only products chosen for the yearly main test-series are included."

The March 2017 test report states: "The results are based on the test set of 329 live test cases (malicious URLs found in the field), consisting of working exploits (i.e. drive-by downloads) and URLs pointing directly to malware. Thus exactly the same infection vectors are used as a typical user would experience in everyday life. The test-cases used cover a wide range of current malicious sites and provide insights into the protection given by the various products (using all their protection features) while surfing the web."

 

[Benno1024]

The argument that "program A has a higher detection rate in some test than program B; therefore it is pointless to run program B" -- is totally bogus.

You do realize that that is an aggregate result, and it is entirely possible that program B detects stuff that program A missed? (and vice-versa)

 

Edited April 26, 2017 by Benno1024

[DodiGlenn_PCMatic]

Since the test was commissioned by PC Pitstop, I thought it would be appropriate to chime in.

As several have pointed out, this is just one test, and there will be many more to come. Our view is that customers should know how their AV products rate, and should look at multiple tests, not just one, to make that determination. We also feel that vendors should embrace third party testing, and use it as an additional arm of Quality Assurance, to help figure out where there may be defects in the product or in processes.

There are other 3rd party tests, not commissioned by PC Pitstop, which show how MalwareBytes 2.0 performed. One such test was done by MRG Effitas, https://www.mrg-effitas.com/wp-content/uploads/2017/02/MRG-Effitas-360-Assessment-Q4-2016.pdf , called the MRG Effitas 360 Degree Assessment & Certification.

Thanks,

Dodi Glenn

VP, Cyber Security

Edited April 30, 2017 by DodiGlenn_PCMatic

[exile360]

On 4/26/2017 at 10:09 AM, Tinstaafl said:

Then I suggest that you actually visit the site and learn what is going on there, rather than making unsupported claims.  They have a test section devoted to "Real-World Protection Tests".  The March 2017 test results are posted here: https://www.av-comparatives.org/dynamic-tests/

"This section contains full product long-term dynamic test reports. These tests evaluate the suites “real-world” protection capabilities with default settings (incl. on-execution protection features). It is our aim to do these tests rigorously. Due to that, these tests are time and resource expensive, so only products chosen for the yearly main test-series are included."

The March 2017 test report states: "The results are based on the test set of 329 live test cases (malicious URLs found in the field), consisting of working exploits (i.e. drive-by downloads) and URLs pointing directly to malware. Thus exactly the same infection vectors are used as a typical user would experience in everyday life. The test-cases used cover a wide range of current malicious sites and provide insights into the protection given by the various products (using all their protection features) while surfing the web."

 

Right, they're using live malware URLs which is not how actual users in the real world come across these threats.  They get redirected to those URLs by exploits and similar tactics that download the droppers (the malware etc.) in the background once the exploit successfully executes.  This means that if you have good exploit protection, you'd never see the malware.  I've done plenty of hunting (deliberately trying to get infected) and the last time I was able to come across any direct link to an actual malicious file (malware, not a PUP) was several years ago back when most of the threats were rogues (fake AVs etc.) before the bad guys started using exploits all the time.  The only exceptions I can think of offhand would be Trojans designed to look like documents when they're actually executables or (more often) documents crafted to contain exploit code that come in attached to spam emails/phishing emails, both of which Malwarebytes does very well against.

This is what I mean about a "real world" test.  Not going to a list of malware domains where direct links to malware are hosted, because that's not how those threats are getting onto users' systems.  It's the spam, exploits and malvertisements.  That's how the bad guys are doing it, and that's what real users have to worry about so that's where our focus is.  We target earlier points in the attack chain to prevent the malware from ever even getting to the user's system in the first place so they don't have to worry about whether or not our protection detects some dropper they'll never see.

Edited April 30, 2017 by exile360

[Tinstaafl]

1 hour ago, exile360 said:

Right, they're using live malware URLs which is not how actual users in the real world come across these threats.  They get redirected to those URLs by exploits and similar tactics that download the droppers (the malware etc.) in the background once the exploit successfully executes.  This means that if you have good exploit protection, you'd never see the malware.  I've done plenty of hunting (deliberately trying to get infected) and the last time I was able to come across any direct link to an actual malicious file (malware, not a PUP) was several years ago back when most of the threats were rogues (fake AVs etc.) before the bad guys started using exploits all the time.  The only exceptions I can think of offhand would be Trojans designed to look like documents when they're actually executables or (more often) documents crafted to contain exploit code that come in attached to spam emails/phishing emails, both of which Malwarebytes does very well against.

This is what I mean about a "real world" test.  Not going to a list of malware domains where direct links to malware are hosted, because that's not how those threats are getting onto users' systems.  It's the spam, exploits and malvertisements.  That's how the bad guys are doing it, and that's what real users have to worry about so that's where our focus is.  We target earlier points in the attack chain to prevent the malware from ever even getting to the user's system in the first place so they don't have to worry about whether or not our protection detects some dropper they'll never see.

I get the differences that you have described.  But I think that malvertising can also redirect you even without loading an exploit, and send your browser to a malicious URL.

I recently experienced this with a fully up to date Firefox browser.  I ran into the "Fake Firefox update" scam.  Normally I am very careful and also run the uBlock Origin browser extension with all of the ad and malware filters set to high paranoia level.  But one particular day I decided to allow my favorite mainstream weather site to be rewarded by disabling the filtering.  This site is handy to have loaded in a tab because it keeps refreshing with current weather data.  Apparently the ads rotate as well.  I had left the room for a few minutes, so nobody was clicking anything.  When I returned to the PC I was staring at a very real looking new page pretending to provide an "urgent" or "critical" update and prompting to download a firefox-patch.js

https://support.mozilla.org/en-US/kb/i-found-fake-firefox-update

The file extension was obviously a scam, so I hit the power switch and shut down.  I was very certain that I had no malware installed, as I run a leading AV, scan the computer regularly with several products including Malwarebytes and HitmanPro, as well as check every executable and driver on the PC against VirusTotal.

Re-booted and scanned everything, but nothing found.  I do keep my ad filters on most of the time now though!  I suppose that if I had clicked that link, it would have been my fault for letting it in, or maybe a good "real-world" test of my real-time exploit protection.  Rather not find out!

[dont_touch_my_buffer]

2 hours ago, exile360 said:

This is what I mean about a "real world" test.  Not going to a list of malware domains where direct links to malware are hosted, because that's not how those threats are getting onto users' systems.  It's the spam, exploits and malvertisements.  That's how the bad guys are doing it, and that's what real users have to worry about so that's where our focus is.  We target earlier points in the attack chain to prevent the malware from ever even getting to the user's system in the first place so they don't have to worry about whether or not our protection detects some dropper they'll never see.

You seem to state that MB failing in "real world" test, but works just fine if there's some social engineering on the front end of the malicious URL..

Accessing the malware via a direct link, or via any other delivery method should have the same results as far as protection is concerned. At the end of the day, it is the malicious URL, accessed directly and/or by redirecting in the background, that delivers the payload.

If your focus is on "spam, exploits and malvertisements", that could be a simple black listing URLs. If that's the case, that can quickly become a "whack-a-mole" game, just like the AV is. Provided that the MB real time web protection works reliably, but that's a whole other issue...

 

[exile360]

36 minutes ago, Tinstaafl said:

I get the differences that you have described.  But I think that malvertising can also redirect you even without loading an exploit, and send your browser to a malicious URL.

I recently experienced this with a fully up to date Firefox browser.  I ran into the "Fake Firefox update" scam.  Normally I am very careful and also run the uBlock Origin browser extension with all of the ad and malware filters set to high paranoia level.  But one particular day I decided to allow my favorite mainstream weather site to be rewarded by disabling the filtering.  This site is handy to have loaded in a tab because it keeps refreshing with current weather data.  Apparently the ads rotate as well.  I had left the room for a few minutes, so nobody was clicking anything.  When I returned to the PC I was staring at a very real looking new page pretending to provide an "urgent" or "critical" update and prompting to download a firefox-patch.js

https://support.mozilla.org/en-US/kb/i-found-fake-firefox-update

The file extension was obviously a scam, so I hit the power switch and shut down.  I was very certain that I had no malware installed, as I run a leading AV, scan the computer regularly with several products including Malwarebytes and HitmanPro, as well as check every executable and driver on the PC against VirusTotal.

Re-booted and scanned everything, but nothing found.  I do keep my ad filters on most of the time now though!  I suppose that if I had clicked that link, it would have been my fault for letting it in, or maybe a good "real-world" test of my real-time exploit protection.  Rather not find out!

It was a .JS file, so most likely an exploit.  Had it actually executed it likely would have tried downloading a malicious binary, though in theory 3.0 should have flagged it the moment the .JS tried to execute, assuming that was in fact the case.  Most of what I come across as far as the fake updates for browsers and plugins these days have been PUPs (bundled installers and adware browser plugins usually) but yeah, there are definitely still some scripted threats like this out there.

[exile360]

20 minutes ago, dont_touch_my_buffer said:

You seem to state that MB failing in "real world" test, but works just fine if there's some social engineering on the front end of the malicious URL..

Accessing the malware via a direct link, or via any other delivery method should have the same results as far as protection is concerned. At the end of the day, it is the malicious URL, accessed directly and/or by redirecting in the background, that delivers the payload.

If your focus is on "spam, exploits and malvertisements", that could be a simple black listing URLs. If that's the case, that can quickly become a "whack-a-mole" game, just like the AV is. Provided that the MB real time web protection works reliably, but that's a whole other issue...

 

It is a whack-a-mole game to block the bad URLs/domains/IPs but we still do so with our web protection, but that's not what I'm talking about here.  What I meant was that the exploit is what typically downloads the payload (usually silently in the background, then proceeds to execute it) and while payloads can certainly change as can the sites they're hosted on, exploits are quite limited and our exploit protection is primarily what I'm referring to that isn't being tested here.  If you stop the attack as early in the process as the exploit phase prior to even the redirect/attempt to download the payload then detecting the payload becomes moot.  I mean there's nothing wrong with trying to detect all the malicious websites and binaries and we certainly still do our best with our malware protection and web blocking as well as our extensive heuristics, but we know that it's always just a matter of time before things get changed and there we are playing catch up again (just like the AVs) so we've now taken a different approach and started to include signature-less components that stop the attack much earlier in the process and work far more effectively against new/unseen threats like our exploit protection.

[Telos]

8 hours ago, DodiGlenn_PCMatic said:

There are other 3rd party tests, not commissioned by PC Pitstop, which show how MalwareBytes 2.0 performed

First, this is somewhat useless. v2 was never an anti-ransomware product though it would detect behavioral issues in certain ransomware which were malware-based.

Second, a YouTube search can easily locate examples for nearly all of the "100%" detection programs in that chart which failed miserably against ransomware challenges. 

I've got to wonder what motivates posting information irrelevant to v3.

[lock]

2 hours ago, Telos said:

I've got to wonder what motivates posting information irrelevant to v3.

We would post info relevant to v.3 , if any of this info would be available....

[DodiGlenn_PCMatic]

The only reason I posted the link was to show that MalwareBytes is being tested by companies that were not commissioned by PC Pitstop. That's why I specifically called out MBAM 2.0, instead of making it sound like it was MBAM 3.0 being tested.

I would love for MalwareBytes to participate in more tests, from various testing houses, such as AV-Test, Virus Bulletin, etc., however, that has not been seen recently.

 

Dodi Glenn

 

[exile360]

8 hours ago, DodiGlenn_PCMatic said:

The only reason I posted the link was to show that MalwareBytes is being tested by companies that were not commissioned by PC Pitstop. That's why I specifically called out MBAM 2.0, instead of making it sound like it was MBAM 3.0 being tested.

I would love for MalwareBytes to participate in more tests, from various testing houses, such as AV-Test, Virus Bulletin, etc., however, that has not been seen recently.

 

Dodi Glenn

 

Unless the tests were specifically commissioned by us, we have no choice in the matter to my knowledge but we aren't stopping any organization from including our products in such tests if they choose to.

[DodiGlenn_PCMatic]

40 minutes ago, exile360 said:

Unless the tests were specifically commissioned by us, we have no choice in the matter to my knowledge but we aren't stopping any organization from including our products in such tests if they choose to.

I appreciate that. I would rather MalwareBytes voluntarily test their products, rather than a competitor, to remove any confusion and assumptions about a commissioned test.

[Telos]

10 hours ago, DodiGlenn_PCMatic said:

I would rather MalwareBytes voluntarily test their products, rather than a competitor...

The catch with that premise, is that there will be those trolls/detractors who suggest that self-testing is self-promoting (some of which frequent this forum), and invalid.

So there is no answer to this paradox that will satisfy all. 

Bear in mind too that there is no anti-malware/ransomware/virus solution that is 100% effective.

[exile360]

16 hours ago, Telos said:

The catch with that premise, is that there will be those trolls/detractors who suggest that self-testing is self-promoting (some of which frequent this forum), and invalid.

So there is no answer to this paradox that will satisfy all. 

Exactly.  Were we to commission or perform our own test, it's likely that many would claim the results are skewed because we could have chosen threats that we know we will detect and which we know competitors (assuming any competitors are included in this hypothetical test) would miss.  It's a catch 22 that way.

[Tinstaafl]

On 5/1/2017 at 9:09 AM, Telos said:

Bear in mind too that there is no anti-malware/ransomware/virus solution that is 100% effective.

Yup!  The effectiveness is also influenced as described by the age-old acronym "PEBCAK" (Problem Exists Between Chair And Keyboard).

Clicking on pop-up alerts and email attachments seems to be the best way to get a malware payload installed...