Testing detection functionality

[randombytes]

I've read the following article:

https://blog.malwarebytes.com/security-world/2017/09/explained-false-positives/

A false-positive is only a false-positive when you detected something that should not have been detected. So if you intentionally detect EICAR, even though it is not malicious, it does not classify as a false-positive. Positively detecting a file that we wanted detected is a true-positive. Now if you detect EICAR as a Trojan or Worm then that is a classification problem. However, if you detect as EICAR_Test_File then that is a true-positive since that was detected as intended.

I've also read the following forum post:

https://forums.malwarebytes.com/topic/193897-eicar-test/

Saying Malwarebytes is not an antivirus and thus should not detect EICAR seems silly to me. Malwarebytes will detect a virus right? If so then why not detect a test virus file? EICAR has nothing to do with vetting a product as an antivirus and everything to do with testing antivirus functionality. Malwarebytes is not an antivirus replacement but it does provide a level of antivirus functionality, so it would still be useful to detect EICAR.

As for intentionally not detecting EICAR, Malwarebytes is doing us customers a disservice as that is the industry accepted standard for testing antimalware configurations without the need for using actual malware. With this, we either need to trust that the implemented antimalware configurations are correct and functioning, without verifying, or use potentially risky files to perform testing on productions systems. Without verifying, we put ourselves at risk to possible configurations problems. This is not just scan configurations such as exclusions but potentially software problems where definitions, drivers, etc are not functioning properly. If you have a test file we should be using please reference that in the article and forums. However, to follow the industry and detect EICAR would be helpful (57/60 vendors on VirusTotal detect EICAR).

[exile360]

Malwarebytes doesn't detect EICAR primarily for 2 reasons as I understand it.  First, the test uses an extremely outdated method of testing an engine's detection capabilities, relying on a strict string of characters to be contained within a file.  Second, since it also is used in non-executable files (such as text files and other non-PE file types), the primary Malware Protection component in Malwarebytes will not detect it because it doesn't look at those file types (malicious scripts and similar threats are handled by the Exploit Protection component which kills a real threat earlier in the attack chain, prior to the actual script execution phase which eliminates the need to attempt to target malicious scripts, which is a futile effort anyway given how trivial it is for the bad guys to alter/rearrange/encrypt/re-encrypt any malicious script file to completely evade any traditional text based/raw script detection technology in any AV/AM product or tool; this is why Malwarebytes instead uses behavioral detection to target exploits since, regardless of how the malicious payload/script etc. may be altered, the actual exploit behaviors used to get the object to download/execute remain limited and consistent because they must use some illegal operation/memory violation of some kind to accomplish the attack).

While I do understand that the vast majority of AV/AM products do detect EICAR, the reality is that it does nothing to prove the efficacy of those products whatsoever because the methods used to detect it are completely irrelevant to any malware that has been created or found in the wild for well over 10 years (probably actually closer to 20+).

With that said, I can provide you with a list of safe items you may use to test the various components in Malwarebytes if you wish, including some PUP installers which are harmless but should be detected by Malwarebytes since they do contain PUPs (Potentially Unwanted Programs).

I'll have to concede to whatever the staff has to say about the situation, but the way I look at it, any product using EICAR as validation for their protection/detection capabilities is not doing their customers/potential customers any favors or inspiring any true confidence in their technology because it is an incredibly obsolete test method.

[lock]

1 hour ago, randombytes said:

As for intentionally not detecting EICAR, Malwarebytes is doing us customers a disservice

It is not only this. By intentionally not detecting EICAR, MBAM reserves the right to say, down the road, that intentionally  did not detect this or that for various reasons.

Look for example of their anti exploit test file  mbae-test.exe ; it is not detected by ANY antiexploit in the market but MBAM !!!!

This is not classified as " an extremely outdated method of testing " and is OK , but Eicar not.

 

[randombytes]

54 minutes ago, exile360 said:

While I do understand that the vast majority of AV/AM products do detect EICAR, the reality is that it does nothing to prove the efficacy of those products whatsoever because the methods used to detect it are completely irrelevant to any malware that has been created or found in the wild for well over 10 years (probably actually closer to 20+).

I think you may be misunderstanding my intentions/point with EICAR. I'm not using it to prove efficacy of modern detection capabilities. I'm using it as a standard method to ensure I've properly configured the product and it is functioning properly. The age of EICAR has nothing to do with it if it works for the scenario I just described. I use EICAR for validation that the scan engine is working as I intended. EICAR is not meant for advanced testing of a scan engine, its just to ensure the basic detection feature of the engine is scanning the area being tested and that successful detection occurs. Here are some useful features of EICAR:

* EICAR is publicly available to everyone accessible via HTTP and HTTPS

  *This allows for testing of engines that scan HTTP traffic or products that do SSL intercept. Without SSL intercept, HTTPS can be used to bypass those products that only do HTTP scanning to get the sample down to a computer.

* Available publicly in ZIP for testing compress file scanning

If Malwarebytes feels that EICAR needs to be revised then I think that is something Malwarebytes should lead since the rest of the industry seems to be okay with leveraging EICAR. At minimum, if Malwarebytes can at least provide a sample that will be detected and has been vetted to be non-malicious that is available in the forms described above that would certainly be helpful.

Thanks for the information, I thought from a previous comment I read that it would be easy for Malwarebytes to detect EICAR but that it was a choice not to. However, from your explanation Malwarebytes' engine technically can't detect it due to its limited scope of supported file types. Side comment - I feel like the product should scan more than just executable files. What about non script malware such as the Carbanak shim database backdoor that injects code? That is not an exploit (Windows feature abuse) nor is it an executable. If the engine is limited to only scanning executables then that is a very good reason it can never replace or classify as antivirus.

 

[exile360]

I already offered to provide some files for you to test detection with if you wish, and several other vendors have created their own testing tools for this very reason (to test specific modules).  The scan engine in Malwarebytes is by no means its primary layer of protection, nor is it anywhere near being its most effective, so even if EICAR were added, all it would do is validate that the least proactive component of Malwarebytes is functioning in a way that is irrelevant to its true capabilities (or even the capabilities of the scan engine itself, which is the very point I've been trying to make, because even a basic script can be designed to detect a string of text like EICAR, but that doesn't mean that such a script would be an actual AV engine).

The way I see it, how something is detected is just as important as whether or not it is detected, especially when attempting to validate the functional status of something as mission critical as an AV/AM product.  That's why we and other vendors have developed specific tools for this very purpose and why it would be far more valid to test using a relatively safe file (like a PUP; which you could easily archive within a ZIP folder if you wish to test archive scanning and can place anywhere you like to test where/what Malwarebytes scans) to determine whether or not Malwarebytes is functional.

If an engine can detect a string of text, that says nothing about its ability to detect any threat that has existed within the last decade+ as I already mentioned, which is the entire purpose of such testing is it not?  I mean what good would it do to validate that a product can detect something that isn't a threat, doesn't look like a threat, isn't detected the way that threats are detected and doesn't do anything beyond what looking at the status of a product in its own interface or in the Windows Action Center would tell you (i.e. whether the product's protection is active)?  That's like using a website from 1998 to test a modern browser to verify that it is functional for loading modern websites; such a test would be invalid and could not be trusted to validate that browser's capabilities to read and render modern web code.  The same is true for testing an AV/AM product with such an irrelevant test method as a basic string of text in a text file (because that is literally all that EICAR is and you can verify this yourself by downloading the text version and looking at it in notepad or any other text editor or by opening any of the other versions of it in notepad/any other text editor).

[randombytes]

I would like to take you up on the offer to provide a testing file. If Malwarebytes could please publish the sample publicly in the forms mentioned above that would be appreciated. This way we get the same benefits of EICAR but can leverage for Malwarebytes products.

 

Coming from an enterprise environment this kind of testing is a standard thing that needs to be performed from time to time. Providing a vetted test for this that is easily accessible to customers will certainly be helpful. Again, not looking to test various detection capabilities, just trying to run a test that shows that basic file detection is working properly or not given the configuration. Here is a simple example:

                *Scanning the file with various scan methods (real-time, scheduled, on demand) to see if the configurations, such as use of the ignore list, cause a detection or not

 

Basically need to make sure that the different settings or configurations being used are working as expected with the file scan engine. This might sound simple but when dealing with various enterprise technologies being able to test functionality and compatibly is essential to limit potential problems down the road. Things aren’t always straight forward as we’d like such as verifying the scan configuration for SAN mount points on Windows servers configured in a cluster. Without testing something could be misconfigured and you may not find out until an incident happens.

 

Thanks for your help and understanding.

[exile360]

No problem at all, I totally understand where you're coming from, though I would still strongly advise testing the other protection mechanisms as well, particularly because given the nature of the modern threat landscape (especially as it has existed and evolved over the past 2~6 years, where individual threats have had shorter and shorter shelf-lives and blended threats/multi-staged attacks have become the norm) as the other layers of protection really do matter far more than the Malware Protection/scan engine components, however I will provide options for all of the ones that I can and you may decide which to use to suit your purposes:

• Malware Protection/scan engine:Advanced SystemCare (this item should be detected as PUP [Potentially Unwanted Program] by both the scanner and Malware Protection components and you must have PUP detection enabled under the protection settings (it is enabled by default); you can test by scanning the file as well as by attempting to execute it/run it and it should be detected by the appropriate module depending on which you do (the Malware Protection component checks files on execution, prior to entering memory to avoid conflicts with AVs and other AM tools/products)
• Exploit Protection:HitmanPro Exploit Test Tool (refer to manual available here); you can test by adding the test tool EXE to the list of Protected Applications under Settings>Protection>Manage Protected Applications and I recommend adding it to the default (Browser) profile as I believe that's the most general purpose exploit shield configuration (though if someone from the Malwarebytes staff has a different recommendation then I concede to their first-hand knowledge).  There is also the tool developed by Malwarebytes for this same purpose which is available here (instructions on using the tool are detailed in that topic)
• Web Protection: Try to visit or contact/ping iptest.malwarebytes.com or the IP address 52.21.84.70 and it should be blocked (any browser, any process on the system including a command prompt if you wish to script/automate the check); additional information available here
  

Unfortunately I couldn't locate anything to easily test Ransomware Protection with, however it is quite frankly more of a reactive solution anyway due to the fact that, even though it uses behavior based detection capabilities rather than signatures, it has to see ransomware activity/behavior to detect anything meaning that by the time it detects the threat, the threat is already running in memory and attempting to encrypt your data (this is where the Ransomware Protection component should intervene, saving your data from encryption by stopping the malicious activity and quarantining the threat).  With that said, given that almost 1:1 ratio of ransomware being deployed by exploits, I'm quite confident that Malwarebytes would stop such an attack far earlier in the kill chain anyway before it gets to the point of downloading/executing a ransomware payload, at least in the vast majority of cases based on what I know of most ransomware these days.

You can also test PUM (Potentially Unwanted Modification) detection which is a component of the scan engine that looks specifically for system setting configuration changes in the registry which are frequently modified by malware and PUPs.  A list of several of these may be found here and there are many others.  If you have any questions on how to set one or more of them up, how to create a batch or reg file/script to automate their creation etc. just let us know and we'll assist.

There are also several other key components to the scan engine and Malware Protection engine such as Linking (an advanced heuristic technique which can use a single detection to connect it to other traces and components of an active/installed infection through the registry and filesystem to more thoroughly detect and remove threats) and the Anti-Rootkit engine which uses DDA (Direct Disk Access) as well as various user-mode and kernel-mode detection techniques to detect and eliminate active rootkit infections along with several advanced remediation capabilities including DOR (Delete On Reboot) which is used to catch infections off-guard and kill them while they sleep early in the boot process as well as special repair and replace capabilities designed to eliminate threats and many of the system components and functions that they often damage (like internet connectivity, security related components such as Windows Defender, Security Center/Action Center, Windows Update, the Windows Firewall and many other components).

Finally, I would also suggest taking a look at the diagram and information found on this page as it provides a decent amount of detail as to how Malwarebytes Premium functions and how it leverages its various layers of defense to thwart an attack during various phases of the kill chain/attack chain, including pre-execution and post-execution, to keep systems protected.

I hope this proves useful to you and if there is anything else I can assist you with please let me know.  I definitely understand the need to validate your protection's functionality and status and I will help in any way that I can to enable you to accomplish your task.

[randombytes]

That is awesome, thanks! Nice aggregation of information. I agree 100% on additional testing and on the layered approach. Antivirus is often the very last technical layer against threats and thus could play a critical role should other controls be bypassed; however, if it’s gotten to the antivirus layer there likely has been a security failure.

If possible, could we get a link to a scan engine testing file that we know will not change? That way there is no concern over malicious code somehow making it into the product. Obviously we don’t want to execute it but mistakes happen so better to safeguard against it.

Even more ideal would be a custom executable like what you linked to for HitmanPro that was created by a software security company. This provides some additional trust that the file is safe for testing. Additionally it could be given a detection name indicating it’s a test file for better visibility into why the detection occurred on the network. This would help to quickly triage the detection during log review.

[exile360]

You're welcome

Unfortunately I don't have access to any static testing files other than the tool they use for testing Exploit Protection, but I will request something from the team so hopefully they will be able to provide something down the road to simplify the testing/protection/scan engine functionality validation process.  I know that we used to have some test executables we used in-house for such validation purposes in QA, however I'm not sure if they still exist or not and I don't know if they'd be willing or able to make those files public or not, but if they really wanted to, whipping up a test EXE and a def to target it shouldn't be too much trouble if this is functionality they want to provide to customers and users.

In the meantime, you can probably just save a copy of that PUP installer somewhere to keep it around for testing your installations while we await a better solution from the company.