MBAM Console Almost Unusable

[m0biustheory]

After the issue this past weekend our MBAM management console has been unresponsive and almost unusable.  I've done a little DB cleanup by deleting entries in the Security Log table (we recieved millions of alerts, and they are still coming in), which reduced our DB quite a bit but it hasnt helped the speed at all.  Any tips?

[Porthos]

1 hour ago, m0biustheory said:

Any tips?

I don't deal wit this section but here is the help for the issue.

In the early morning of Saturday, January 27, 2018, a faulty Web Protection update was released which caused a connection issue for many of our customers. As a side effect of the web protection blocks, the product also spiked memory usage and possibly caused a crash. We triaged the issue quickly and pushed a protection update on Saturday, January 27, 2018 at 10:48a PST.

The affected products were Malwarebytes 3 Premium, Malwarebytes Management Console (MBMC), and Malwarebytes Endpoint Protection (aka Malwarebytes Cloud Console). Malwarebytes for Mac, Android, AdwCleaner, Incident Response and Breach Remediation were not and are not affected. For a complete description and root cause analysis please click here.

Please note endpoints were not affected if they were turned off before Saturday, Jan 27, 2018 and then were not turned back on until after Saturday, Jan 27, 2018 at 11am PST. For affected endpoints, this thread is intended as recovery guidance.

 

Guidance below applies to corporate customers using the on-premise Malwarebytes Management Console (MBMC) as well as corporate customers using the cloud-based Malwarebytes Endpoint Protection (aka Malwarebytes Cloud Console). If you're a home user and/or Malwarebytes 3 Premium user, click here for details on how to recover your systems.

 

For corporate customers running the on-premise Malwarebytes Management Console (MBMC)

1. In the Malwarebytes Management Console, edit the Policy and disable real-time protection
2. Once real-time is protection is disabled and your clients can communicate, highlight the endpoints on the Client tab and click the Update Database button at the top. This should fix it for most endpoints.
3. If any endpoints fail to get the update, you will have to force an update. This can be done locally on the endpoint or remotely over the network.
   • Locally on the endpoint (logged in to the machine). You can point your endpoint users to do this themselves:
     1. Download and execute MBAM Rules Offline Updater
     2. Reboot the computer
   • Remotely over the network
     1. Make sure your machine is on a non-blocked IP (i.e. 10.x.x.x or 192.x.x.x). Blocked IP ranges are from 128.x.x.x to 191.x.x.x.
        *NOTE* It is recommended to not use your MBMC server for this task
     2. Download the following script and extract it to a folder on your computer
     3. Create a file named hostnames.txt in the same folder, adding one IP per line for each of your endpoint IPs. You can export a list of IPs with the faulty update from the Management Console (sort by update version, select affected ones, copy, and paste into notepad).
     4. If your internal DNS is not on a blocked IP range, you can feed hostnames.txt with hostnames instead of IPs
     5. Edit the script and type in the *LOCAL* admin username and password for endpoints (i.e. NOT the domain admin) in the first 2 lines
     6. Run the batch file, which will delete the faulty database file and schedule a reboot in 30 seconds
4. Once all the machines are updated, turn on real-time protection in the Management Console Policy settings.
5. If the Management Server SQL database grows heavily and takes up too much space, feel free to truncate the contents of the TBL_ClientSecurityLog and TBL_ClientSystemLog SQL tables. Detailed instructions can be found in this document. IMPORTANT: this will remove ALL detection history and is irreversible.

 

For corporate customers running the cloud-based Malwarebytes Endpoint Protection (aka Malwarebytes Cloud Console)

1. In the Malwarebytes Cloud Console, go to Settings -> Policy and disable Web Protection and Self-Protection (if enabled). Do this for all the Policies
2. On the Endpoints section, choose "select all" and choose "Check for Protection Updates" from the Actions button. This should fix it for most endpoints.
3. If any endpoints fail to get the protection updates, you will have to force an update. This can be done locally on the endpoint or remotely over the network.
   • Locally on the endpoint (logged in to the machine). You can point your endpoint users to do this themselves:
     1. Login to the machine and start a scan by right-clicking on the Malwarebytes traybar icon. This will force an update and fix the issue.
     2. Cancel the scan and reboot the machine. This should fix the problem in most cases.
     3. If the above doesn't work or the machine is unresponsive, download mbep-fixer.exe to your Desktop.
     4. Execute mbep-fixer.exe. You will need to execute this as admin.
     5. Reboot.
   • Remotely over the network
     1. Make sure your machine is on a non-blocked IP (i.e. 10.x.x.x or 192.x.x.x). Blocked IP ranges are from 128.x.x.x to 191.x.x.x.
     2. Download the following script and extract it to a folder on your computer
     3. Create a file named hostnames.txt in the same folder, adding one IP per line for each of your endpoint IPs
     4. If your internal DNS is not on a blocked IP range, you can feed hostnames.txt with hostnames instead of IPs
     5. Edit the script and type in the *LOCAL* admin username and password for endpoints (i.e. NOT the domain admin) in the first 2 lines
     6. Run the batch file, which will delete the faulty database file and schedule a reboot in 30 seconds
4. Once all machines are updated and connecting correctly, go to the Cloud Console, Settings, Policy, and enable Web Protection and Self-Protection again.

If the above guidance does not help and you are a corporate customer, please contact corporate-support@malwarebytes.com for further support.

 

[djacobson]

@m0biustheory, client security log and system log need to be wiped for the days of the FP to get your speed back. You also need to delete the client logs, because if they see their records missing, they will resubmit and fill the DB again. Your endpoints are likely to have tens of thousands of log records each, location is C:\ProgramData\Malwarebytes\Malwarebytes' Anti-Malware\Logs.

Edited January 31, 2018 by djacobson

[SaraVN]

I have been deleting the logs as instructed above every day this week. Multiple times a day. Now my servers hard drive is full, the databases are currently 0, but growing quickly. 

We've cleaned up most of our clients -but there are four that are still reporting the "bad" database version, even after following the directions. We are now in the process of removing the client from those machines. 

How do I clean up my server so that it is functional again? the SCDB database is showing as 16GB on the disk, with the scdb_log as 4.85 GB. I don't know if that's the problem or if it's something else, but I need to get the Console to be stable.

Anyone else seeing this?

[gonzo]

Please download this Admin Guide, and pay special attention to EMERGENCY MAINTENANCE on page 5.

https://www.malwarebytes.com/pdf/guides/MMLNBPGuide.pdf?d=2018-02-02-09-48-30--0800

The server is rejecting requests from endpoints due to the volume of issues being reported, and the error log is getting an entry for each rejection.  Take care of that directory, and follow the other advice you were given here, and your problem will go away.

[OldAdminDude]

On ‎2‎/‎2‎/‎2018 at 1:15 PM, gonzo said:

Please download this Admin Guide, and pay special attention to EMERGENCY MAINTENANCE on page 5.

https://www.malwarebytes.com/pdf/guides/MMLNBPGuide.pdf?d=2018-02-02-09-48-30--0800

The server is rejecting requests from endpoints due to the volume of issues being reported, and the error log is getting an entry for each rejection.  Take care of that directory, and follow the other advice you were given here, and your problem will go away.

The problem I'm having is that we've deleted the contents of the log directory (C:\ProgramData\Malwarebytes\Malwarebytes' Anti-Malware\Logs) to the point where those directories stay empty, aside from new ones formed every day, and we're still getting log files dated 1/31/18 from the machines with empty log directories.  We've truncated the database as per the direction, and the console will show 0 detections.  We turn back on the firewall rule blocking  port 18457, and the floodgates open, with traffic from empty-direcotyr machines streaming in madly.  Nothing we've tried seems like more than a stopgap.  Service ticket I have open is 00058186.  This is getting old.

[Jinzing]

We are experiencing the same problem in our environment. OldAdminDude if you come up with a solution, please share. Thanks

[uncoitsecurity]

This is what we did in our environment.

1. Backup the database.

2. Delete all logs from the database as per instructions from Malwarebytes.

3. Deploy cleanup script which deletes logs from the clients.

• Malwarebytes provided instructions on writing a GPO script to delete the logs but we used a SCCM script in our shop. The script we used to delete those files reached 80% of the machines in our environment before we began this next phase.

4. Using the Window’s firewall on the server we restricted the number of clients which were able to communicate to the Malwarebytes server by blocking port 18457 communication.

5. Open communication to a small network segment.

• Using task manager we would monitor network and RAM usage, if it increased significantly we would restrict communication from the network range and reopen a smaller segment.

• Monitor client server communication using netstat -a

• Identify problem machines: (over the course of the operation we had about 80 systems that needed a manual removal of the logs)

  • Machines had more than 2 connections on port 18457 (typically it was passing many small files)

  • Machines remained in the list for an extended period of time (typically it was passing a large file slowly)

• Map the log locations on the problem machines and remove the log files manually.

• When resource use stabilized we would open another segment to the management server.

6. We deleted all logs from the database using the previous instructions to account for any machines that did not successfully execute our cleanup script.

[m0biustheory]

Just a follow up:

The performance issues subsided with the prescribed solutions in here but we are still receiving alerts from endpoints that have had their logs deleted.  I'm receiving alerts today from a workstation that has been off for 2 weeks now.  These logs have to be stored somewhere else we are unaware of... I've cleared the log tables in the database multiple times and these alerts can't be generated from the endpoints.  Where are they coming from?  

 

Any tips?

[SaraVN]

I'm so frustrated with MalwareBytes ever since the issues during the last weekend in January. I have hundreds of clients that say that  haven't been scanned since January 28th, I run the quick scan on them, it shows them running the scan, and then they pop back on the list of devices that haven't been scanned in over 30 days. I removed my connection to AD so that I could remove all the clients from the console. They are quickly repopulating, and so far it appears more of them are reporting back that they have been scanned recently, but I'm still not convinced that they are all correctly completing the scans.

Adding this to my issue with not being able to report which computers have and don't have MalwareBytes installed through SCCM, and my boss isn't happy.

[djacobson]

@SaraVN why not create a script to perform a reg query function on the service entries? If they exist you can output to a file or something with the machine name and present services. Windows itself has plenty of functionality for finding this kind of stuff out.

Anti-Exploit
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MbaeSvc
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESProtectionDriver

Anti-Malware
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MBAMService
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MBAMScheduler
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MBAMProtector

Anti-Ransomware
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MB3Service
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MB3SwissArmy

Managed Client Communicator 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SCCommService

[djacobson]

@m0biustheory the SQL server must still have some pending submissions that it itself is still processing. Once items are within the clientsystemlog and clientsecuritylog tables, they have been completed, so clearing that does nothing against items still waiting in the wings, items which the clients have already submitted outside of their own pending folder.

Microsoft SQL has areas where it stores pending items. The alerts you're seeing are probably pending items stored in TempDB system database being transferred to the SCDB database. Or maybe even from the transaction log. You can search MS KB articles about clearing that sort of thing, or wait it out and let the pending items submit until they are done, the caveat being we don't have a way to tell when they will be done, or how much they still need to send.

 

Also guys, blocking port 18457 is a whole lot of work, you can easily just stop the MBMC server service, "MEEServerService" for older versions - "Malwarebytes Management Service" for newer installs, and accomplish the same thing; prevent client check-in, SQL submission and writing to SQL.

[SaraVN]

On ‎4‎/‎3‎/‎2018 at 1:25 PM, djacobson said:

@SaraVN why not create a script to perform a reg query function on the service entries? If they exist you can output to a file or something with the machine name and present services. Windows itself has plenty of functionality for finding this kind of stuff out.

Anti-Exploit
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MbaeSvc
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\ESProtectionDriver

Anti-Malware
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MBAMService
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MBAMScheduler
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MBAMProtector

Anti-Ransomware
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MB3Service
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\MB3SwissArmy

Managed Client Communicator 
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SCCommService

@djacobson Why should I have to create something to monitor an individual software, when I have a system in place that monitors the other 100+ software packages we use? SCCM isn't rare in enterprise situations, in fact it's just the opposite, very common. While your product does a good job with malware detection, I just don't think it's enterprise grade yet.

Also, since I can't trust the uninstall to remove the MSI Signature in the registry, why would I trust that it removes these registry entries?

[djacobson]

I can tell you what parts of Malwarebytes to check for to determine installation. How you go about implementing that is up to you as the admin of the environment.

You want to use an outside tool to find what's installed and that's perfectly fine, but I can't support SCCM, while it may be common, it's a paid product Microsoft sells versus includes within Windows (as opposed to GPO), and so it is outside of our scope. But I'm not here to tell you 'sorry we can't help, you're using a tool we do not support', I am here to give you whatever you need within my means to help make it happen, no matter what you use.

To that end, I was meaning to create a script for SCCM send out or use SCCM itself to check for those keys. They are for the services the program uses and a reliable way to check for an install, while Windows uninstaller can miss some directories, it doesn't often miss the services. You can even check their running state value in that same key path. Say your SCCommService key is present but the start value shows the service is in a stopped state. Boom, you just found out that the product is installed but not communicating because the comm service is not running. Simple and effective.

To circle back on the inaccurate last scan status you're experiencing, have you done the FP record maintenance on the SQL side and deleted the client archive and pending log areas? Removing clients from the client view is just a deletion of the client's entry on the dbo.tbl_Client table, having that work out for you suggests to me that there are still concerns around your MBMC's database.