Jump to content
PatrickFAU

MBMC RAM usage and crash

Recommended Posts

The Malware Bytes Management Console has been spiking up to use all of the RAM on the machine and then slows until it stops responding.

This started Monday when we were finally able to start getting machines back online from the weekend, and i'm sure is related to receiving all the data from the machines.   I updated from 1.6x to 1.8.x last night, the licenses show licensed, but the RAM crawls back up until it's unresponsive again.

Share this post


Link to post
Share on other sites

:welcome:

Hope this helps

In the early morning of Saturday, January 27, 2018, a faulty Web Protection update was released which caused a connection issue for many of our customers. As a side effect of the web protection blocks, the product also spiked memory usage and possibly caused a crash. We triaged the issue quickly and pushed a protection update on Saturday, January 27, 2018 at 10:48a PST.

The affected products were Malwarebytes 3 Premium, Malwarebytes Management Console (MBMC), and Malwarebytes Endpoint Protection (aka Malwarebytes Cloud Console). Malwarebytes for Mac, Android, AdwCleaner, Incident Response and Breach Remediation were not and are not affected. For a complete description and root cause analysis please click here.

Please note endpoints were not affected if they were turned off before Saturday, Jan 27, 2018 and then were not turned back on until after Saturday, Jan 27, 2018 at 11am PST. For affected endpoints, this thread is intended as recovery guidance.

 

Guidance below applies to corporate customers using the on-premise Malwarebytes Management Console (MBMC) as well as corporate customers using the cloud-based Malwarebytes Endpoint Protection (aka Malwarebytes Cloud Console). If you're a home user and/or Malwarebytes 3 Premium user, click here for details on how to recover your systems.

 

For corporate customers running the on-premise Malwarebytes Management Console (MBMC)

  1. In the Malwarebytes Management Console, edit the Policy and disable real-time protection
  2. Once real-time is protection is disabled and your clients can communicate, highlight the endpoints on the Client tab and click the Update Database button at the top. This should fix it for most endpoints.
  3. If any endpoints fail to get the update, you will have to force an update. This can be done locally on the endpoint or remotely over the network.
    • Locally on the endpoint (logged in to the machine). You can point your endpoint users to do this themselves:
      1. Download and execute MBAM Rules Offline Updater
      2. Reboot the computer
    • Remotely over the network
      1. Make sure your machine is on a non-blocked IP (i.e. 10.x.x.x or 192.x.x.x). Blocked IP ranges are from 128.x.x.x to 191.x.x.x.
        *NOTE* It is recommended to not use your MBMC server for this task
      2. Download the following script and extract it to a folder on your computer
      3. Create a file named hostnames.txt in the same folder, adding one IP per line for each of your endpoint IPs. You can export a list of IPs with the faulty update from the Management Console (sort by update version, select affected ones, copy, and paste into notepad).
      4. If your internal DNS is not on a blocked IP range, you can feed hostnames.txt with hostnames instead of IPs
      5. Edit the script and type in the *LOCAL* admin username and password for endpoints (i.e. NOT the domain admin) in the first 2 lines
      6. Run the batch file, which will delete the faulty database file and schedule a reboot in 30 seconds
  4. Once all the machines are updated, turn on real-time protection in the Management Console Policy settings.
  5. If the Management Server SQL database grows heavily and takes up too much space, feel free to truncate the contents of the TBL_ClientSecurityLog and TBL_ClientSystemLog SQL tables. Detailed instructions can be found in this document. IMPORTANT: this will remove ALL detection history and is irreversible.

 

For corporate customers running the cloud-based Malwarebytes Endpoint Protection (aka Malwarebytes Cloud Console)

  1. In the Malwarebytes Cloud Console, go to Settings -> Policy and disable Web Protection and Self-Protection (if enabled). Do this for all the Policies
  2. On the Endpoints section, choose "select all" and choose "Check for Protection Updates" from the Actions button. This should fix it for most endpoints.
  3. If any endpoints fail to get the protection updates, you will have to force an update. This can be done locally on the endpoint or remotely over the network.
    • Locally on the endpoint (logged in to the machine). You can point your endpoint users to do this themselves:
      1. Login to the machine and start a scan by right-clicking on the Malwarebytes traybar icon. This will force an update and fix the issue.
      2. Cancel the scan and reboot the machine. This should fix the problem in most cases.
      3. If the above doesn't work or the machine is unresponsive, download mbep-fixer.exe to your Desktop.
      4. Execute mbep-fixer.exe. You will need to execute this as admin.
      5. Reboot.
    • Remotely over the network
      1. Make sure your machine is on a non-blocked IP (i.e. 10.x.x.x or 192.x.x.x). Blocked IP ranges are from 128.x.x.x to 191.x.x.x.
      2. Download the following script and extract it to a folder on your computer
      3. Create a file named hostnames.txt in the same folder, adding one IP per line for each of your endpoint IPs
      4. If your internal DNS is not on a blocked IP range, you can feed hostnames.txt with hostnames instead of IPs
      5. Edit the script and type in the *LOCAL* admin username and password for endpoints (i.e. NOT the domain admin) in the first 2 lines
      6. Run the batch file, which will delete the faulty database file and schedule a reboot in 30 seconds
  4. Once all machines are updated and connecting correctly, go to the Cloud Console, Settings, Policy, and enable Web Protection and Self-Protection again.

If the above guidance does not help and you are a corporate customer, please contact corporate-support@malwarebytes.com for further support.

Share this post


Link to post
Share on other sites

Sorry, The best way to deal with this is contacting corporate-support@malwarebytes.com

Share this post


Link to post
Share on other sites

@PatrickFAU in the aftermath of the FP, we've found that MBMC admins will have an SQL that is now too full because of the FP hits. Do you have a ticket open? The process to clean this up will involve SQL Management Studio.

Share this post


Link to post
Share on other sites

Thank you for the reply, i have a ticket (#00051828)  open and have done the trunc logs four times

 

Is there a different task in  SSMS i need to do?

Share this post


Link to post
Share on other sites

Seconding the issue.  I have a ticket open too and have not heard back anything.  Ticket is 00051816.  Whoever gets the fix first, please post it.

I am familiar with truncating the database to get out the oldest logs by date, but my request is to purge a date range (1/27-1/29) but keep all of the good data before and after.

Edited by OldAdminDude

Share this post


Link to post
Share on other sites
3 hours ago, PatrickFAU said:

Thank you for the reply, i have a ticket (#00051828)  open and have done the trunc logs four times

 

Is there a different task in  SSMS i need to do?

Patrick, do you work at FAU? I am having the same issue and have a ticket open with them also. If you find anything before I do please let me know. Thank you!

Share this post


Link to post
Share on other sites

I've noticed you have checked this thread several times today, so I'm guessing you may be affected by this issue.  If so, please read the thread I have linked here.

Share this post


Link to post
Share on other sites

I am facing the same issue, doubled the RAM, upgraded the console to 1.8.x. Still, the console hangs.

Share this post


Link to post
Share on other sites

I'm still trying to figure it out.  I've received scripts to clear out the logs on the endpoint, however the devices calling home are still spiking the RAM

Share this post


Link to post
Share on other sites

Same issue here, I was able to clear the extra records from the ClientSecurityLogs DB, and was able to reduce DB size from 10 GB to 1 GB, then restarted the server.

Now, when starting up the server, the memory is MAXED out from the MalwareBytes service.

Share this post


Link to post
Share on other sites

All of your clients are resubmitting logs, you must clean them from your clients or they will resubmit fill your SQL again, location is C:\ProgramData\Malwarebytes\Malwarebytes' Anti-Malware\Logs

If you have a ticket open, hold tight, we are still trying to get to everyone's ticket, volume is still very high.

You can turn off your console's service, "Malwarebytes Management Service" to prevent the submission behavior as you try to clean the clients log folders.

Share this post


Link to post
Share on other sites
6 minutes ago, djacobson said:

All of your clients are resubmitting logs, you must clean them from your clients or they will resubmit fill your SQL again, location is C:\ProgramData\Malwarebytes\Malwarebytes' Anti-Malware\Logs

If you have a ticket open, hold tight, we are still trying to get to everyone's ticket, volume is still very high.

You can turn off your console's service, "Malwarebytes Management Service" to prevent the submission behavior as you try to clean the clients log folders.

How much log submission overwhelms the MBConsole? I truncated down to 4 days, but which is 20-30 logs per machine. Should I just wipe all teh logs?

 

I do have a ticket as well. Thanks.

Edited by techsmith

Share this post


Link to post
Share on other sites

Have a ticket - Have cleared the logs and they are still calling in.  is the data stored elsewhere and just referenced in the logs?

Share this post


Link to post
Share on other sites

It's all the machines doing it at once when they check-in and see their archived records are no longer in SQL. The process is to delete the entire contents of the clients log folders. 

Example script:
del /f /s /q "C:\ProgramData\Malwarebytes\Malwarebytes' Anti-Malware\Logs\"*.*

We have a process for only pulling out FP records from the SQL and leaving normal hit data in-place, this was designed to help our customers who have HIPAA requirements.

Share this post


Link to post
Share on other sites

I am using Powershell since it flies through the clients for us in PDQ Deploy much quicker. I will reduce the log retention to 1 day.

 

Get-ChildItem -Path "C:\ProgramData\Malwarebytes\Malwarebytes' Anti-Malware\Logs" -Recurse | Where-Object {($_.LastWriteTime -lt (Get-Date).AddDays(-5))} | Remove-Item

 

 

Share this post


Link to post
Share on other sites

Logs which have been archived to SQL have their names appended with "archived", there is no need to keep any of them on the client due to the resubmission behavior. This clean-up must be done prior to removing records from the SQL or the server will be inundated once again. Another key item is your Managed Client version MUST match your console version or your check-in timers will be grossly inaccurate.

5a7a24d86d1dc_consoleclientversions.thumb.JPG.04ff1f5c1f7117899aab3167cd556a85.JPG

Share this post


Link to post
Share on other sites

Then skip it, I brought that up as an extra point, just focus on deleting the client logs. If your Express SQL files are past 10gb, you're locked out anyway, no need to focus on the server until you make it so your endpoints will not resubmit, once you finish that, then whatever agent has your ticket will deal with your console and SQL.

Share this post


Link to post
Share on other sites

So we had done all of the above. There are no logs files on the user clients. We have cleaned the database of all log files. All services are off.

When we turn the services back on it works for a moment and then all the logs (that have been deleted on the client) start flooding back in.

Why?

THERE ARE MORE LOGS...  (on the clients)

x:\programdata\sccom\txthrlog\tempthreatlog_XX.....XXX.txt

This must be a left over from stopping the clients. There are others logs adjacent to this folder however the above logs clearly captured its data during the issue Saturday morning.

Change the name of this file or delete it. No more flooding.

Edited by dyoderii

Share this post


Link to post
Share on other sites

Good find Dyoderi, I just checked and even with mbytes/logs cleared we were still getting flooded.

 

Is this an okay approach Malwarebytes staff?

 

 

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Recently Browsing   0 members

    No registered users viewing this page.

×
×
  • Create New...

Important Information

This site uses cookies - We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.