Continuous reboot of MBEP agent

[kmerolla]

Woke up this am to 2 reports that my clients (Windows 7 and WIndows 10) running Endpoint Protection have been continuously rebooting since 2:30 am Eastern.  I can only assume a this point that this is somehow related to the Friday release.

Anyone else out there seeing this? Any way I can stop this from rebooting my 1400 servers and endpoints on a Monday morning?

[kmerolla]

OK, I am up to 6 affected systems.  Looks like the MBAMService is failing to upgrade to version 3.3.2.  I know there's an issue regarding this version, however,  the issue occurs AFTER it''s installed.  On my endpoints it's not getting installed in the first place, just repeatedly failing and rebooting the endpoint.

Policy is set to not reboot.

 

2018-01-15 02:38:02.540   Setup version: Inno Setup version 5.5.8 (u)
2018-01-15 02:38:02.540   Original Setup EXE: C:\PROGRAMDATA\MALWAREBYTES\MBAMSERVICE\instlrupdate\mb3-setup-common-3.3.2.2243.exe
2018-01-15 02:38:02.540   Setup command line: /SL5="$109000B8,66933770,119296,C:\PROGRAMDATA\MALWAREBYTES\MBAMSERVICE\instlrupdate\mb3-setup-common-3.3.2.2243.exe" /VERYSILENT /SUPPRESSMSGBOXES /NORESTART /LOG="C:\ProgramData\Malwarebytes Endpoint Agent\Logs\mbaminstall20180115023802.log"
2018-01-15 02:38:02.540   Windows version: 10.0.15063  (NT platform: Yes)
2018-01-15 02:38:02.541   64-bit Windows: Yes
2018-01-15 02:38:02.541   Processor architecture: x64
2018-01-15 02:38:02.541   User privileges: Administrative
2018-01-15 02:38:02.543   64-bit install mode: Yes
2018-01-15 02:38:02.547   Created temporary directory: C:\WINDOWS\TEMP\is-4634A.tmp
2018-01-15 02:38:02.564   Extracting temporary file: C:\WINDOWS\TEMP\is-4634A.tmp\suhlpr.dll
2018-01-15 02:38:02.872   uninstall of MBAM 1.x failed
2018-01-15 02:38:02.923   Copied installer to temp directory C:\WINDOWS\TEMP\mb3-setup-common-3.3.2.2243.exe
2018-01-15 02:38:02.923   Extracting temporary file: C:\WINDOWS\TEMP\is-4634A.tmp\mb-clean.exe
2018-01-15 02:38:03.137   CurStepChanged raised an exception (fatal).
2018-01-15 02:38:03.137   Deinitializing Setup.
2018-01-15 02:38:03.142   Log closed.

 

[KDawg]

K you are are correct unfortunately this push does appear to be failing

A temporary fix for these affected machines:

1. Stop Malwarebytes Endpoint Agent Service
2. Uninstall Malwarebytes 3.3.2 from appwiz.cpl (add/remove programs)
3. Install Malwarebytes 3.1.8 using the installer in this box link: https://malwarebytes.box.com/s/h1ji0u2hziqu5e7z99ovtx87peq69q20
(password: Chart)
4. Start Malwarebytes Endpoint Agent Service

Moving forward our dev team would like the following to gain some insight and resolve this. If we are able to collect a registry dump by File > Export at the "computer" level in Reg Edit from an endpoint getting the update loop

Please upload to this site referencing the case#00047930

https://www.malwarebytes.com/support/business/businessfileupload/

It pains me to hear about this happening to you guys, we are working to get this resolved ASAP

[kmerolla]

@KDawg On the affected systems Malwarebytes 3.1.8 has already been uninstalled by the upgrade process.  Should I force re-install it?  What will stop it from re-attempting the upgrade?

Getting the Registry files now.

[kmerolla]

@KDawg, Registry archive uploaded.  Also I believe I have found an initial workaround.  I moved 2 affected systems into an IR only policy and they have stopped prompting to reboot for now.

[Brandon_Lutz]

I got reports as well for the same issue. Thank you to kmerolla to providing the work around. I've moved all of my clients over to an IR only policy and that appears to have stopped the reboot notifications.

[KDawg]

@Brandon_Lutz

I apologize about the inconvenience again we are working urgently to resolve. If you are able to collect the following from an endpoint with this issue reported it may expedite our resolution.

 If we are able to collect a registry dump by File > Export at the "computer" level in Reg Edit from an endpoint getting the update loop

Please upload to this site referencing the case#0000000

https://www.malwarebytes.com/support/business/businessfileupload/

many thanks

[Brandon_Lutz]

KDawg,

I submitted the registry dump file as requested referencing case# 0000000

Also in my case, it seems the only affected users on my network are Windows 10 users. None of my Windows 7 users have reported this issue.

[djacobson]

@Brandon_Lutz @kmerolla what installer did you guys use in your deployment? Full, web, msi agent only or D&D tool?

[Brandon_Lutz]

I used the Deployment and Discovery tool when I installed MEP on all of our clients.

[djacobson]

Thanks @Brandon_Lutz, I want to reproduce exactly as you guys are doing it.

Edited January 15, 2018 by djacobson

[Brandon_Lutz]

Just now, djacobson said:

Thanks @Brandon_Lutz, I want to reproduce exactly as you guys are doing it.

No problem. I havn't deployed a new client with the D&D tool in over a month however with the exception of a couple of servers. Those have been unaffected by this issue it appears.

[kmerolla]

Dyllon, we deploy using full exe with prerequisites via SCCM.  Also only happening to Windows 10 endpoints.

Working with Josh on my support case and and the dev team identified 3 reg keys that, when removed, allowed the upgrade to occur silently and without issue on 2 of my affected endpoints.

[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Malwarebytes' Managed Client]
[HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{72BE25D7-574A-4F4D-B9B3-907D239CE1C7}]
[HKEY_USERS\<user SID>\Software\Malwarebytes' Anti-Malware]

(we are migrating off 1.8 MBMC but some legacy agents survived.  These keys may have been left behind when the installer initially placed MBEP on the system). I think our SCCM uninstall routine isn't cleanly removing all the things.

Support also indicated that the deployment of 3.3.2 has been suspended so no new cases should pop up.

-Kevin

[djacobson]

That's what I found and am testing. Dev will write these into the cleanup process prior to install in the future. I am trying to find out which key, or combo's of keys, is the exact one that trips this. 

Edited January 15, 2018 by djacobson

[kbross]

I know others have reported this on the Windows 10 platform only, but I have several Windows 7 machines showing the same behavior.

[djacobson]

@kbross what version of MBMC agent did you deploy over?

[kbross]

@djacobson Are you referring to the original installer used? I'm not sure I recall in this instance, but I can try to find out.

[djacobson]

No, I mean what version of Malwarebytes did you have deployed before your EP product. This issue only happens when you install EP over a previous MBMC deployment.

[kbross]

@djacobson Not sure which version was there previously since it was uninstalled.

[mkhanolkar]

Same issue with our environment. In our case, the previous client was uninstalled via "MSIEXEC /X {product_code}" before Endpoint Agent was installed. The reboot prompt didn't show up until after manually restarting the first time. If it helps I can upload a verbose MSI log. Here's a snippet from the end:

MSI (s) (20:50) [10:01:48:213]: User policy value 'DisableRollback' is 0
MSI (s) (20:50) [10:01:48:213]: Machine policy value 'DisableRollback' is 0
MSI (s) (20:50) [10:01:48:213]: Incrementing counter to disable shutdown. Counter after increment: 0
MSI (s) (20:50) [10:01:48:228]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2 
MSI (s) (20:50) [10:01:48:228]: Note: 1: 2265 2:  3: -2147287035 
MSI (s) (20:50) [10:01:48:228]: Note: 1: 1402 2: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Installer\Rollback\Scripts 3: 2 
MSI (s) (20:50) [10:01:48:260]: Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied.  Counter after decrement: -1
MSI (s) (20:50) [10:01:48:260]: Destroying RemoteAPI object.
MSI (s) (20:D0) [10:01:48:260]: Custom Action Manager thread ending.
MSI (c) (28:38) [10:01:48:306]: Decrementing counter to disable shutdown. If counter >= 0, shutdown will be denied.  Counter after decrement: -1
MSI (c) (28:38) [10:01:48:306]: MainEngineThread is returning 0
=== Verbose logging stopped: 1/16/2018  10:01:48 ===
 

[djacobson]

@mkhanolkar that was good thinking but the nature of Windows uninstall, it is not enough, and even the cloud installer will perform some level of cleanup though it is missing some items, there will be registry keys and folders still left on the machine despite uninstalling and letting the deploy perform some cleanup. This is why we need the reg dumps, to find the keys we are missing in cleanup. We have separate cleaner tools that can be used for a totally fresh install but they are not well known.

MBMC Managed Cleaner tool msi - https://malwarebytes.box.com/s/rck2gbt0kqqdp8iw1uk7u6pmjg0gajkr

Consumer, Cloud and MBMC managed client tool - https://support.malwarebytes.com/docs/DOC-1112

[nsparks]

We are experiencing this exact same issue on several workstations. Has a fix been issued yet? Prompts for reboot over and over and over and after the weekend fun of the web protection, our users are frustrated. I've sent in several tickets and have had no response.

[gtatech]

This is happening to me with Windows 7 Professional.  Uninstall / Re-Install with Web Downloader does not resolve the issue.

[randid]

Just wanted to add this is also happening to me, i was doing testing on a windows 10 machine, also uninstalled the old MB at the same time like the others, i cannot deploy this until its fixed.

[jarodss]

Any fix to this yet?