Endpoints Going Offline

[IT_Guy]

Looking through my event logs to figure out why my endpoints are going offline and not coming back online, I see dozens of these errors in my event viewer:

2017-10-04 01:05:35,661-04:00 [71] ERROR BoomerangHandler Could not sync
Newtonsoft.Json.JsonReaderException: Unexpected character encountered while parsing value: <. Path '', line 0, position 0.
   at Newtonsoft.Json.JsonTextReader.ParseValue()
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(JsonReader reader, JsonContract contract, Boolean hasConverter)
   at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent)
   at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType)
   at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings)
   at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings)
   at EAEngine.Boomerang.BoomerangHandler.<Sync>d__40.MoveNext()
 

And then that gets followed up by:

Activation context generation failed for "C:\Program Files\Malwarebytes Endpoint Agent\MBCloudEA.exe".Error in manifest or policy file "C:\Program Files\Malwarebytes Endpoint Agent\MBCloudEA.exe.Config" on line 0. Invalid Xml syntax.

When I check that MBCloudEA.exe.Config it is an empty 17kb file. On working computers the file is full of data, I am going to try copying it from one machine to another and see if that fixes the problem.

I've got a feeling this is going to lead to uninstalling/reinstalling the endpoints.

[IT_Guy]

Well that didn't fix it.

Also, when uninstalling it I get an error now that prevents uninstalling. And I can't reinstall it.

 

[KDawg]

IT_Guy I can see you are experiencing some serious issues here, we are here to help and appreciate your patience while we work through this. 

 

 In regards to the larger issues you seem to be experiencing can you check if the cloud agent and malwarebytes services are running on one of the machines that is not showing up as online. If the services are stopped, are you able to restart? Does the restart of the service then cause the endpoint to show up as online?

 

Please use the following clean tool to uninstall all of the Malwarebytes products currently

https://downloads.malwarebytes.com/file/mb_clean

Run this once with the GUI as an administrator then allow the machine to restart

Once the machine is back up please run this again, this time from an Admin CMD prompt window with a /cloud switch following the .exe file path

Again allow the machine to restart

At this point we should be cleared of old installs and ready to re-install with the package, please let me know if the problem persists

[IT_Guy]

One of the machines with this problem is right beside me, everything running fine, all three services loaded, system tray application says it's running.

If the solution to the problem is to go around and remove and reinstall the client on the 60-70 workstations this is presenting on, that is not a viable solution every week. The deployment tool doesn't work properly so it has to be deployed manually which takes a good 20 minutes per computer, that is 20+ hours to fix this and I haven't seen anything indicating Malwarebytes has patched or fixed anything to solve this problem which means I might have to do it again next week.

 

Is there anywhere in the console to see which version of Malwarebytes has been deployed? v3.1.8 vs v3.2.2? Or to verify that the software is actually running and not just the endpoint agent (which doesn't report its own activity anyway)?

[IT_Guy]

UPDATE:

I'm now down to 34 endpoints being 'online' even though everyone is here and we are probably closer to 90 endpoints online. 14 of them went offline within 5 minutes of each other and another 30 in the hour before that. That's 45 endpoints offline in the last 45 minutes. Unless there is a fire in the building I'm not aware of chances are pretty good 45 people didn't just leave.

[IT_Guy]

UPDATE:

The number is going up and down but hovering around 30-40 endpoints online. I know for a fact that there are at least twice that many computers up and running right now. A machine that stays on all the time was last seen 20 minutes ago, but that machine has been online for over a week even though the cloud intermittently shows it being offline.

 

Is this just a cloud problem? How often is it checking in?

 

I am steadily losing faith that my network is in fact secure.

[KDawg]

This is we believe this may be an issue on our side with the Boomerang Error, this has been escalated to our development team and we should have an update on what is going on shortly.

[IT_Guy]

Any update as to what's going on?

I'm sitting at about 50 of 112 workstations as showing online. This computer beside me that has not been turned off in two weeks says it was last seen yesterday at noon.

[KDawg]

Still not confirmed resolved

However an attempted fix is being pushed this afternoon from our development team

[spnkzss]

I have about 17 of my 113 that I know of that have the same issues.  This does not include my random laptops that I have no clue are affected or just haven't been turned on.  The service is offline and won't start stating "Activation context generation failed for "C:\Program Files\Malwarebytes Endpoint Agent\MBCloudEA.exe".Error in manifest or policy file "C:\Program Files\Malwarebytes Endpoint Agent\MBCloudEA.exe.Config" on line 0. Invalid Xml syntax.".  It seems to have been caused by a new version that was pushed down.  I understand that things happen, but knowing that I'm going to have to go around to each PC and run the manual removal tool and reboot is not a good solution.  Especially when 7 of my machines so far are active environment servers.  My bigger concern is knowing if any of my random laptops that connect to the internet every once in awhile are affected or just haven't been turn on.

[KDawg]

We are still currently investigating the side by side errors, we have re-enabled boomerang with a fix.

If you are experiencing offline clients please use our clean tool below:

https://downloads.malwarebytes.com/file/mb_clean

From an administrator command prompt run the .exe with a /cloud switch please allow the restart.

Once cleaned please re-install with the prerequisites exe from your cloud console Endpoints > Add

 

If the clients still has any issues or you get a side-by-side error please let me know, or update your current service case.

[alb40]

Similar problem here. Opened a ticket last week. No solution yet. Not happy with customer service.

Also smaller offices are crippled by MB trying to update.

[IT_Guy]

So per Malwarebytes instructions, I cleaned MWAM off one of my endpoints that wasn't showing online in the cloud. Reinstalled it and rebooted.

Showed up in the cloud, scheduled a scan, scanned properly.

Exactly 12 hours after the endpoint registered it went offline in the cloud again with no amount of rebooting getting it to show up again.

Reinstalling the software is one thing if it's going to work, only 30 minutes per computer, should be done in about 20 hours. But if the fix only lasts 12 hours, then it's not a fix.

[KDawg]

Can you please collect for me the diagnostic logs from that problem endpoint?

In an administrator command prompt:

cd C:\Program Files\Malwarebytes Endpoint Agent\

Then use:

MBCloudEa.exe -diag

 

Please upload to our file upload site below and PM to let me know.

https://www.malwarebytes.com/support/business/businessfileupload/

 

Many Thanks

[King_Of_The_Castle]

Any updates on this? I'm also experiencing this problem on a few endpoints every week, getting tired of uninstalling / re-installing every time this happens...

 

Thanks.

[Kalrand]

We had 14 endpoints go offline this morning, all of the physical computers are up and running. To combat this since we've seen it in the past is to restart the Malwarebytes Endpoint Agent. We automate it a little by using PDQ Deploy with a script to restart the agent. Below is a snippet of the script we use, I left out some of our logging functions that you probably wouldn't need.

START /WAIT net stop "MBEndpointAgent">nul 2>&1
timeout /T 3 /nobreak>nul
START /WAIT net start "MBEndpointAgent">nul 2>&1

 

[King_Of_The_Castle]

On 5/10/2018 at 7:33 AM, Kalrand said:

We had 14 endpoints go offline this morning, all of the physical computers are up and running. To combat this since we've seen it in the past is to restart the Malwarebytes Endpoint Agent. We automate it a little by using PDQ Deploy with a script to restart the agent. Below is a snippet of the script we use, I left out some of our logging functions that you probably wouldn't need.

START /WAIT net stop "MBEndpointAgent">nul 2>&1
timeout /T 3 /nobreak>nul
START /WAIT net start "MBEndpointAgent">nul 2>&1

 

I tried this on a couple of endpoints to see if it would really prevent them from going offline. Unfortunately one of them went offline a couple of days after I scheduled the service to restart everyday. Even though this is not the kind of behavior I expect from a corporate product from MalwareBytes....I'm so disappointed.

[Kalrand]

After you ran the above did the Endpoints start reporting?

Another part of the script that we use is to delay the start of the Agent by this command, for some reason with the Agent and Windows 10 we had more offline endpoints if we didn't delay it's startup. The only issue is the next update of the agent will undo that.

sc config "MBEndpointAgent" start= delayed-auto

 

[Kalrand]

5 hours ago, Markpol said:

We are having the exact same issue. Have used the MB uninstall process on each, seems to help for a time then it come down again. Going to Services and restarting the service seems to help each time. We have both Win 7 and Win 10 running, the problem only seems to occur with Win 10

This is really not an acceptable state of affairs. I now must check the status daily (one would think a notification could at least be sent if a machine does not check in after X amount of time...) and then having to go to each endpoint to restart. We have moved most of our operations to Chrome OS (something I HIGHLY recommend to anyone who cares about security) but the remaining machines with this solution are in a constant state of checking and rehabilitating. 

MB, please fix this problem or we and I suspect many others will have to look elsewhere for a security solution.

Thanks,

Mark

I agree that this is troublesome and we also run into this issue, and have with Sophos before switching to Malwarebytes. I truely believe that the issue is with Windows 10 as it wasn't an issue before on Windows 7 (for Sophos or Malwarebytes). Do you have a means to run a batch script on the endpoints, like PDQ Deploy? The below is the script we use to help mitigate the issue.

START /WAIT net stop "MBEndpointAgent">nul 2>&1
timeout /T 3 /nobreak>nul
START /WAIT net start "MBEndpointAgent">nul 2>&1
sc config "MBEndpointAgent" start= delayed-auto

Granted this is a stop-gap and not a solution, on occasion will need to be reapplied as updates reset the Agent start mode. When the Agent is in a offline state it does not prevent Malwarebytes from protecting the system, this only affects the communication to the cloud (updating current policies and schedules) and on-demand scanning.

Edited July 12, 2018 by Kalrand