TonyCummins #1 Posted November 16, 2017 My endpoints have suddenly started to create lots and lots of events like the following since last night. Any ideas ?? 2017-11-15 19:58:15,647-07:00 [5 ] ERROR EAWebClient Error Getting /api/v1/machine/sync : System.AggregateException: One or more errors occurred. ---> System.Web.HttpException: HTTP Request failed. Http Code: 403 Reason:Forbidden Body Response: <html> <head><title>403 Forbidden</title></head> <body> <h1>403 Forbidden</h1> <ul> <li>Code: AccessDenied</li> <li>Message: Access Denied</li> <li>RequestId: 9F464C5517C0F32C</li> <li>HostId: RD+xNPrBcYrYxN9K4ZvcgnCKFDOo2CvkRjLUArxZ0KL41Ajt+NnoBwNKgLmwbXMc1oNdtR0notY=</li> </ul> <h3>An Error Occurred While Attempting to Retrieve a Custom Error Document</h3> <ul> <li>Code: AccessDenied</li> <li>Message: Access Denied</li> </ul> <hr/> </body> </html> at NebulaCommunication.HttpResponseMessageExtensions.<EnsureSuccessStatusCodeAndReadBody>d__2.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at NebulaCommunication.Auth.AuthTokenRefreshHandler.RefreshTokens(CancellationToken cancellationToken) at NebulaCommunication.Auth.AuthTokenRefreshHandler.ValidateAuthorizationHeader(HttpRequestMessage request, CancellationToken cancellationToken) at NebulaCommunication.Auth.AuthTokenRefreshHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken) at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at NebulaCommunication.MBWebClient.ConvertToSendAsync(HttpMethod method, HttpContent content, String requestUri, IEnumerable`1 headers, CancellationToken cancellationToken) at NebulaCommunication.MBWebClient.GetAsync(String requestUri, IEnumerable`1 headers, CancellationToken cancellationToken) at EAEngine.Http.EAWebClient.<Get>d__16.MoveNext() ---> (Inner Exception #0) System.Web.HttpException (0x80004005): HTTP Request failed. Http Code: 403 Reason:Forbidden Body Response: <html> <head><title>403 Forbidden</title></head> <body> <h1>403 Forbidden</h1> <ul> <li>Code: AccessDenied</li> <li>Message: Access Denied</li> <li>RequestId: 9F464C5517C0F32C</li> <li>HostId: RD+xNPrBcYrYxN9K4ZvcgnCKFDOo2CvkRjLUArxZ0KL41Ajt+NnoBwNKgLmwbXMc1oNdtR0notY=</li> </ul> <h3>An Error Occurred While Attempting to Retrieve a Custom Error Document</h3> <ul> <li>Code: AccessDenied</li> <li>Message: Access Denied</li> </ul> <hr/> </body> </html> at NebulaCommunication.HttpResponseMessageExtensions.<EnsureSuccessStatusCodeAndReadBody>d__2.MoveNext()<--- 2017-11-15 20:13:16,412-07:00 [5 ] ERROR EAWebClient Error Getting /api/v1/machine/sync : System.AggregateException: One or more errors occurred. ---> System.Web.HttpException: HTTP Request failed. Http Code: 403 Reason:Forbidden Body Response: <html> <head><title>403 Forbidden</title></head> <body> <h1>403 Forbidden</h1> <ul> <li>Code: AccessDenied</li> <li>Message: Access Denied</li> <li>RequestId: 940B59B08370EBFF</li> <li>HostId: ku/FbE1d4PL0IWS4YRcLTvwxYHAMMS/NIlaFY1jQOrynQwkUQ4JV3F5mty8PITEmnTQ5V2YG+6c=</li> </ul> <h3>An Error Occurred While Attempting to Retrieve a Custom Error Document</h3> <ul> <li>Code: AccessDenied</li> <li>Message: Access Denied</li> </ul> <hr/> </body> </html> at NebulaCommunication.HttpResponseMessageExtensions.<EnsureSuccessStatusCodeAndReadBody>d__2.MoveNext() --- End of inner exception stack trace --- at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions) at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken) at NebulaCommunication.Auth.AuthTokenRefreshHandler.RefreshTokens(CancellationToken cancellationToken) at NebulaCommunication.Auth.AuthTokenRefreshHandler.ValidateAuthorizationHeader(HttpRequestMessage request, CancellationToken cancellationToken) at NebulaCommunication.Auth.AuthTokenRefreshHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpMessageInvoker.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken) at System.Net.Http.HttpClient.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at NebulaCommunication.MBWebClient.ConvertToSendAsync(HttpMethod method, HttpContent content, String requestUri, IEnumerable`1 headers, CancellationToken cancellationToken) at NebulaCommunication.MBWebClient.GetAsync(String requestUri, IEnumerable`1 headers, CancellationToken cancellationToken) at EAEngine.Http.EAWebClient.<Get>d__16.MoveNext() ---> (Inner Exception #0) System.Web.HttpException (0x80004005): HTTP Request failed. Http Code: 403 Reason:Forbidden Body Response: <html> <head><title>403 Forbidden</title></head> <body> <h1>403 Forbidden</h1> <ul> <li>Code: AccessDenied</li> <li>Message: Access Denied</li> <li>RequestId: 940B59B08370EBFF</li> <li>HostId: ku/FbE1d4PL0IWS4YRcLTvwxYHAMMS/NIlaFY1jQOrynQwkUQ4JV3F5mty8PITEmnTQ5V2YG+6c=</li> </ul> <h3>An Error Occurred While Attempting to Retrieve a Custom Error Document</h3> <ul> <li>Code: AccessDenied</li> <li>Message: Access Denied</li> </ul> <hr/> </body> </html> at NebulaCommunication.HttpResponseMessageExtensions.<EnsureSuccessStatusCodeAndReadBody>d__2.MoveNext()<--- Share this post Link to post Share on other sites
CHall #2 Posted November 16, 2017 Out of curiosity, I checked my own Event Viewer and was shocked to see that I have over 2000 Errors from source "Malwarebytes Endpoint Agent", numerous events per day going back to October 4, which was the day it was installed (re-installed) on my PC. Not the same error either, too many different ones to be able to list here. I am loyal to Malwarebytes as both I and others have said in other posts, due to the fact that they always saved me from infected PCs in the past, for free, but my faith in this Endpoint Protection cloud-based product is utterly destroyed. My biggest complaint is a memory leak of some kind that requires me to restart all our PCs on at least a weekly basis, otherwise things start to break, lose contact with the cloud console (go offline), start high CPU usage on its service, and/or interfere with other applications on our PCs and network. At this point, I just live with it, frustrated with the choice I made for our company's AV/AM/AR protection until the annual subscription is up. Share this post Link to post Share on other sites
TonyCummins #3 Posted November 16, 2017 24 minutes ago, CHall said: Out of curiosity, I checked my own Event Viewer and was shocked to see that I have over 2000 Errors from source "Malwarebytes Endpoint Agent", numerous events per day going back to October 4, which was the day it was installed (re-installed) on my PC. Not the same error either, too many different ones to be able to list here. I am loyal to Malwarebytes as both I and others have said in other posts, due to the fact that they always saved me from infected PCs in the past, for free, but my faith in this Endpoint Protection cloud-based product is utterly destroyed. My biggest complaint is a memory leak of some kind that requires me to restart all our PCs on at least a weekly basis, otherwise things start to break, lose contact with the cloud console (go offline), start high CPU usage on its service, and/or interfere with other applications on our PCs and network. At this point, I just live with it, frustrated with the choice I made for our company's AV/AM/AR protection until the annual subscription is up. CHall, I too am a loyal "home' user who was so happy they had produced a cloud based product and convinced my IT manager to move from controlnow/solarwinds endpoint protection and am now really starting to regret my decision. I really wish i had stumbled upon these forums before i pulled the trigger on purchase as it really seems like i'm being used to beta test a product in a production environment. Share this post Link to post Share on other sites
CHall #4 Posted November 16, 2017 1 hour ago, TonyCummins said: CHall, I too am a loyal "home' user who was so happy they had produced a cloud based product and convinced my IT manager to move from controlnow/solarwinds endpoint protection and am now really starting to regret my decision. I really wish i had stumbled upon these forums before i pulled the trigger on purchase as it really seems like i'm being used to beta test a product in a production environment. Exactly. I went through weeks of technical hell right from the original purchase with no useful help from their "support" people before I finally found this forum and discovered I was not alone with these issues. I wouldn't have purchased had I read this forum beforehand. I too feel like a beta tester for this product. Also, we too are a user of ControlNow (Solarwinds), however I made the decision to keep that subscription and run it alongside MB for the first year, which was a fortunate move. Btw, that's one of the applications that MB starts interfering with when the memory leak is left to grow. Upon a fresh boot, the MBAMService.exe memory starts at about 250,000 K. Over a few days, it will climb on some PCs to 400,000 K - 500,000 K and at that point, those PCs stop updating ControlNow's virus definitions and appear with critical alerts on the ControlNow Dashboard. If I kill the MBAMService.exe process and restart it fresh, the ControlNow critical alerts go away and everything resumes normal operation. Left unattended, one-by-one our PCs will collect on the ControlNow dashboard with critical notifications. We've used ControlNow for years (since it was GFI) and never had issues until MB was installed. There are other applications on our network that also start breaking when the memory leak reaches that level. Back after the original installations, before I was aware of all the MB issues, the memory leak would grow to well over a gigabyte and CPU usage would spin-out and slow everything down. Share this post Link to post Share on other sites
TonyCummins #5 Posted November 16, 2017 You are describing the exact same scenario we are having ! I think the turnaround of support tickets is a lot to be desired....and sometimes it feels like its just a canned response to keep the ticket within a 2 day response time. **sigh Anywayyy....i'm really hoping they get the bugs worked out real soon Share this post Link to post Share on other sites
djacobson #6 Posted November 16, 2017 Whew, tough room! Anyway, what you are seeing in your logs is an attempt to update the builds, a new build began to be pushed out last night. This event looks like the downtime that was planned for the update. Here are the changes... New Features: Created new “Detections” page in the cloud console—combining the previous “Threats” and “Real-Time Protection” pages Added on-demand reporting—beginning with Detection Summary reports—that are generated in CSV format (additional reports coming soon) Improvements: Completed multiple improvements to the cloud console user interface Display software as a single application (“Malwarebytes Endpoint Agent”) in Add/Remove Programs window on the endpoint Drastically reduced events logged to Windows Event Viewer Enhanced the cloud console “Dashboard” page to include Real-Time Protection data Added additional information to detection details (with more to come in the future) Extended the download timeout period up to 30 minutes for software installations to assist with slow network connections Updated the end-user license agreement Fixed: Addressed an issue discovered when moving large numbers of endpoints between groups Fixed: Localized the Timestamp on the Quarantine page Fixed: Proxy settings correctly support hostnames Fixed: The handling of Unicode characters in scan results Share this post Link to post Share on other sites
djacobson #7 Posted November 16, 2017 The Solarwinds conflict cannot be fixed due to both programs utilizing Windows BFE service for block techniques. You don't have to remove it but you'll need to choose which program you want doing your web blocking. Share this post Link to post Share on other sites
TonyCummins #8 Posted November 16, 2017 @djacobson Is there any way to get a notification when a new update is scheduled to be pushed? Had i known the event viewer "noise" was benign and maybe generated due to a new push / update, it might've saved this request for support. Tony Share this post Link to post Share on other sites
djacobson #9 Posted November 16, 2017 (edited) Update notifications are always on this page - https://support.malwarebytes.com/community/business - I know it's a bear to have to keep track of yet something else on some other web page, but if you are having troubles, or notice weird stuff in the logs, try to make a habit of checking that spot just to see if it is an outage we are working to address or update in progress we are pushing out. Our PM for the cloud product also mentioned he would be pushing an email out to all active cloud subscribers. Edited November 16, 2017 by djacobson Share this post Link to post Share on other sites
spnkzss #10 Posted November 17, 2017 14 hours ago, djacobson said: Our PM for the cloud product also mentioned he would be pushing an email out to all active cloud subscribers. Pretty sure I never saw that Communicating stuff like this WITHOUT us having to go looking for anything would probably squelch 90% of our bitching. As I've been saying since day one, it's the lack of communication that frustrates me the most. Problems will occur, but as long as I hear about it and WHY, then I'm normally ok about it. Share this post Link to post Share on other sites
djacobson #11 Posted November 17, 2017 We are working on getting the resources to do this. It'll be through the MyAccount portal, though there is a lot of backend work which needs to be completed first before it can be built, so for now the announcement spot on the KB page is the best we got. Share this post Link to post Share on other sites
spnkzss #12 Posted November 17, 2017 Could we get a post on this forum? That would probably reach the most of us. Share this post Link to post Share on other sites
djacobson #13 Posted November 17, 2017 That's a good idea, I'll see if I can make that happen and make it a pinned topic in the EP section. Share this post Link to post Share on other sites
spnkzss #14 Posted November 17, 2017 Perfect. Thank you. Share this post Link to post Share on other sites
TonyCummins #15 Posted November 17, 2017 1 hour ago, djacobson said: That's a good idea, I'll see if I can make that happen and make it a pinned topic in the EP section. Thanks Share this post Link to post Share on other sites
