Yesterday's Database Update Issue

[RubbeR DuckY]

It saddens me to report that at around 3 PM PST yesterday, Malwarebytes released a definitions update that disabled thousands of computers worldwide. Within 8 minutes, the update was pulled from our servers. Immediately thereafter, users flocked to our support helpdesk and forums to ask us for a fix.

I want to offer my sincere apology to our millions of customers and free users. I started this company because I thought everyone was entitled to malware-free computing. We acted overzealously in that mission and realize far superior procedures around updating are needed. More was expected of us, and we failed.

So what's my promise to you? Working day and night, we are commissioning several new resources to stop this from happening again. We are building more redundancy to check our researchers' work and improving our peer review.

Here's what we've done to address the issue. We immediately wrote a tool to fix the issue and published instructions on our forums. If you are affected by the issue, please visit the page. If you need assistance or are uncomfortable performing the fix manually, please contact our support team. We have our entire support staff answering tickets feverishly. Tickets are being answered within an hour, and we will reach out to you by phone if e-mail support is not enough.

Please, once again, accept an apology on behalf of our entire company. Let's get you fixed up and back to a malware-free existence!

Marcin

[Agent88]

I purchased copies of Malwarebytes for all 3 of my PCs some time ago. I am a busy person, so I usually take the "set it and forget it" protocol. Today, for the first time since a member, I got a "blocked" message popup while on the net. It went away too quickly before I could respond, so I opened Malwarebytes and ran a scan. I found no threats, but I decided to check the log. That's where I saw the IP of the blocked address. I ran a reverse lookup and found the address in a Dallas suburb (supposedly).

This being a good time to learn a little more about Malwarebytes, I thought I would see if you had a community forum where I could see what was going on. Walla! You did and I signed up. First thing I did was check the news, and that's when I got a scare! It appears you admit your program shut down many customers just yesterday. This would have been disastrous for me if I had been one of those unlucky ones. I would not have known what to do since I never went beyond the download and install of Malwarebytes Pro several months ago.

It was a little confusing after signing up for a forum account when I clicked on support, only to find I needed another log-in account. Never-the-less, I surfed the info on "Chameleon" and found out this was the help I would have needed (possibly). I am still unsure if the PC I am on is blocked, would I be able to log into my Malwarebytes support on another, since they are all on the same LAN.

My point is, that although I appreciate your transparency about yesterday's fiasco (which I thankfully was oblivious to until today) however, your support advice is, like so many other tech support sites, oriented towards IT professionals. You should hire someone trained as a tech writer that will recognize the need to translate technical jargon and procedures into non-IT professional commentary. To wit: Express the advice to view the video on Chameleon STRONGLY, before an attack occurs, etc. When the ship hits the sand, anxiety levels rise and interferes with the ability to sort out the unknown procedures for the moment's need. We should always heed the Boy Scout advice "Be Prepared".... but alas, we tend to get too busy to read mountains of computer-ese. Make it simple, make it short. In other words, structure it for the less than professional IT customer.

And what is with this input request for my "Cleverbridge" number? I am sure I have a receipt for this somewhere, but it may take hours to locate. Why not have a registration site where this number is stored for retrieval if needed much like other vendors (Acronis, Symantec, etc.)? Sheesh! Gimme a break!

[whonew]

I am not so sure you should feel bad at all, for my problem with this began on the 14th when I plugged my computer in after bringing it home from a computer repair shop. The young man who did that repair got it back yesterday the 15th to give me a CD for my new motherboard. I needed drivers, after I reformatted it my self and lost all that he had done for me.

[Penelope]

Thank you for your apology, Marcin. Two of our four computers were completely disabled because of this database update and it took us most of last evening to be able to even boot them again. So I appreciate you taking responsibility for what happened and working so quickly to make it right.

[era]

I have been a faithful, paying customer of Malwarebytes for years and I think it is a wonderful product. I was very upset by what happened yesterday and I still have one pc that won't boot. You acted quickly with a fix, you're offering full support, and sharing what happened and what you're doing to prevent it is what I wanted to hear. Mistakes happen and it's a lot about how they're handled once made. I believe you truly care and stand behind your company and your product. I appreciate all you've done and all you're doing. Best of luck and thank you, Marcin.

[fallenangeloa]

I wanted to also make sure that I thanked you for being transparent and quickly establishing exactly what had happened as well as a prompt fix. I do understand that these issues do come up within the AV sphere, so I want to also apologize for any harsh words that I may have said while going through the issue yesterday. I cannot speak for those that I've helped - I doubt they will continue using this program. As for myself, the fact you have handled this exceptionally well in your honesty has renewed my faith in this company and its product. Again, thank you.

[robkarp]

It saddens me to report that at around 3 PM PST yesterday, Malwarebytes released a definitions update that disabled thousands of computers worldwide. Within 8 minutes, the update was pulled from our servers. Immediately thereafter, users flocked to our support helpdesk and forums to ask us for a fix.

I am one of your thousands and am using the paid PRO version. I could not repair my PC (it literally ate itself to the point that it was impossible to execute any file at all -- nothing could be run) and had to restore from my last disk image which, thankfully, was only an hour old at the time of this disaster. I am writing to strongly urge you to change the default set-up MBAM configuration to not automatically quarantine suspected files. This one change would have saved us thousands (and you) considerable grief yesterday. I never paid much attention to the setting as I trusted your default options. My bad. Again, suggest you change this.

[neant]

I am one of your thousands and am using the paid PRO version. I could not repair my PC (it literally ate itself to the point that it was impossible to execute any file at all -- nothing could be run) and had to restore from my last disk image which, thankfully, was only an hour old at the time of this disaster. I am writing to strongly urge you to change the default set-up MBAM configuration to not automatically quarantine suspected files. This one change would have saved us thousands (and you) considerable grief yesterday. I never paid much attention to the setting as I trusted your default options. My bad. Again, suggest you change this.

I think it's a good idea, please consider it!

[mountaintree16]

Thank you Marcin for your statement and explanation.

Thankfully I was not affected as our machine was off and not in use all day yesterday.

I agree, I think that it would be worth considering not having that option ticked by default, but to make users aware of the option if they do want it ticked. I unchecked that option on my computer after learning about this, just in case. I myself would prefer to be asked by Malwarebytes if such a threat is found, anyway, just in case.

[DarkSnakeKobra]

I agree, I think that it would be worth considering not having that option ticked by default, but to make users aware of the option if they do want it ticked. I unchecked that option on my computer after learning about this, just in case. I myself would prefer to be asked by Malwarebytes if such a threat is found, anyway, just in case.

I agree.

Normally when I install Windows I adjust the settings and be sure to check that off, but somehow I managed to fail to uncheck it leaving me with a bugged computer. This wouldn't have normally affected me if I didn't miss this setting. Fortunately I knew this was a false positive right away after seeing taskeng quarentined and swiftly disabled MBAM and restored everything before too much damage was done. I believe I had less then 50 files quarentined.The system appeared to be usable, but my webcam started with an error so I reinstalled Windows to be on the safe side.

[mtieu]

There was no quick way to repair hundreds of computers on the Enterprise level. Computers had to be put in safe mode but that was not possible because of physical locations of these computers. Me and my team were up to very early this morning fixing these computers. You should have dedicated support for your business and Enterprise customers who pay alot of money for this product and get the same email help that anyone else gets. Im not saying that we have to be treated any differently, but when you have hundreds or even thousands of computers to deal with instead of just one, it would have been very helpful if I had an engineer working with our team to get this resolved. This is one of the fatal mistakes, along with not testing that update to begin with. The problem is though, is that there is no reason that core system files in windows/system32 should ever be flagged as a trojan downloader. If this is the competency level of your developers and engineers working on your definitions, I strongly suggest you step your game up. Sounds like the work of a scorned employee or someone already on their way out with a chip on their shoulder, because if it isn't, I am highly dissapointed in your product and procedures. We licensed 2100 computers and I believe we are going to scrap this product. I think its a great product, honestly I do, but management is not so convinced and we are looking into other options and products. I recommended this product for our environment and championed the deployment, and whatever the process may be, it has failed us. No excuses. I thought MEE was enterprise ready, but I was sorely mistaken. You dont have the testing and quality control of what it takes to be in the Enterprise game just yet. Close, but not there yet. I feel that I should have waited until your product was throughly tested and developed into a 2.0 or something. I was afraid I was going to be handed my walking papers today because of your "great" product. Good thing I got it all repaired, but I believe the damage is already done. This may be the death knell from corporations wanting to buy into MalwareBytes Enterprise Edition. I have the pro version at home and I love it, but when thousands of computers are affected at a large company, it leaves a really bad taste in everyones mouths, from management down.

Sadly, I am not the only Administrator that feels this way, because its all over the place now and cannot be trusted Enterprise level. I really liked this product and I even praised it in our meeting we had yesterday morning. This situation made me eat my words and made me look really bad honestly. I don't know how to fix this and i would really like to get a refund on this purchase, but in the end, its going to take some heavy convincing to get this redeployed out in our environment from a management stand point. Many man hours and manpower was wasted because of your 8 minute untested update.

[catscomputer]

I've used and recommended Malwarebytes for several years now, and have great faith in the product and the integrity of the company behind it. I have really appreciated the advice I've received from knowledgeable and trustworthy people here on the forum (trusted advisors, experts and MBAM staff especially) and thanks to this place I've learned an enormous amount about safe computing, and computers in general. I also recognise that you provide an excellent product for free, help on the forums and through the support desk for free (not always related to infections or malwarebytes issues either), and the paid product is extremely good value considering it's a lifetime transferable license. You also own your mistakes which is another thing I value.

That said, the reason I am not in the same boat as many people who received the bad update yesterday is because I have disallowed automatic quarantining, and, when I got the first (of many, many) pop ups yesterday, my default reaction was to not trust Malwarebytes and allow the process(es). Why? Because this is the forth realtime detection I've had from MBAM since December 2012, and all four of them have been false positives. That, and it came within seconds of an update. I was also caught up in these two FPs in February: local host 127.0.0.1 , Trojan.Backdoor.MRX and this one in December: notepad.exe. All of these FPs (with perhaps the exception of the uninstallers detected as Trojan.Backdoor.MRX, except that it detected so many different uninstallers that it would be unusual not to have at least one or two of them on a system) were system files and things found on all windows computers, and I can't help but ask the question how any of these four bad updates made it out onto your update servers in the first place. Surely they are tested before they are released?? Please understand - my issue is not so much the number of FPs but rather what they were. This is what has moved me to post. I FULLY accept that FPs happen, and I have always been impressed with the very swift action by MBAM to correct them, but when they happen with everyday files found on all machines I am left feeling untrusting of MBAMs pop-ups notifying me of a malicious processes, and it really shouldn't be that way around. It should be that I trust MBAM and treat detections as malware until proven otherwise.

I continue to have utmost faith in your product in it's detection and cleaning of real malware, and the skills and integrity of the people in your company - it remains 'must have' software for me, but I really sincerely hope that these new measures you're putting in place ensure thorough testing on all OS' before updates are released? I want to be able to trust pop ups from MBAM again. I know FPs will still happen even with this, but not on such common place system files like these ones were and the ones in the links above.

Thanks for the transparency Marcin, and also for allowing feedback from your userbase.

I really feel for MBAMs crew having to deal with the aftermath of this. I really hope the fall out is as minimal as possible.

[John L. Galt]

I've used and recommended Malwarebytes for several years now, and have great faith in the product and the integrity of the company behind it. I have really appreciated the advice I've received from knowledgeable and trustworthy people here on the forum (trusted advisors, experts and MBAM staff especially) and thanks to this place I've learned an enormous amount about safe computing, and computers in general. I also recognise that you provide an excellent product for free, help on the forums and through the support desk for free (not always related to infections or malwarebytes issues either), and the paid product is extremely good value considering it's a lifetime transferable license. You also own your mistakes which is another thing I value.

That said, the reason I am not in the same boat as many people who received the bad update yesterday is because I have disallowed automatic quarantining, and, when I got the first (of many, many) pop ups yesterday, my default reaction was to not trust Malwarebytes and allow the process(es). Why? Because this is the forth realtime detection I've had from MBAM since December 2012, and all four of them have been false positives. That, and it came within seconds of an update. I was also caught up in these two FPs in February: local host 127.0.0.1 , Trojan.Backdoor.MRX and this one in December: notepad.exe. All of these FPs (with perhaps the exception of the uninstallers detected as Trojan.Backdoor.MRX, except that it detected so many different uninstallers that it would be unusual not to have at least one or two of them on a system) were system files and things found on all windows computers, and I can't help but ask the question how any of these four bad updates made it out onto your update servers in the first place. Surely they are tested before they are released?? Please understand - my issue is not so much the number of FPs but rather what they were. This is what has moved me to post. I FULLY accept that FPs happen, and I have always been impressed with the very swift action by MBAM to correct them, but when they happen with everyday files found on all machines I am left feeling untrusting of MBAMs pop-ups notifying me of a malicious processes, and it really shouldn't be that way around. It should be that I trust MBAM and treat detections as malware until proven otherwise.

I continue to have utmost faith in your product in it's detection and cleaning of real malware, and the skills and integrity of the people in your company - it remains 'must have' software for me, but I really sincerely hope that these new measures you're putting in place ensure thorough testing on all OS' before updates are released? I want to be able to trust pop ups from MBAM again. I know FPs will still happen even with this, but not on such common place system files like these ones were and the ones in the links above.

Thanks for the transparency Marcin, and also for allowing feedback from your userbase.

I really feel for MBAMs crew having to deal with the aftermath of this. I really hope the fall out is as minimal as possible.

First of all, as a 100% volunteer here with no financial ties to the company whatsoever, let me say this: Thank you for stating your concerns in a very objective manner.

That being said, I agree with you one hundred percent - I missed seeing the last few for whatever reason, and was not aware that this sort of thing has been much more frequent than I supposed. And Marcin has already stated that, in continuing with his noteworthy transparency, has stated in another thread that he is going to make public the steps they take to prevent this from ever happening again. As for whether they match your hope - well, one he announces them you'll be able to judge them yourself. I too will be watching carefully - but he's a smart guy, and he has a reputation to uphold - and I'm confident that he'll come through with flying colors. And piggybacking on that will be MBAM coming through with flying colors as well, with a new and much improved testing for def files as well as some sort of additional checking after they are pushed public so that this does not happen again.

And, yes, you're right - I feel bad for the support staff, both here and those out in the wild that have to support numerous customers that have lost access to their computers - but I feel even more badly for those who were hurt in Boston.

All this, right here - it's not even close to what those people went through (and are still going through).

[XweAponX]

It saddens me to report that at around 3 PM PST yesterday, Malwarebytes released a definitions update that disabled thousands of computers worldwide. Within 8 minutes, the update was pulled from our servers. Immediately thereafter, users flocked to our support helpdesk and forums to ask us for a fix.

I want to offer my sincere apology to our millions of customers and free users. I started this company because I thought everyone was entitled to malware-free computing. We acted overzealously in that mission and realize far superior procedures around updating are needed. More was expected of us, and we failed.

So what's my promise to you? Working day and night, we are commissioning several new resources to stop this from happening again. We are building more redundancy to check our researchers' work and improving our peer review.

Here's what we've done to address the issue. We immediately wrote a tool to fix the issue and published instructions on our forums. If you are affected by the issue, please visit the page. If you need assistance or are uncomfortable performing the fix manually, please contact our support team. We have our entire support staff answering tickets feverishly. Tickets are being answered within an hour, and we will reach out to you by phone if e-mail support is not enough.

Please, once again, accept an apology on behalf of our entire company. Let's get you fixed up and back to a malware-free existence!

Marcin

Ok - Apology Accepted. The next update for Malwarebytes should have Auto Quarantine disabled by default. Im not really that angry, but it's my clients, they are seething with ire about this. I know you realise, that this virtually ground businesses which depend on networked systems to a sudden halt.

I had thought you had been hacked and anonymous had pirated your update package. I really didn't know what to make of this when the calls started streaming in - I thought it was a new world wide virus, or a bad Microsoft update, or maybe Microsoft reached in and shut our systems down. All of the Windows 7 systems I maintain had Black Screens of Death. I googled and googled and found no explanation. I was finally at my 3rd business of the day when I made the correlation in my mind - All of the affected systems had updated versions of Malwarebytes.

So, I hope you realise, I am not that angry, it's my clients who are salivating and making me remove the program - Hopefully I will get them to trust this software again ion the future, once they see that without it, they have no Malware protection at all.

[Rompin Raider]

BTW I followed your fix for Windows 7 (I have Pro and was on line when it hit the other day)...it left 7 files quarantined; wouldn't disappear after "restore all"...checked the OS and all 7 were in tact so I deleted them and good to go. Thanks for the immediate response. I had this happen before on one of the top internet security programs....so I know it does happen sometimes! Cheers!

[Coss]

I read a lot of these replies, and I have a question. I was also one of the lucky ones that watched my machine "eating its self" and I was able to do a restore. But one reason why I was able to do this was because as soon as I saw the pop-ups coming up in rapid succession, I knew enough to turn my machine off. I didn't sit there going oh me oh my, I wonder why it's doing that. Seeing something like that happen told me there is something very wrong happening.

Common sense is what kicked in. I didn't panic, or try to ignore it. Common sense told me something was wrong. So my instinct kicked in and I pulled the plug. I went to another machine and started looking for a cause. I don't understand how 155+ end users didn't see something like that happening and not respond to it. If the admins turned off the pop-ups, shame on you; you have no one to blame except yourself.

This was bad, no doubt about it, and by all practical standards it should have never happened. But it did; venting about it and saying that you'll never trust a product again means that you must see yourselves as perfect. Mistakes happen. Look at how to deal with it, and make contingency plans for unplanned disasters. Yelling about something, and saying to make unreasonable changes isn't the answer. Deal with it, don't dwell on it, and move forward.

Let the hate mail roll, I know this is going to make a number of people unhappy and they will let me know about it. Shows the depth of their maturity.

[mountaintree16]

Thankfully I was not affected, but had I seen this happening, I would have shut my machine off immediately too I think. I now have the option UNchecked, just in case, although I do trust that this will never happen again. Malwarebytes is run by good people and they never meant for this to happen. I trust that it will never happen again.

Unfortunately, for many people perhaps it happened while they were not physically present at their computers, and/or to hundreds of work computers with the Enterprise edition, which of course makes it all the worse, when its out of control and happening to a lot of machines... not a good scene all around. I trust that Malwarebytes will do their best to make it right, and I still trust their product and will continue to recommend it. It does make me sad to learn of what happened, especially for those who are not so tech-savvy, and those who had it happen on a lot of machines at once... that is certainly very difficult to have to deal with.

I believe the whole point of the option being auto-checked is for people who don't really know what they are looking at when asked by their security software what to do with a found file. As I believe AdvancedSetup mentioned, most if not all, Anti-Virus programs have this feature auto-enabled as well. I am no computer expert by any means, but I consider myself to be an above average user (largely thanks to this forum), so I tweak the settings on my security software to fit my abilities/knowledge, but when helping someone who is not so technologically inclined, I use the auto features and trust that the products won't wreak havoc in the very unlikely event that something like this happens. I believe Kaspersky suffered an unfortunate event a few years back, and I think McAfee may as well. Malwarebytes is certainly not the first and unfortunately won't be the last, but I am confident that this is the last time this will happen for Malwarebytes.

[bluesman821]

Fortunately I was not effecting by this. As a precaution I have have unchecked the auto quarantine. I know these thing happen,but I hope better precautions will be installed. This will not cause me to abandon MWB. I will continue to use it on both computers.

[Coss]

Fortunately I was not effecting by this. As a precaution I have have unchecked the auto quarantine. I know these thing happen,but I hope better precautions will be installed. This will not cause me to abandon MWB. I will continue to use it on both computers.

So when MB finds a bad file, what is it going to do with it? Log it? A lot of good that will do. Delete it? Worse then quarantine because then the files it fines are gone.

Or since you did turn it off, what do you do? Or what did you want the program to do?

I agree with Mountaintree16, a number of good companies have had bad things happen. If you judge a manufacturer by one problem, what are you left with? 3rd or 4th best? Might as well just not put anything in, and hope that your Internet browsing never takes you to an infected site. MalwareBytes is rated #1, and for good reason, they produce one of the best anti malware products out today. They had a stumble, and from the opening statement, they are taking steps to make sure it doesn't happen again.

Proof is in the pudding. I will continue using MB, and have no problem believing that they will stand behind their product. For the enterprise people; I've worked in those situations for many years and I understand how the end users are upset. Good reason to be, it was a bad experience. But once they stop yelling, and if you have a decent IT lead, they will stop yelling and go back to work trusting that the IT department will do what is necessary. The people in IT, sorry that your work load just skyrocketed, but if you've been in the career for a while, you'll know it wasn't the first time, nor will it be the last, that's part of this life.

My experience? I started doing programming in 1967, got out of it in 71, went back into IT in 85 and I'm still doing it. So I do have just a bit of experience. I have my own consulting company (and do Very well thank you); I just had a hard time having other people telling me what and how to do things.

[AdvancedSetup]

Yes I worked at a very large International company years ago as one of the supervisors on the security team. We had one of our remote offices install a firewall product on all the desktops that was not approved by the main office as it had not been vetted. Turns out it did have some bad code in it and it toasted about 500 desktop computers so they could no longer boot. But because we had a well trained staff and had spent a lot of time and money on the infrastructure to recover from something like that, we were able to repair and bring up all of them within 2 days. But now days not many companies seem to have the money to invest in that sort of infrastructure.

[rgabbard]

Ok, so what I don't understand... I have about 80 clients on the enterprise version. So far only about 15 were actually affected by the bad update, but not until a scan was run at 6pm. So why didn't my clients update between 3pm and 6pm? Update settings are set for 10 minutes, and the server and all the clients affected were online all day.

[John L. Galt]

Quite possibly because there was no definition update in that period.

[rgabbard]

Quite possibly because there was no definition update in that period.

Possibly, but that means the other clients that I know were online should have been affected as well.

[John L. Galt]

Also, 3 PM PDT - which is 6 PM EDT

[rgabbard]

Also, 3 PM PDT - which is 6 PM EDT

Good catch, totally oblivious to that... Been a bit scatter brained since this happened. Already had a lot on my plate, then had to take time out to resolve this. Still cleaning up residual issues.