Win7 freeze, Event ID 36887 from "scan file system"

[LiquidTension]

3 hours ago, DSperber said:

If the mouse and keyboard are unresponsive, are you saying that in fact the system will still recognize that I've held down right-CTRL and pressed SCROLL LOCK twice?  This sequence of keystrokes will actually be seen, even though my normal use of keyboard is frozen?

It depends on the nature of the freeze/deadlock. We've found that in some cases, the designated key sequence is able to initiate a blue screen despite other keyboard input being unresponsive. Ultimately, it's trial and error, so I don't have a definitive answer for you.

If you find the key sequence does not have the intended effect, we can move onto other troubleshooting. I would still like to explore the possibility of Macrium Reflect's involvement given we have strong findings that explicitly implicate the program in the cause of a freeze/deadlock.

[DSperber]

Well, not amazingly I suppose... it froze!  Very quickly in fact.

Onscreen clues showed 2:53AM (clock time from Aida64 which had frozen). I had re-booted at 1:06AM. Clockwise clock was still progressing, as was Windows clock in lower-right corner. PERFMON was still moving (note that on ASUS I typically see PERFMON still moving after a freeze, whereas for some reason on M910t PERFMON is also typically frozen).

Note that 2:53AM happens to be the overnight once-per-day scheduled MBAM scan time. I don't know if this is coincidence or relevant, but looking at "reports" I do not see last night's scan report.  There is only one scan report present, from 4/1 at 11:58PM, which is when I reinstalled MBAM last night and did my first scan.  So the 4/2 2:53AM scan report should have been the second report, if it was actually performed completely.  Obviously not.

Remarkably when I pressed right-CTRL and then SCROLL LOCK twice, sure enough a BSOD was tripped! And it appeared to be taking the complete dump as the displayed completion percentage advanced until it finished.  That's the good news.

Unfortunately the bad news is that there is not a MEMORY.DMP file on my system. I don't know where it might have gone if it's supposed to be in |Windows, if it actually did get created, but it is nowhere on any drive.  Certainly not where it's supposed to be.

One more note (and this is quite typical).  The first re-boot didn't actually complete.  The network icon (in System Tray) had a little blue mark on it indicating no internet access.  Actually looking at the Event Log for that time I see the typical entries indicating that DHCP didn't assign an IP address to ASUS. I don't know if the router is "stuck", or if the Intel I219 network adapter in ASUS is "stuck" or what, but this is quite common on the first TWO re-boots after a freeze. In fact I typically have to eventually re-boot THREE times before all of the System Tray icons finally populate quickly and completely, including the network icon properly showing "internet access".

Also not atypical, that first re-boot itself FROZE!!!  At the moment there still was not a complete set of System Tray icons there, including no MBAM, no Dyn Updater, and no Team Viewer. All of these are internet-related apps which I assume are negatively impacted and prevented from starting properly from the underlying issue with the network adapter "failing" and that I don't have an IP address assigned or resulting internet access through the router. I didn't want to force BSOD again, because I didn't want to lose what I thought would have been the original MEMORY.DMP from the 2:53AM freeze, so I just re-booted for the second time.  I don't know if this situation is what disappeared the MEMORY.DMP file which might actually have been taken, or if in fact something about my configuration truly prevented a complete dump from being produced even though everything sure looked like it was happening at the BSOD I did force successfully.

Anyway, the second re-boot was eventually successful in presenting everything in the System Tray, although it took quite a while. I then re-booted a third time, and now it was much quicker and more normal and everything was present... and I have internet access. Note that the HTPC hardware includes Ceton driver access to an external Motorola DTR700 digital tuning adapter box, connected via USB. It's virtually guaranteed that the USB interface to the tuning adapter is unsuccessful for the first few re-boots after a freeze. Much like how the network adapter initialization seems to illustrate issues right after a freeze, I also absolutely see similar instability in USB-connected devices right after a freeze. Some USB devices simply don't seem to be present event though they are plugged in, and sometimes I have to unplug them, re-boot, re-plug them, and now they're back.

So the physical effects of a freeze are far-reaching in impacting hardware in the machine. I don't believe it's "failing hardware" that is causing the freeze. I believe it is MBSM and the freeze which "locks up" NIC and USB hardware.  Eventually, after three re-boots typically, everything is finally cleaned up and everything is back to normal. The DTR700 is back and "locked" to the head-end properly, all USB devices are working, network access is present, and all internet-based apps are again working just fine.

 

So, unfortunately I have no MEMORY.DMP for you at the moment. I don't know what happened. I was quite sure I'd calculated 32 x 1024 = 32768MB as the PAGEFILE size I needed to hold all of my 32GB of memory, so that if I allocated 32800MB as the initial size (and 48000MB as the maximum size) this would be acceptable to hold a complete dump. It sure seemed to be being produced, and yet it's not present.  Are my sizes correct?

Is there a problem because C is on M.2 NVMe SSD? Does it require space on HDD or SATA SSD?  Is there anything else you want me to double-check to ensure I've configured things properly to produce the full dump?

Anyway, I've reset everything, and will now just to have to continue waiting for the next freeze to hopefully occur again sometime today or tonight... if history can be depended on to repeat itself.

[DSperber]

One more note...

I still have MBAM+MSE uninstalled on M910t, which is still running Bitdefender. It has now been up for 13 days without a freeze or re-boot!!

And although I was concerned that by reinstalling MBAM (and the other software products) on ASUS I might also bring down M910t if ASUS froze, the evidence from last night indicates that this is not the case. Apparently the fact that I'm not running MBAM on M910t makes it "immune" to freeze, even when ASUS has frozen.  Even though the two machines should be network-handshaking (since they each have mapped network drive letters assigned for partitions hosted by the other machine, so if one freezes you'd think that would eventually cause problems) and probably weren't doing so properly or completely because ASUS was frozen, that didn't seem to result in M910t also eventually freezing. Interesting observation. Again, MBAM is not present on M910t but Bitdefender is.

So M910t continues to count past 13 days without a freeze, even though ASUS has now frozen almost immediately after reinstalling MBAM.

[DSperber]

Just for the sake of information, the following screenshot shows my normal situation on ASUS, as far as background programs and processes that are always active.

Normally the "Time" value shown from Aida64 (supposedly updated every 4 seconds) matches the clock value shown by Clockwise, and also the clock value shown by Wndows in the lower-right corner. When freeze occurs the "Time" in Aida64 has stopped updating, and is simply stopped at the time of the freeze. But the Clockwise clock value continues to progress normally every second, and PERFMON's "EKG" display for CPU% also continues to update every second.

And finally, the output of DUMeter (upper right corner) which is also supposed to update every second has frozen. The last 5 minutes or so of network traffic are displayed. One curious thing I've noticed is that about 30 seconds ago there is always a "straight-up spike line". In other words some kind of network traffic (either from ASUS to M910t, or from ASUS to Ceton TV Tuner card, or from ASUS to somewhere on the internet from some app) occurred about 30 seconds before the freeze.  Don't know what that might mean, and I have no way to determine what the source of that "spike" is, but it always seems to be present on my frozen screen.

[DSperber]

Ok. I looked at the System Event Log and found an Event ID 12 error message from Subsys-SMSS this morning, when I had created that BSOD which presumably was also going to generate MEMORY.DMP:

"The crash dump file could not be created due to a lack of free space on the destination drive. Increasing the amount of free space on the destination drive may help prevent this error."

Although my C-partition certainly had an adequately large PAGEFILE.SYS, I suppose there wasn't also sufficient FREE SPACE in order to create the MEMORY.DMP output.  This additional requirement had not been stated in any of the MS articles, only that PAGEFILE.SYS needed to be at least MEMORY+100MB. Actually I thought it said MEMORY+1MB (for header), but articles I looked at today had +100MB, not 1MB. Or, maybe I just misread what was there.

I've now cleaned some things up on C, as well as shrinking the partition just to its right by 30GB and adding that newly available 30GB to C.  So there's now 70GB free on my C partition, even with the 32.2GB PAGEFILE.SYS present. Note that I again enlarged my PAGEFILE.SYS to now be 33000MB, rather than 32800MB... to account for the +100MB instead of +1MB. Anyway, 33000MB for PAGEFILE.SYS obviously is now acceptable either way.

To test this out I then manually triggered a BSOD (with right-CTRL + SCROLL LOCK twice), and sure enough now I was able to produce MEMORY.DMP. Came to about 34GB. I suspect this was definitely not possible this morning with the freespace and partition size that existed then, so I guess that's what went wrong this morning which prevented it from being produced. I've now deleted this test MEMORY.DMP in anticipation of now successfully creating a new MEMORY.DMP after the next freeze whenever it occurs.

So now it's just sit back again and wait. Note that when the freeze occurred last night at 2:53AM, I was not using the machine at all. It was just a "spontaneous freeze", perhaps tied to the MBAM scan which was just starting at that moment, or from something else. But whatever the cause I mention that I was asleep, and ASUS was completely "idle".

[DSperber]

Ok. Another "spontaneous" freeze occurred on ASUS at about 9:38PM this evening. I hadn't worked on the machine since re-booting it this afternoon at 4:30PM after enlarging the PAGEFILE one more time, and enlarging the C-partition so there was now about 70GB of free space in order to be adequate to hold the 34GB MEMORY.DMP file.

So this time, right-CTRL + SCROLL LOCK twice genuinely produced MEMORY.DMP successfully.  I've zipped it, and uploaded it to a file hosting site.

I have sent you a PM with the URL for you to download it.  It's about 2.0GB in compressed ZIP form, which expands to about 34GB.

Let me know if you are able to retrieve it without problems, and then eventually what your analysis discovers. If you'd like me to upload any additional dumps, or if you want me first re-configure something or try something special before trying generate the next freeze/dump, just let me know.

[LiquidTension]

Thank you for the update! I will get back to you shortly with the results of the dump analysis.

[DSperber]

I see that you haven't yet posted any status update on the early analysis of the dump I sent.  So presumably you're still working to try and find "giant clues" as I like to call them (i.e. unexpected surprises or absolute anomalies or "very interesting repeat coincidences" or "should not occur conditions which nevertheless have occurred again") which might prove useful if pursued deeper.

To possibly assist you, I make my own observations.

(1) I uninstalled Bitdefender and reinstalled MBAM + MSE late Monday evening 4/1. I also reinstalled two other software products I had also originally uninstalled when uninstalling MBAM + MSE and migrating to Bitdefender: (a) Team Viewer 14, and (b) Dyn Updater 5.4.6.

Although I had my own separate reasons for being unhappy about certain behaviors of Team Viewer and Dyn Updater in my Win7 HTPC systems, like with MBAM I also used Team Viewer and Dyn Updater in ALL of my otther 20 remotely maintained machines for friends and family (again which are both desktop and laptop, as well as Win7, Win8.1 and Win10.) all of which were not freezing. Nevertheless, when deciding to take the extreme action of migrating to Bitdefender from MBAM + MSE I decided to go even more extreme and remove both Team Viewer as well as Dyn Updater, since neither of these was absolutely crucial to my Win7 HTPC machines or how I used them or accessed them remotely if necessary.

So, truth be told, after the decision to dedicate my ASUS machine to this suspected MBAM-related freeze-symptom exploratory research, I put back everything "the way it was"... including not only removing Bitdefender and reinstalling MBAM + MSE, but also reinstalling Team Viewer and Dyn Updater as well. Note that I also have RealVNC Server/Viewer software installed on all of my machines (to support remote access to this machine, as well as remote access to other machines) but this has never been suspect and has always been installed and present and running in the background both when freezes occurred as well as when not occurring.

So when the first freeze occurred several hours after the above software swap surgery and re-boot was performed late Monday night,  all of this "background always-running" software was installed and running completely and normally and as expected. MBAM + MSE, Team Viewer, and Dyn Updater (along with Aida64, DUMeter, Clockwise, and PERFMON).  Unfortunately I didn't have enough free space on my C-partition to take MEMORY.DMP so sadly we lost that first freeze event without anything diagnostic to look at.

(2) After re-booting from the freeze and re-configuring things on my C-partition to now have 70GB of free space I was able to both (a) get another spontaneous freeze after about 5 hours, as well as (b) produce a complete MEMORY.DMP for analysis.  This happened late on Tuesday night, and that is the MEMORY.DMP you are working on.

I point out that just as with the previous freeze, the additional background programs of Team Viewer and Dyn Updater (and all the others named) were also running. So in the MEMORY.DMP you are studying, Team Viewer and Dyn Updater should be visible. When these two programs are active they are each represented in TASKMGR by at least their own always-running background service (with PID):

(a) Dyn Updater always-running background service

(b) Team Viewer always-running background service

Furthermore, each program also has a system tray object (with PID) normally auto-started and associated with communicating to the program via right-click and popup menu, that can optionally also be ended (from the popup menu) but doesn't influence the continued always-running operation of the background task:

(a) Dyn Updater notification area (system tray) object

(b) Team Viewer notification area (system tray) object

(c) associated with the system tray icon, Team Viewer also has two additional (PID) services that run "hidden" when the tray icon is open

(3) When I re-booted after the Tuesday evening freeze from which I produced MEMORY.DMP and sent it to you, for unimportant reasons I "ended" the optional system tray icons for both Team Viewer as well as Dyn Updater (through their right-click popup menu, selecting "exit"). That means ever since Tuesday evening only the "always-running" background service tasks for these two products have been active.  Neither of the optional system tray icons and the associated PID and foreground programs have been running.

I mention this because quite surprisingly for the past almost 2 days now, ASUS has not frozen again since its last freeze.  While not unheard of, it is unusual... and I might add a bit unexpected to go so long without a freeze on this machine.

I really didn't think ending the system tray icons for these two programs would make any difference, since their corresponding background services were still active anyway. It's one thing to have completely uninstalled both of these two other products when I simultaneously migrated to Bitdefender from MBAM + MSE, but given that I've now reinstalled them Monday night I honestly thought it would be the re-existence of them both (even if only in the background) which would be important, and not perhaps more related to their optional foreground system tray icons.

Anyway I bring this up because of the unexpected almost 2-day freeze-free behavior of ASUS, and yet MBAM + MSE is back, and Team Viewer is back, and Dyn Updater is back.  I had two freezes the first day after all this software was reinstalled, but not no freeze for almost 2 days.  What might be relevant here? Well, the foreground system tray icons for both Team Viewer and Dyn Updater were both "stopped" right after re-boot, and have remained stopped for these almost 2 days.  And no freeze, although MBAM is present.

You'd think if MBAM by itself was culpable, that it shouldn't matter whether other products (or even all features of other products) are also "active".  But, maybe it's actually just that... an unusual combination of factors which is really responsible here. I really don't know if it's actually just all random and a matter of time before ASUS will freeze again with just MBAM active and the two tray icons for Team Viewer and Dyn Updater gone (while their always-running background services still run), or if only the simultaneous presence of one or both of these other system tray programs along with MBAM is really crucial.  Or, in fact, could it actually be just the presence of one or both of these other system tray programs that is actually the culprit, and MBAM is actually blame-free?  Hard to know, but at the moment the 2-day freeze-free operation of MBAM-only is an interesting observation.

Anyway, I've turned just Team Viewer tray icon back on, leaving Dyn Updater tray turned off.  So that's one of the two 3rd-party products which may or may not be freeze-related, which has been partially turned off for the past 2 freeze-free days. I've now brought it back up, so that both MBAM + MSE along with Team Viewer (both background service and tray icon) are now active, along with Dyn Updater background service only.  Again, MBAM has been running "by itself" (along with the two always-running background services of the other two products) for the past 2 days, and no freeze.

Let's see if I can induce a freeze by bringing back just Team Viewer foreground system tray icon.

[LiquidTension]

Hello,

Thanks for your patience.

Analysis of the dump explicitly implicates mrcbt.sys; a driver file belonging to Macrium Reflect. This is consistent with our internal testing of the freeze, along with reports from other users experiencing the same issue. In your case, the freeze is caused by Macrium Reflect's (specifically, the mrcbt.sys driver) attempt to acquire a lock on a registry resource which Malwarebytes has already called the RegUnloadKey function on.

The mrcbt.sys driver is associated with Macrium Reflect's "Changed Block Tracking" feature (Other Tasks -> Edit Defaults -> Advanced -> Advanced Incrementals). We've found the following steps mitigate the freeze whilst still allowing full image backups to be performed.

• Open Programs and Features.
• Right-click Macrium Reflect and click Change.
• Click Next followed by Modify.
• Uncheck Install CBT and click Next.
• Click Install.
• Reboot the computer <- Important!
   

We're still investigating whether there is anything we can do on our side to help mitigate this issue. However, do keep in mind that it is the Changed Block Tracking feature in Macrium Reflect that is triggering the freeze.

[DSperber]

Fascinating discovery!  Remarkable insight. Note that I am STILL running Macrium Reflect on both ASUS (which we are using for the freeze research and now has MBAM + MSE reinstalled, along with reinstalling Team Viewer and Dyn Updater) as well as on M910t (which is still running with Bitdefender instead of MBAM + MSE, and where Team Viewer and Dyn Updater have been uninstalled as well). So on ASUS where MBAM is back, it is exactly the same Macrium Reflect and its CBT functionality which is still running unchanged, but which doesn't seem to be problematic to Bitdefender (when it was being used), but rather only problematic to MBAM.

In contrast, on M910t where Bitdefender and Macrium Reflect are still running, and MBAM is gone, the system has been up and running problem-free freeze-free now for the past 16 days! So again, if Macrium Reflect and CBR is involved, however it impacts MBAM is seemingly much different than how it impacts (or doesn't impact) Bitdefender. Interesting to note.

 

If you don't mind I will pass on your findings to the Macrium Reflect Tech Support group, to see what they have to say about it. Perhaps they can contribute something of value, or something additional that you might pursue examining or looking for in that dump.

For your own information, you can look at the "Release Notes" for Macrium Reflect version history to see when things changed significantly in that product. Note that the last major version upgrade was the release of version 7.2 back on October 30, 2018. I know my freeze instability began sometime after July 2018 but before November 2018, so I'm not sure whatever in Macrium Reflect might be involved is officially new in 7.2. The Release Notes contains imbedded links to additional documents providing more in-depth details about changes when appropriate.

Also, you can look at the "What's New in 7.2" for Macrium Reflect, to see specifically what happened starting in November.

Note that CBT technology first came out with version 7.0 of Macrium Reflect which was all the way back in Feb 2017, so it's been around quite a while and hasn't been problematic (for me, anyway, with all the other software products I use including MBAM) until sometime after July 2018. That must have been around the time of some other signficiant change(s) in other software product(s) that I use, such as what you're now uncovering regarding an incompatibility between how MBAM does something vs. how Macrium Reflect CBT expects "things" to behave in a generic Windows environment regardless of other software products.  I don't know, but let's see what the Macrium Reflect people have to say.

 

Finally, I mention that (in another coincidence) on Wednesday afternoon I happened to install the latest version update for Macrium Reflect, to 7.2.4156. You can read what's in it in the "Release Notes" I pointed to earlier.

I was supposed to re-boot to complete the update (perhaps because of the change to MRCBT.SYS), but I put off re-booting because I wanted to see if I would get another freeze (that I could blame on MBAM being reinstalled). Unfortunately I STILL haven't seen another freeze since the Tuesday evening event which corresponds to the MEMORY.DMP you're looking at. As I posted earlier, the unexpected freeze-free behavior I attributed possibly to the fact that I had also turned off the system tray programs for Team Viewer and Dyn Updater which I had unconsciously done on Tuesday evening right after the re-boot following that freeze.

Well, even though yesterday I turned Team Viewer tray back on, I still haven't got a freeze.  So its' now been 2 days 10 hours since I re-booted, and still no new freeze. Perhaps it's the fact that I didn't re-activate Dyn Updater tray, or maybe the update to Macrium Reflect on Wednesday interfered somehow with MRCBT.SYS and how it was involved with Tuesday night's freeze, or who knows. I certainly didn't re-boot as requested.

So, now I will just go ahead and re-boot. This will start everything fresh... MBAM + MSE, Team Viewer background + tray, Dyn Updater backgroun + tray, latest Macrium Reflect 7.2.4156 along with latest MRCBT.SYS, Aida64, Clockwise, DUMeter, and PERFMON.

Let's see if this restart can coax another freeze... or not. If not, what would we say... that something in this latest 7.2.4156 update to Macrium Reflect made some significant change to MRCBT.SYS that "cured" the conflict with MBAM?  Or might is still just be a random confluence of timing and system-wide events, so that a freeze will still at some point in the future still occur unpredictably?  Again I point out that the timing of the freeze is really unpredictable and totally random, from a "busy system" with lots of open programs, do an "idle" system with absolutely nothing running other than the background processes. No connection to running an MBAM scan or Macrium Reflect backup, or anything understandable.

Completely unpredictable as to exactly what conditions might trigger the freeze since you'd think whatever actually happened on Tuesday evening that you see inside MEMORY DMP must surely have occurred dozens or hundreds or thousands of times previously without a conflict between MR and MBAM. What peculiar timing or event sequence made this particular situation occur that managed to avoid it all those other previous times?  Just nano-seconds timing overlap that turned out to be crucial? Would a few more milliseconds more or less between what MBAM did and what MR then tried to do, would that have avoided the freeze condition?

 

Anyway, I feel that Macrium Reflect in UK should be apprised of what you've found.  Can't hurt, and can only help, I'm sure.

I will now re-boot and we'll start again, waiting again for the next freeze to occur... if it will. Do we want it to freeze again, or not? If it does I will again send you the new MEMORY.DMP. Remember, if this occurs you will now be looking at the behavior of the latest 7.2.4156 version of MR and MRCBT.SYS, not the previous 7.2.4063 which is in your current MEMORY.DMP.

[LiquidTension]

Please do share the findings with Macrium. I would also strongly encourage you to share the MEMORY.dmp file with them as well.

If you encounter another freeze with the latest version of mrcbt.sys installed, we'd be happy to take a look at the dump file.

[DSperber]

1 hour ago, LiquidTension said:

• We're still investigating whether there is anything we can do on our side to help mitigate this issue. However, do keep in mind that it is the Changed Block Tracking feature in Macrium Reflect that is triggering the freeze.

On this point, I'm sure the Macrium Reflect people will insist it must be a combination of what they're doing in MRCBT.SYS and what MBAM is doing which is somehow incompatible.  MRCBT.SYS could be the "victim" rather than the "culprit", triggered into causing the freeze by something unexpected that MBAM has done.

Again I point out that simply by removing MBAM + MSE on M910t, and installing Bitdefender instead, while keeping exactly the same version of Macrium Reflect with MRCBT.SYS unchanged, my other M910t machine has now been up for 16 days straight 24/7 without a freeze. Macrium Reflect untouched.  Only MBAM removed and Bitdefender installed. And no freeze.

So it is reasonable to argue that it is NOT [just] Macrium Reflect CBT technology which is responsible, though it truly might be "involved".

But it is even more reasonable to argue that it is something about CBT and MBAM together which is clearly responsible, which apparently is not the case with CBT and Bitdefender.

Let's see how they respond, and what they might contribue to the problem solving plan. That's what's important here, to FIX THINGS so that it doesn't occur in whichever or both software products deserve fixing.

For the time being I actually prefer keeping CBT active and using Bitdefender as my own "workaround" choice right now.  I need M910t (which is my "production HTPC") to remain up 24/7 and 100% stable. I have now achieved that (after 8 months of struggle) and will stay with Bitdefender there.

If and when MBAM and/or Macrium Reflect develop whatever true fix is required to prevent the underlying freeze symptom from occurring, then I'll have a second "workaround" choice to think about. In the meantime I will my ASUS machine to the "lab research", hoping to be a part of the eventual true solution for MBAM and/or Macrium Reflect.

[LiquidTension]

That's understandable. We appreciate your efforts with assisting the troubleshooting of the issue.

Please note that the freeze has still been observed by some users even after Malwarebytes is no longer installed. With that said, investigation is still on-going and we are actively looking into whether a change on our side can help alleviate this issue for users with Macrium Reflect and Malwarebytes installed.

[DSperber]

1 hour ago, LiquidTension said:

Please note that the freeze has still been observed by some users even after Malwarebytes is no longer installed. With that said, investigation is still on-going and we are actively looking into whether a change on our side can help alleviate this issue for users with Macrium Reflect and Malwarebytes installed.

This is mystifying to me, if the direct cause of the freeze is that MR is attempting a LOCK on a Registry resource that MBAM has previously done its own RegUnloadKey (UNLOCK?) on.  If MBAM has been uninstalled, then the crucially necessary UNLOCK (by MBAM) which precedes the UNLOCK (by MR) won't get done, at least not by MBAM.  So if the freeze occurrence which is shown in my Tuesday MEMORY.DMP depends on a LOCK attempted after the resource has previously been UNLOCK'ed, but that previous UNLOCK is no longer being performed (because MBAM has been uninstalled), that what is it that triggers the freeze in systems where MR is running by itself and there is no MBAM present at all?  Some other software product?  Some other place in Macrium Reflect? Could it be a similar but different freeze trigger sequence when MBAM is not in the environment?  I'd not been aware of freezes occurring in systems that only had Macrium Reflect, and the Release Notes don't actually talk about fixed problems using that word.

Also, How is it possible that when I run Bitdefender instead of MBAM + MSE (on M910t) that I've now run for over 16 days without a freeze... with the exact same MR and its MRCBT.SYS installed? This is the situation where MBAM has been uninstalled, and it's only Macrium Reflect present (along with Bitdefender, which of course is not MBAM).  This kind of points to the RegUnloadKey done by MBAM which is the "culprit" (since it clearly is not also being done by Bitdefender), not the subsequent LOCK done by Macrium Reflect.

And again, I never had a freeze before sometime after July 2018.  So software updates from all vendors must be reviewed, as another source of possible clues.

Perplexing. I've now opened a ticket with Macrium Reflect support, so let's see what they have to say, or what they'd like me to do for them next. I've now re-booted (as per their update to 7.2.4156 requests) so everything is once again back running normally on ASUS.  This includes the tray icons active for both Team Viewer as well as Dyn Updater, and is exactly as it was back on Tuesday night when I uninstalled Bitdefender and reinstalled MBAM and got two freezes within 12 hours (and none since Tuesday night!).

[DSperber]

2 hours ago, LiquidTension said:

Please do share the findings with Macrium. I would also strongly encourage you to share the MEMORY.dmp file with them as well.

If you encounter another freeze with the latest version of mrcbt.sys installed, we'd be happy to take a look at the dump file.

Yes, they've now gotten back to me on the ticket and requested the Tuesday MEMORY.DMP for themselves, since that's the only one at the moment that I have. I'm still waiting for the next freeze to hopefully occur today, in which case I'll send that new one (with the just released 7.2.4156 of Macrium Reflect and MRCBT.SYS now running), if it ever happens.

In the meantime, I sent you a PM which I hope you saw, explaining that I'd foolishly deleted my own copy of MEMORY.DMP in both its unzipped original and zipped compressed form, both locally and on my file sharing site. If I could request that you send back to me (on your own hosting site, providing the URL) that very same compressed 2GB version of the 34GB MEMORY.DMP so that I could send it to Macrium, I would very much  appreciate it.  I'm sure they're very much interested to see it for themselves.

Thanks. And we'll see if/when the next freeze occurs on my ASUS machine.  All set, and things look exactly as they did back on Tuesday night except for the updated version of Macrium Reflect now running.

[DSperber]

23 hours ago, LiquidTension said:

Please do share the findings with Macrium. I would also strongly encourage you to share the MEMORY.dmp file with them as well.

If you encounter another freeze with the latest version of mrcbt.sys installed, we'd be happy to take a look at the dump file.

Still no re-freeze symptom occurring again on ASUS, not since Tuesday night (when Macrium Reflect 7.2.4063 was installed), and not since Wednesday afternoon after applying the Macrium Reflect update to 7.2.4156, and also not since re-booting Friday morning (I had deferred doing that as requested by the Macrium Reflect update, but decided to go ahead with it yesterday).

So I'm once again running as I was on Tuesday night, including also with Team Viewer and Dyn Updater active.  Perhaps I just haven't lucked-in to set up whatever is the particular set of circumstances conducive to triggering a freeze... but maybe it will still come.  Really just unpredictable and random.  If it could be reproduced on -demand we would already know what was causing it.  Clearly there are some "surrounding circumstances" or other conditions or things that have run or are running, etc., all of which work together to make the 'soup" that is required only then for the freeze to maybe possibly appear.

Anyway, no new freeze or MEMORY.DMP yet to report.

 

However, I have now gotten a personal response back from Jamie (from Macrium) who is taking over the ticket.  He is one of the key developers on the CBT driver component team and is the one who will be looking into the Tuesday 4/2 MEMORY.DMP I sent to them, and which you looked at as well.

He has asked me to ask you a few questions. I will act as the relay, and communicate your response back to him. Whereas my own personal questions are from the perspective of a "user", his questions are more technical. And I'm sure he's looking for truly technical answers. So please do answer them in terms as technical as you'd like, using whatever terminlogy or specifics or attachments or screenshots or whatever that you'd like, and I will communicate all of it to Jamie at Macrium.

Thanks.  Here is what he asks:

---------------------------------------------------------------------------------------------------------------------

Thank you for taking the time to document your issue in such detail and for providing the associated files that will help us investigate. I am one of the developers who work on the Change Block Tracker (CBT) component and I will be taking over your case and performing analysis on the crash dump report in the coming days.

My initial reaction to your post is one of confusion as while our CBT driver does read some values from the current “SYSTEM” Registry hive, it makes no attempt to acquire a lock on them. Furthermore, it is not possible to unload the current SYSTEM hive as it is used by Windows itself.

Working on the assumption that something has been lost in translation between their developers and their customer support, would it be possible for you to ask the nice people at Malwarebytes to provide the exact path of the Registry key that they believe our CBT driver is referencing and also to provide some further details on what they have acquired from the minidump that leads them to believe that MRCBT is holding a lock on the Registry?
 

Kind regards,

 

Macrium Support

[DSperber]

Update: after re-booting on Friday 4/5 at 9:15AM I have been waiting almost two days for another freeze to occur, with MBAM and MR installed as well as Team Viewer and Dyn Updater fully active. The Tuesday 4/2 dump I originally sent you included MR 7.2.4063 which was the installed version at the time. On Wednesday 4/3 I updated MR to the new 7.2.4156 which had just been released, but put off the requested re-boot awaiting the next freeze which I hoped wouldn't be too long in arriving.

 

Well, it took one day and 19 hours of waiting, but the freeze.finally occurred early this morning 4/7 at 04:18AM (while I was sleeping and the machine was completely "idle" except for the normally operating background programs, HTPC/WMC components, etc. I discovered the freeze this morning at around 9AM and forced a BSOD and complete MEMORY.DMP which once again is about 34GB as it was back on Tuesday.

 

I have zipped MEMORY.DMP for upload to you, but for some reason the compressed size of today's dump is about 16GB, rather than the much smaller 2GB compressed size of Tuesday's dump. I can only assume that there was much more of my 32GB physical memory used at one time or another over the past 1 day 19 hours, which all together simply didn't compress as well as Tuesday's dump which probably had a lot more unused "zero'd" memory and thus compressed much better.  Again, the underlying raw MEMORY.DMP itself was 34GB in both cases.

 

So, unfortunately, at my personal broadband ISP service upload speed it's going to take 2-3 hours for me to upload this second 16GB ZIP'd file. I'm sending it the Macrium Reflect FTP site they provided to me for upload, and I will PM you with the URL for that dump when when the upload completes so that you can download it directly from their FTP site.  But it's going to be a few more hours before it's completed and available Shouldn't be a problem for you, because today is Sunday.  It'll be ready for you by Monday morning.

 

I'm also attaching here a small photo of what my screen looked like when I found it. This is to give some context to what my own freeze symptom looks like.

 

Note that the "time" and "up time" values presented in Aida64 have "frozen" as of 4:18:27AM and 1d 19:04:41, respectively.  Aida64 updates its values every 4 seconds, so screen updating to this window stopped back at 4:18:27AM.

 

In the meantime, the Clockwise window (just off lower-left corner of the Aida64 window) has continued to update its own system clock value, which shows 9:02:02AM when I discovered the freeze. Also note that the Windows system clock in the lower-right corner of the screen also shows 9:02AM.  So both of these on-screen clock values are still "running and active" and obviously not frozen. Also I can move the mouse around and the onscreen cursor moves around, but right-click and left-cick on anything doesn't work to produce the expected result.

 

Also, note that the PERFMON window in the lower-right is also continuing to progress left-to-right updated every second, and with properly updated clock values on the X-axis. However the blue "graph" output is extremely "ragged" and irregular, and shows many white spaces between the blue output. This suggests to me that the system was unable to write to the screen during these interspersed blank periods, while then starting again displaying blue, until the next white area.  So this output is not completely frozen, but does certainly appear impacted from its normal behavior.

 

I didn't photo the upper-right portion of the screen, but that's where the DUMeter window appears, and it is completely frozen onscreen as well (just like Aida64) and is no longer progressing at all, which is normally right-to-left updated every second. I also can't communicate to the ASUS machine (from operating M910t) using either RealVNC or Team Viewer.

 

The above comments are simply meant to provide some context to what I mean when I say "freeze".  Note that this is somewhat different on Z170 than it was on M910t (before I uninstalled MBAM + MSE and installed Bitdefender to replace them, and the freezes finally stopped entirely). On M910t I see a "total freeze" of PERFMON, whereas it seems to half-operate on Z170. Also, on M910t the mouse cursor is frozen and can't be moved around the screen as it still can on Z170 (even if mouse-clicks have no effect).

 

Anyway, still hours to go on the dump upload. Sorry for this (and for how long it's going to take you to download it for study), but it turned out to be 16GB compressed.  Same 34GB uncompressed.

[DSperber]

Update again... freeze again, this time not during "idle" but during an actual "busy" period during which I was composing an email and typing away madly. This second freeze occurred at 11:58AM, which is just about 2 1/2 hours after this morning's re-boot at around 9:15AM (following the 4:58AM first freeze).

While the upload of the 16GB compressed memory dump to the hosting site was proceeding in the background at 10 Mb/s (using WinSCP as the file transfer program), I was just doing normal work.  I had Firefox open with two tabs in the background, and I was working in my email client typing an email in the foreground.  Not overly busy, but definitely not "idle".

I got to a point where I needed to backspace in the email I was composing, and suddenly it didn't respond.  I was able to use the mouse click just once to be able to see the Aida64 window where sure enough less than two minutes earlier the "time" value had frozen. In other words I was still able to work within my email composition window for those two minutes, but eventually the effects of the freeze caught up with what I was doing with keyboard and mouse, and that was it.

So now I will attempt to upload BOTH of these freezes from today. I will provide URL information when I get hopefully them uploaded. The second one compresses to 10GB, so I'll try doing that one first.  I'll then address the first one which was 16GB compressed.

 

[DSperber]

Ok. Both ZIP dumps have now been uploaded successfully.

I've sent a PM with URL instructions for download of the two ZIP files.

[DSperber]

I've now heard back from Macrium Reflect support, who've have a chance to look closely into the three dumps I provided to them over the past few days.

From their response it sounds like we may have made FANTASTIC PROGRESS on discovering the actual cause of the freeze, and fixing it.  Perhaps the instant-fix is just a temporary workaround until the permanent more elegant complete fix is available, but it does sound like real insight into what is happening is now available.

Here is what they said to me on my MR ticket (and, by the way, THIS IS WHY I GENERALLY LICENSE THE NON-FREE VERSION OF A PRODUCT... to support the vendor and its product and its development team, and to guarantee product support like this in case of a real issue).

--------------------------------------------------------------------------------------------------------------------------------

Hi Darryl,

 

After performing analysis on the crash reports that you sent, it appears that your system has processed a rather complex series of events that has lead to a race-condition between the Malwarebytes product and our MRCBT.SYS driver during a file-system TRIM operation.

We are still performing analysis, but in the mean time, we have implemented a workaround for this situation and I am pleased to provide links (below) to an updated driver that should stop your computer(s) from freezing.

Please can you download the relevant driver and replace the existing driver in your C:\Windows\system32\drivers folder, then restart Windows for the changes to take effect.

I understand that this will take some time for you to report any findings, but please let us know the outcome at your convenience.

32bit - ...
64bit - ...

Kind regards,

 

Macrium Support

------------------------------------------------------------------------------------------------------------

I am about to install the temporary freeze-avoiding workaround-version of MRCBT.SYS that they have provided to me, and re-boot.  Then it's a matter of just waiting for time to pass, to see if a freeze returns. If history means anything the particular "fluke" combination of circumstances can occur very quickly, or within a few hours of re-booting, or a few times a day, or maybe not until a day or two has gone by.  But typically no more than two days at best is needed to set up the enivornment so that a freeze by that time is almost certain.

We shall see.

In the meantime, based on @LiquidTension's comment that at least one report exists of a freeze (dump to go with it?) that occurred WITHOUT MBAM PRESENT AT ALL (does that mean it was temporarily uninstalled in order to completely remove it from possibly contributing to the freeze "soup" mix of circumstances?), but only Macrium Reflect. I will pass that information on to MR, so that they are fully aware whatever is the story with MRCBT.SYS and its ultimate fix, that it is also apparently possible to expose their freeze-causing defect having nothing to do at all with MBAM!

In otherwords, while MBAM being present might exacerbate things, it's not required.  The underlying problem with MRCBT.SYS may truly be in the design of MRCBT.SYS by itself, where the true fix must therefore also be. I will pass along this notion on my reply to my ongoing MR open ticket conversation, just to be sure they are aware of the fact that MBAM has seen this MR-only freeze. Hopefully that particular MR-only freeze dump (in the hands of MBAM support) is still available, and I can get that URL in order to pass on that dump to the MR people for them to examine as well.

Anyway, looks like we're really making terrific progress here.

[DSperber]

One more comment...

The subject title for this thread describes TWO issues I have been fighting with.  One is the freeze, and the second is the error 36887 errors occurring on the System Event Log whenever a SCAN is performed by MBAM... but only on this particular machine of mine. I don't see that error on my other machines, despite the fact that I run MBAM on ALL machines.

"I have been studying the Event Log closely, and have observed that Malwarebytes Premium daily scan at 6:30AM seems to ALWAYS produce two "error" entries in the log for event ID 36887, with "the following fatal alert was received 70" typically reported first followed by "the following fatal alert was received 40" reported second. I can reproduce this pair of errors "on demand" simply by running the scan manually specifically when the "scan file system" item is reached, so I'm 100% certain these two errors are coming from MBAM."

So it still is appropriate to try and understand what the story is that is responsible for these Event Log error messages produced when the "scan file system" item is reached during the nightly SCAN.

That's why I found it interesting that the MR technician referenced those same words "file system" in his summary statement:

"After performing analysis on the crash reports that you sent, it appears that your system has processed a rather complex series of events that has lead to a race-condition between the Malwarebytes product and our MRCBT.SYS driver during a file-system TRIM operation."

So in passing, I wonder if there's any connection here, in possibly explaining not only the freeze story but also why this particular machine also produces 36887 errors regularly? Of course the contradicting information is that my other M910t machine, also running nightly SCAN from MBAM, does NOT generate 36887 errors, and yet it was susceptible to freeze as well when running MBAM?

 

Also, one must not lose sight of the fact that I had "solved" the freeze problem on M910t about three weeks ago, by uninstalling MBAM + MSE, and installing Bitdefender. So if the underlying problem was a defect solely in MRCBT.SYS completely unrelated to MBAM, and I hadn't touched Macrium Reflect three weeks ago, how is it that the freeze disappeared on M910t... even with the so-called "defective MRCBT.SYS"? Obviously using Bitdefender instead of MBAM injected some kind of "fix", as I've now run freeze-free for almost 3 weeks.

I will also pass this info on to the Macrium Reflect tech staff, as another relevant clue.

[DSperber]

I've now received a more detailed explanation of the analysis performed by the Macrium Reflect CBT team, on the freeze dumps I provided.

--------------------------------------------------------------------------------------------------------------------------------

Firstly, taking everything into account, I can see no flaw in what you have been advised by Malwarebytes, however; we have never seen this issue arise in any of our testing, nor has it been reported to us by any other customer. As such, we can only go off the information in the dump files you sent.

 

Basically, the state of your system at the point the dump file was generated, show that the Malwarebytes product had asked Windows to Unload a Registry hive (database of Registry key/value pairs). This in turn had caused Windows to delete (or move) a section of the Registry file, which in turn had generated a TRIM operation on a volume that CBT was tracking.

 

At this point, CBT needed to access some settings that are normally stored in memory (cached), however; for some reason (I can only guess that it needed the resources), Windows had elected to discard this memory. At this point, CBT tried to re-read the settings back from the Registry, but it can't because Windows is holding a global lock on the Registry in response to the request from the Malwarebytes product.

Malwarebytes and our CBT driver are now waiting for each other to complete before either of them can move forward. 

Normally, this would be detrimental to both products, but would not cause the system to freeze, however; a further 45 requests to access the Registry (from various processes) come in and all of these now block, waiting for the global lock on the Registry to be released. I presume that one or more of these is a critical Windows component and this is what results in the freeze.

 

Our solution to this, was to read these particular settings at driver start-up, then store them in memory that Windows cannot discard. This removes all possibility of CBT needing to access the Registry during a TRIM operation.

Without access to the Malwarebytes product source code, I am not entirely sure what their product is doing, but I do not believe that they are doing anything wrong per se, and while it is a rare occurrence - I believe that our CBT driver should be able to cope with these circumstances.

 

I have attached the output of the analysis that we performed on the dump file. It is intended for software developers, but it is quite easy to see which other processes were also waiting to access the Registry and why.

 

---------------------------------------------------------------------------------------------------------------------------------------------

 

This and at least 2 of the other dumps that I looked at, including the 16GB file that you uploaded to our FTP site, indicate the same sequence of events - they all show that CBT is stuck waiting on a global Registry lock that the Malwarebytes product holds.

Please note that this locking/waiting mechanism is not something that either MBAM or CBT have directly asked the system to do, it is a side-effect of the implementation of the Windows Registry and as such, they can change from one Windows version to the next.

To be honest, I haven’t really done much investigation into how your computer got into this state. After confirming that CBT was indeed dead-locked, waiting on the global Registry lock resulting from the Malwarebytes action, my sole focus was on implementing a solution that would allow CBT to run freely under the same set of circumstances.

As I stated previously, I didn’t discover anything that contradicts the information that Malwarebytes have given you, but I am afraid that I do not have an answer to what is unique about the combination of our software and your computer(s) that triggered this.

I hope that this goes some way to explaining our findings.

Kind regards,

 

Macrium Support

 

-----------------------------------------------------------------------------------------------------------------

 

[LiquidTension]

Hi @DSperber,

Thanks very much for keeping us up-to-date with the contact you've had with Macrium. Our analysis of the dumps you recently provided show the same set of conditions and deadlock cause.

Please do let us know how you get on with the updated version of mrcbt.sys that Macrium provided you.
 

Quote

So in passing, I wonder if there's any connection here, in possibly explaining not only the freeze story but also why this particular machine also produces 36887 errors regularly?

The Malwarebytes service logging indicates a correlation between the Self-Protection module and those events being logged to the Windows System Event Log. If you find they still persist, we can certainly continue looking into this. I don't believe they are associated with the freeze/deadlock, but it would certainly be worth verifying if they still persist after the deadlock no longer occurs.
 

Quote

So if the underlying problem was a defect solely in MRCBT.SYS completely unrelated to MBAM, and I hadn't touched Macrium Reflect three weeks ago, how is it that the freeze disappeared on M910t...

Your deadlock occurs between a RegUnloadKey operation performed by MBAMService (the main Malwarebytes service) and a RtlQueryRegistryValues operation performed by mrcbt.sys (in which it tries to acquire a CmpRegistryLock resource). Taking Malwarebytes out of the equation changes the scenario.

[DSperber]

I asked one more question of Macrium:

-----------------------------------------------------------------------------------------------------

Final question...

 

Even though you've devised a workaround that allows CBT itself to operate freely despite the Registry lock resulting from the MBAM action, what about all the other 45 queued requests that build up from other tasks, all due to the same Registry lock?  Is not this whole never-freed backlog what results in the eventual total freeze?

 

Would they not still queue up, eventually again exhausting the size of the queue and resulting in the freeze even though CBT itself isn't any longer caught in this queue because of your code change to CBT?

 

Or, is everything finally released when the TRIM operation completes?

 

Or, if not the end of the TRIM operation, what is it that would eventually unlock things allowing all of the queued up requests on Registry finally proceed and the queue empty out... so that no fatal freeze occurs?

 

-----------------------------------------------------------------------------------------------------------------

 

To which they replied:

 

With the workaround in place, Windows will complete the TRIM operation and allow Malwarebytes to release the lock. Once this happens, the other processes will no longer need to wait for Malwarebytes and everything should run smoothly.

 

My only concern is that CBT and/or Malwarebytes could have been the first point of failure in a sequence of events that was destined to end badly anyway. For example, if another process is leaking resources and this is what prompted Windows to discard the memory containing the CBT settings, then a different failure could arise further down the line.

 

This will only become apparent through testing, but should you run into problems, please let us know and we will do our best to help.

 

Kind regards,

 

Macrium Support

------------------------------------------------------------------------------------------------------------

But the good news is that I've now reached the 24-hour still freeze-free point on my ASUS machine on which the revised test version of MRCBT.SYS was installed.

Fingers crossed we're finally out of the woods on this one.

[exile360]

Man, I sure hope so.  This has been a deep dive for sure and we are very grateful to you for sticking it out and doing such extensive investigation, documentation and testing/verification to isolate and (hopefully; fingers crossed here as well ) resolve this issue once and for all.