MBEP on Server 2012R2 causing crash/reboot every ~25min since latest update

[HCHTech]

Bugcheck is 0x00000133 DPC_Watchdog Violation

Analysis of the minidump shows faulting module is mwac.sys.   Dump details posted below.  This appears to be similar to this post from March of 2018 in the consumer forums.

Problem stopped when I uninstalled MBEP, started again when I did a reinstall using a fresh download from the Nebula console.   This particular machine has been running MBEP w/o issue for 2 years now, and no configurations were changed.  It is a Hyper-V Host.

Solution from the linked post was to disable web protection (makes sense - mwac.sys is the web protection service), but I thought I should post first before trying that, plus I'd rather not "test" on my client's production hardware.

Anything else I should be doing?

 

 

=======MiniDump=======

DPC_WATCHDOG_VIOLATION (133)
The DPC watchdog detected a prolonged run time at an IRQL of DISPATCH_LEVEL
or above.
Arguments:
Arg1: 0000000000000000, A single DPC or ISR exceeded its time allotment. The offending
    component can usually be identified with a stack trace.
Arg2: 0000000000000501, The DPC time count (in ticks).
Arg3: 0000000000000500, The DPC time allotment (in ticks).
Arg4: 0000000000000000, cast to nt!DPC_WATCHDOG_GLOBAL_TRIAGE_BLOCK, which contains
    additional information regarding this single DPC timeout

Debugging Details:
------------------

*** WARNING: Unable to verify timestamp for mwac.sys
fffff801a48abe58: Unable to get Flags value from nt!KdVersionBlock
GetUlongPtrFromAddress: unable to read from fffff801a4968308

KEY_VALUES_STRING: 1

    Key  : Analysis.CPU.Sec
    Value: 1

    Key  : Analysis.DebugAnalysisProvider.CPP
    Value: Create: 8007007e on HCHMARK2019

    Key  : Analysis.DebugData
    Value: CreateObject

    Key  : Analysis.DebugModel
    Value: CreateObject

    Key  : Analysis.Elapsed.Sec
    Value: 2

    Key  : Analysis.Memory.CommitPeak.Mb
    Value: 73

    Key  : Analysis.System
    Value: CreateObject

BUGCHECK_CODE:  133

BUGCHECK_P1: 0

BUGCHECK_P2: 501

BUGCHECK_P3: 500

BUGCHECK_P4: 0

DPC_TIMEOUT_TYPE:  SINGLE_DPC_TIMEOUT_EXCEEDED

CUSTOMER_CRASH_COUNT:  1

PROCESS_NAME:  System

STACK_TEXT:  
ffffd000`5a9a2c88 fffff801`a477386a : 00000000`00000133 00000000`00000000 00000000`00000501 00000000`00000500 : nt!KeBugCheckEx
ffffd000`5a9a2c90 fffff801`a4647fd1 : 0000065d`b4ea0572 00000000`0002ef21 00000000`0000000b fffff801`a475ae77 : nt! ?? ::FNODOBFM::`string'+0x563a
ffffd000`5a9a2d20 fffff801`a4d9dac5 : ffffe000`88026900 ffffd000`5a8ef000 ffffe801`5c439880 ffffd000`5a8db180 : nt!KeClockInterruptNotify+0x91
ffffd000`5a9a2f40 fffff801`a46cd943 : ffff8488`f7390aba 00000000`00000000 ffffe000`8953ddb0 ffffe000`8953ddb0 : hal!HalpTimerClockIpiRoutine+0x15
ffffd000`5a9a2f70 fffff801`a475a9ca : ffffe000`880e3c30 ffffd000`59dd9970 ffffe000`b370c030 ffffe000`8953ddb0 : nt!KiCallInterruptServiceRoutine+0xa3
ffffd000`5a9a2fb0 fffff801`a475ae77 : 00000000`00000000 ffffe000`b370c030 ffffe000`00000000 00001f80`00d3027e : nt!KiInterruptSubDispatchNoLockNoEtw+0xea
ffffd000`59dd9790 fffff801`a469c077 : ffffe000`b370c030 00000000`00010008 00000000`00000000 ffffd000`59dda3a0 : nt!KiInterruptDispatchNoLockNoEtw+0x37
ffffd000`59dd9920 fffff801`fed486f6 : ffffe000`aeadc010 00000000`00000001 ffffd000`59dd99f8 fffff801`00000000 : nt!KxWaitForLockOwnerShip+0x2b
ffffd000`59dd9950 ffffe000`aeadc010 : 00000000`00000001 ffffd000`59dd99f8 fffff801`00000000 ffffd000`59dcb970 : mwac+0x136f6
ffffd000`59dd9958 00000000`00000001 : ffffd000`59dd99f8 fffff801`00000000 ffffd000`59dcb970 fffff801`fed51bd1 : 0xffffe000`aeadc010
ffffd000`59dd9960 ffffd000`59dd99f8 : fffff801`00000000 ffffd000`59dcb970 fffff801`fed51bd1 ffffd000`59dd9b02 : 0x1
ffffd000`59dd9968 fffff801`00000000 : ffffd000`59dcb970 fffff801`fed51bd1 ffffd000`59dd9b02 fffff801`00000000 : 0xffffd000`59dd99f8
ffffd000`59dd9970 ffffd000`59dcb970 : fffff801`fed51bd1 ffffd000`59dd9b02 fffff801`00000000 00000000`00000000 : 0xfffff801`00000000
ffffd000`59dd9978 fffff801`fed51bd1 : ffffd000`59dd9b02 fffff801`00000000 00000000`00000000 fffff801`fed3bf65 : 0xffffd000`59dcb970
ffffd000`59dd9980 ffffd000`59dd9b02 : fffff801`00000000 00000000`00000000 fffff801`fed3bf65 00000000`00000001 : mwac+0x1cbd1
ffffd000`59dd9988 fffff801`00000000 : 00000000`00000000 fffff801`fed3bf65 00000000`00000001 fffff801`fed40c23 : 0xffffd000`59dd9b02
ffffd000`59dd9990 00000000`00000000 : fffff801`fed3bf65 00000000`00000001 fffff801`fed40c23 ffffe000`8953ddb0 : 0xfffff801`00000000

SYMBOL_NAME:  mwac+136f6

MODULE_NAME: mwac

IMAGE_NAME:  mwac.sys

STACK_COMMAND:  .thread ; .cxr ; kb

BUCKET_ID_FUNC_OFFSET:  136f6

FAILURE_BUCKET_ID:  0x133_DPC_mwac!unknown_function

OS_VERSION:  8.1.9600.19761

BUILDLAB_STR:  winblue_ltsb

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 8.1

FAILURE_ID_HASH:  {a74fd326-3c20-c188-394a-5bdcf9fda410}
 

=====================

Edited October 26, 2020 by HCHTech

[exile360]

Greetings,

It would be a good idea to check and see if any third party WFP filters/drivers are installed on the system, as it may be a matter of a conflict with the WFP driver/filter used by the Web Protection component in Malwarebytes causing the issue.  Details about the issue can be found in this support article, including a list of some known conflicting applications, however that list is not comprehensive and we've already seen additional apps which conflict with WFP filter used by Malwarebytes' Web Protection.

If you aren't sure how to check installed WFP filters, the official Malwarebytes Support Tool shows them in its mbst-check-results.txt file that it creates:

1. Download and run the Malwarebytes Support Tool
2. Accept the EULA and click Advanced tab on the left (not Start Repair)
3. Click the Gather Logs button and once it completes you can open the resulting ZIP file on the desktop to find the log I mentioned above in the root of the ZIP file

You'll find the list of filters under the Registered WFP Filters section near the bottom of the log.  Any labeled as MWAC belong to Malwarebytes and there are likely to be at least a few others which are built in Windows components since WFP is the same platform/APIs used for the built in Windows Firewall/Windows Firewall with Advanced Security.

Please let us know how it goes and what your findings are.

Thanks

Edited October 27, 2020 by exile360

[HCHTech]

I don't see any that look out of place (see paste of the section below), plus any conflicts would have had to have been introduced today, as this is a sudden onset problem and no changes were made to the server.

 

========

Registered WFP Filters
==================================
FWPM_LAYER_ALE_AUTH_CONNECT_V4
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.

FWPM_LAYER_ALE_AUTH_CONNECT_V6
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.

FWPM_LAYER_ALE_AUTH_RECV_ACCEPT_V4
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.

FWPM_LAYER_ALE_AUTH_RECV_ACCEPT_V6
Teredo socket option opt out block filter                            
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.

FWPM_LAYER_ALE_CONNECT_REDIRECT_V4

FWPM_LAYER_ALE_CONNECT_REDIRECT_V6

FWPM_LAYER_ALE_RESOURCE_ASSIGNMENT_V4

FWPM_LAYER_ALE_RESOURCE_ASSIGNMENT_V6

FWPM_LAYER_ALE_RESOURCE_RELEASE_V4

FWPM_LAYER_ALE_RESOURCE_RELEASE_V6

FWPM_LAYER_INBOUND_TRANSPORT_V4
IKEv2 Server Quick mode IPsec tunnel policy (v4)(* to *)             
IKEv2 Server Quick mode IPsec tunnel policy (v4)(* to *)             
VPN Reconnect Filter                                                 VPN Reconnect IPv4 Callout Filter
VPN Reconnect Filter                                                 VPN Reconnect IPv4 Callout Filter
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
L2TP Server Filter1                                                  
L2TP Server Inbound Filter                                           
L2TP Server Inbound Filter                                           

FWPM_LAYER_INBOUND_TRANSPORT_V6
IKEv2 Server Quick mode IPsec tunnel policy (v6)(* to *)             
IKEv2 Server Quick mode IPsec tunnel policy (v6)(* to *)             
VPN Reconnect Filter                                                 VPN Reconnect IPv6 Callout Filter
VPN Reconnect Filter                                                 VPN Reconnect IPv6 Callout Filter
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
L2TP Server Filter1                                                  
L2TP Server Inbound Filter                                           
L2TP Server Inbound Filter                                           

FWPM_LAYER_OUTBOUND_TRANSPORT_V4
IKEv2 Server Quick mode IPsec tunnel policy (v4)(* to *)             
IKEv2 Server Quick mode IPsec tunnel policy (v4)(* to *)             
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
L2TP Server Filter1                                                  
L2TP Server Outbound Filter                                          
L2TP Server Outbound Filter                                          

FWPM_LAYER_OUTBOUND_TRANSPORT_V6
IKEv2 Server Quick mode IPsec tunnel policy (v6)(* to *)             
IKEv2 Server Quick mode IPsec tunnel policy (v6)(* to *)             
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
WFP Built-in IKE Exemption Filter                                    Default exemption filter for IKE traffic.
L2TP Server Filter1                                                  
L2TP Server Outbound Filter                                          
L2TP Server Outbound Filter                                          

-----END OF FILE-----

[exile360]

It may have been due to an update to one of the applications on the system or to MBEP itself.  What VPN are the clients using?  I see some VPN related items listed among the WFP entries.  The most recent builds of Malwarebytes do tend to conflict with many common VPN apps, so that might be the cause.  You can test by disabling or removing the VPN, assuming disabling it removes/disables the WFP filter used by the VPN.

[HCHTech]

They have a Sonicwall and are using NetExtender.   For now, I disabled Web Protection, it's an update to something that introduced the conflict, so it's more important to fix it first and diagnose later.    They also have Solarwinds managed AV, which is a form of BitDefender, which on their bad list, but living quite happily (so far) on a couple of hundred workstations across several clients I have with MBEP.   I'm going to disable Web Protection on the rest of those clients today as a pre-emptive strike.  I don't want this to happen to anyone else.  

 

BTW - shouldn't I be seeing some entries for Malwarebytes in the WFP filters? (the run of the support tool was done before disabling web protection)

Edited October 27, 2020 by HCHTech
Additional data

[exile360]

OK, if they have BitDefender then it is very likely to be a conflict between BitDefender's web filtering component and Malwarebytes' Web Protection, and yes, there should be WFP entries related to MWAC but the logs show none.  This could be due to BitDefender as it may have blocked them from being created or from an issue with Malwarebytes the last time it was installed/updated.

[HCHTech]

So far so good, I disabled WP on all computers on all of my clients with MBEP.   MBEP is the latest resident in my security stack, so they will lose if there is a conflict.  I'm not changing my AV, it's integrated with my RMM, and I'm too invested now to switch horses.

BTW, it seems a bit of a cop out for MB to just point fingers at such a long list of AV products.   A better answer would be for development to figure out how to make them work together.  Or at the very least to auto-disable WP if one of those products is detected.  I appreciate this isn't a trivial task, but their whole market position is as an additional layer of security, so that position falls down a bit (or a lot) when you read the fine print to see "except when a, b, c, d, e, f, g, h, i, j, k or l".   Unquantified now is how much less protected my clients are with WP disabled.   There is no module labeled web-protection or similar in the Solarwinds version of Bit Defender, so I can't tell if it even offers that kind of protection.   I do know that it is a customized version of the enterprise version of BitDefender, NOT the consumer version.

Edited October 28, 2020 by HCHTech

[exile360]

No problem at all, I understand completely, I'm just glad we were able to find the issue and a viable workaround.

To shore things up a bit, you might consider installing Malwarebytes Browser Guard, assuming your clients run browsers compatible with it, and assuming such use is permitted by Malwarebytes (I couldn't locate any info specifically allowing or prohibiting it for Malwarebytes Browser Guard, so a staff member may correct me if such use is not acceptable) as it works with Bitdefender and blocks much of the same content as Web Protection, though only for the browser it's installed in, of course.  It also blocks a few items not blocked by Web Protection, including many ads and trackers, and also has behavior based blocking for tech support scam sites (something only possible from within the browser, so not even Web Protection has that capability).

It works with most Chromium based browsers, including the newest MS Edge, as well as Mozilla Firefox.