speed and dns cache

[elementaos]

I noticed my firefox is extremely slow when i use hosts by hosts-file.net. For each click it looks like it reads whole 17 MB hosts file. All webpages are extremely slow and freeze for about 10 seconds to minute after any click or in some cases movement. When I replaced hosts with windows original empty file, firefox became normally fast. Speed difference is about 10-60x. Seems stupid how this works. It's obviously better to have more websites blocked but it will slow down everything, so these two parameters are against each other. Why everybody reccommends and adds in app to disable dns cache? Would not enabled dns cache speed something up? I don't get it. What is dns cache purpose for hosts file? It caches hosts file into RAM? Then why disable it?

Finally I think theoretically good solution would be to use 2 tools to speed up hosts file access, while not removing or reducing number of blocked pages. By making RAM disk let's say of 18 MB size. Then use junction by sysinternals to make symbolic link (or whatever that is called, just like in linux it exists) for hosts file to this ram disk. Especially useful for us who never turn off laptop but put it to sleep only. Or maybe automatize whole process for those who reset often. Good idea? Gonna try tommorow. Seems stupid to read 17 MB file for every click i make on firefox.

Edited June 11, 2018 by elementaos

[exile360]

The problem with leaving the DNS Client service (which I assume is what you're referring to as the "DNS cache" since that's one of its purposes) is that when using a large HOSTS file, it attempts to constantly read/cache the entire contents of your HOSTS file which usually results in the machine locking up due to constant high CPU usage from svchost.exe (the DNS Client service attempting to index the large HOSTS file).  If you're seeing slow browsing when using a large HOSTS file there are a couple of things you can do to eliminate the issue and keep your HOSTS file (neither of which require you to enable the DNS Client service).

First, you can use a third party tool such as HostsServer which was created by the developers of HostsMan, a tool that is used for managing several HOSTS files (including HOSTS from multiple custom sources along with well known ones like hpHosts and MVPSHosts) to make browsing faster when using a large HOSTS file.  HostsServer redirects the null lookups that occur as a result of the hosts in your HOSTS file pointing to 127.0.0.1 to itself so that the machine doesn't have to wait for all those blocked connections to timeout (waiting for them to timeout is what's causing your slow browsing experience, by the way).  You can also try changing your HOSTS file so that the address the blocked sites point to is 0.0.0.0 rather than 127.0.0.1 as that sometimes works (though not in all situations) to eliminate these performance issues.  That's the method I use.  I'm using HostsMan and currently have a HOSTS file with over 1 million entries in it (it's approximately 26MB in size at the moment) and browsing is fast for me in all of my browsers (IE11, Chrome, Firefox and SRWare Iron).

Another alternative, especially if the HOSTS file you're using isn't too large (for example, since you're only using a HOSTS file from a single source unlike me (I combine HOSTS files from over 10 separate sources in addition to some custom block lists of my own creation) would be to "optimize" your HOSTS file.  What I mean by that is, you can actually place multiple domains on a single line in a HOSTS file and still have the machine understand and block them correctly.  A tool such as HostsMan can do this automatically via one of its options or you can do it manually.  The end result is the same number of domains blocked, but using far fewer lines, and reducing the number of total lines improves the performance and indexing speed should you choose to keep the DNS Client service active (DNS caching).

More info is available at the following links:

https://www.wilderssecurity.com/threads/a-tip-to-cut-down-the-hosts-file.321755/
http://winhelp2002.mvps.org/hosts.htm#DNS
https://download.cnet.com/s/hostsserver/

I've also found that since I started using 0.0.0.0 rather than 127.0.0.1 and using Simple DNSCrypt (which sets my DNS to 127.0.0.1 deliberately in order to use their alternate DNS) that I have no issues with slow browsing speeds or page loading times.

Edited June 11, 2018 by exile360

[elementaos]

Interesting, thanks. I hope you remove duplicate lines when using hosts from 2 or more different sources, because many lines are same. You can use https://www.raymond.cc/blog/remove-duplicate-lines-with-textcrawler-scratchpad-or-notepad/

Edited June 11, 2018 by elementaos

[exile360]

You're welcome.  If you need any help or advice on issues like this, including a deeper dive into HOSTS files (including my multitude of various sources) you can create a topic here and we can help you out.

Yep, no duplicates in my HOSTS file.  HostsMan finds and removes all duplicate lines automatically so each entry in my HOSTS file is unique.

[elementaos]

Actually nothing helps, just tried using HostsMan. Converted all to 0.0.0.0 and 9 hosts per line... Tried restarting pc. But as soon as pc starts up there is high cpu and disk usage. And these 2 programs show that HOSTS file is being continually read. For 5 minutes already few GB get read although hosts file is now only about 12 MB. Something is wrong. Whole pc is slow even before opening any web browser. I made sure i do not replace localhost with 0.0.0.0 and tried removing it too, no difference. When i removed hosts file pc is fast again.

Edited June 11, 2018 by elementaos

[exile360]

Did you disable the DNS Client service in services.msc?  If that service is active, that's the most likely source of the CPU usage.

[elementaos]

It can't be disabled in windows 10. Even msvp or hosts-file mentions that. Had to use their workaround by disabling it using Autoruns by Sysinternals. It is disabled tough, as reported in services in os or in hostsman. Did that yesterday.

[exile360]

And did you reboot the PC after disabling it?

[exile360]

By the way, if you're using an antivirus and haven't excluded your HOSTS file from being scanned, that could also be the reason, as could a software firewall if you're using one (other than the built in Windows Firewall).

[elementaos]

I turned off windows defender completely using group policy. Yes I restarted pc at least 10 times since yesterday. Now when i made my new hosts file and added just 2 entries for test, it works fine. Hosts get blocked and procmon shows no activity with hosts file. Could it be that only large file gets problems or file from hosts-file.net has some bugs inside?

[elementaos]

Actually this happens even with my small 2 lines file.

[exile360]

Interesting, then it likely isn't the DNS Client service.  Can you try resetting the HOSTS file to default/normal to see if the issue goes away?  Also, if you open Task Manager and locate the instance of svchost.exe using the CPU, right-click on it and select Go to Service(s) and make a note of all of the highlighted services in the list.  Hopefully that will help us narrow down where the CPU spikes are coming from.

[elementaos]

Only one out of many svchost.exe uses some cpu, and it is service WdiSystemHost (Diagnostic System Host). But then regardless of that I noticed firefox does the same separately, and in less than 5 minutes it has already read about 376 MB of hosts, and svchost has read 17 MB (17948728 B, exactly same as file size). So this means svchost is OK? It's not reading anymore bytes once this size fullfiled as being read. Firefox increases forever :( And I said it happens with normal hosts too, just more difficult to detect, firefox reads more than file size of hosts. Could it be something weird with my network? For example I use your-freedom app to access free internet and it makes server at localhost port 80 and 1080. Then I use proxifier to force all apps to use that ip and port as proxy. And my your-freedom uses dns mode as real internet, meaning every day I use over GB of dns traffic. So called dns tunneling, probably illegal type of connection that can be used to access free internet worldwide. By the time i written this my hosts file read bytes are already 1 GB. Svchost didn't do any more reading than hosts file size. Only firefox.

[elementaos]

Here is how your-freedom dns mode works, it somehow reads random hostnames, many times per second, each hostname is different then previous one. So this is the culprit? Can this specific app (it's actually java) somehow skip hosts file. Because it's useless, it won't find anything there useful for it. All it's hostnames are fake and random but similar. Used only to tunnel all traffic. There are options in my app.

[elementaos]

It's not yf or proxifier. Even when i completely close them, meaning when i do not have access to internet but have access to dns, the same happens with firefox.

[exile360]

I'm not sure what it is then.  When I use Firefox, even with my massive HOSTS file, it doesn't use much CPU at all except when I'm loading a page, but after that it drops back down to 0.  It might be something specific to Firefox in Windows 10 but I can't test that since I'm using Windows 7.

[exile360]

By the way, if you end up not being able to use a HOSTS file but still want to block a lot of the bad stuff on the web in Firefox, including malware, malvertisements, ad trackers, scam sites, phishing sites and more then you might give the new Malwarebytes for Firefox beta plugin a try.  It enhances the web blocking capabilities in Malwarebytes Premium, but even without a license for Malwarebytes, it still blocks a lot of undesirable content on the web.  If you combine it with something like Adblock Plus or uBlock Origin then you should be able to eliminate a lot of what the HOSTS files would block for you anyway without having to use one.

[elementaos]

Well, the reason I decided to use hosts file method few days ago was because adblockers have the same problem. Adblockplus slows down my browsers, especially firefox. I am using firefox 52.8 ESR for my win xp and win 10. I currently have both installed on my laptop. Adblockplus easylist is large in size, and looks like is getting larger every day. It's few MB. Also I don't like it because it freezes whenever i browser its options, for example just to write one custom filter, it takes about a minute because of slowdowns between each click. Hosts at least will block everything system-wide. I mean all these adblockers, addons, hosts rely on some large database meaning they use hdd.

So now I tried on windows xp. Installed and disabled dns client. I used exactly same firefox as in windows 10 (i do not use quantum version because of need for some old plugins like unmht...) and because using junction app i can let browser in both os-es use same data folder. And results are exactly same in xp. Except that browser is not slow. Meaning I observed in procmon that 3 GB are already read in about 20 minutes of using internet. This looks like stupid idea to me. Wastes hard disk, and must be more or less slow. Maybe it is not slow on xp because file is defragmented or because windows 10 has some stupid processes/services/whatever. I am 100% sure it is same with all of you, you all waste hard disk, probably about 10-50 GB just to read hosts file. In xp, probably just like for your win7, it is unnoticable - meaning its fast.

In win10 I noticed firewall was turned on and defender service was on although i disabled it in group policy. So I additionaly disabled it in autoruns by sysinternals, and disabled firewall, and disabled those win10 notifications (that if clicked would reenable firewall)...but even then after restart situation is same. Also tried reenabling dns client, and looks like this just additionaly slows everything down, procmon shows hosts is beaing read very slowly but constantly. So useless.

 

So I will try with my idea of putting hosts into ram using virtual ram disk and junction, so that i only have to read from hdd once (17 MB) instead of many GB. Not only this should improve speed but also save my hdd.

 

And I hate windows 10, looks like everything is slower than in xp. More processes and services, problems with admin privilegies, problems with associations (almost no program capable of making associations by itself), more restrictions (very difficult to disable windows defender or dns client...). The only reason i am using it is because xp has no support for new apps and some https encryption algorhitms (well firefox has, but internet download manager doesn't, although internet download accelerator has, it is worse dm than idm)...although idm "made a solution" and microsoft "made update to supprot TLS 1.1 and 1.2 in xp" looks like this is just fake. Because these programs would have to be remade to support that. For example internet explore 8 by itself doesnt support such protocols and idm is relying on that, regardless of updates.

Edited June 12, 2018 by elementaos

[elementaos]

[exile360]

Wow, yeah, I don't blame you there.  I don't see any of these issues in Windows 7, but if I did I probably wouldn't continue to use the HOSTS file either.  Have you tried HostsServer yet to see if it helps at all?  It's supposed to improve performance when using a large HOSTS file, especially when using it with the DNS Client service disabled as it replaces its caching functions.

You could attack the problem in a different way by using some alternate means of blocking these sites, such as adding them to the Windows Firewall with Advanced Security rules (in Windows 10; not XP obviously since it doesn't exist there and doesn't support that).  You can find info on how to do that here as well as here although I believe that method requires you to block the actual IP addresses of the sites you want blocked, meaning safe sites that share IP addresses/servers with the sites you're blocking will also be blocked (the pitfall of IP blocking tools).

There are also other third party tools that can be used for blocking/filtering sites that are reasonable alternatives to the HOSTS file such as (possibly, at least based on my research so far) Privoxy and even Proxifier which you're already using.  If you can import the list of domains to block from your HOSTS file into Proxifier then you should be all set, assuming it doesn't cause tons of CPU usage in Firefox once you do.  Also, according to this you should be able to do the same with Fiddler and/or Charles Proxy.

Honestly though, I think the issue comes down to some kind of problem with the version of Firefox you're forced to use because you're on XP (at least for XP, but the cause could be the same on 10 since you're using the same version of FX there as well), but if one of these alternatives works then you should be all set.

[exile360]

I also discovered this if you want to use the blocklist functionality in DNSCrypt as an alternative to a HOSTS file (blocking custom lists of domains and IP addresses is a built-in feature for DNSCrypt and is something I've used in the past with Simple DNSCrypt before I discovered I could use the Windows Firewall to block custom lists of IP addresses which I did in order to block Microsoft's telemetry servers).

[elementaos]

Yeah I will try more methods sometimes later, to compare which is the best. I found currently best solution. First made virtual drive on drive letter Z and copied whole etc folder there. Then removed "c:/windows/system32/drivers/etc/" folder.  Used ImDisk RamDisk Configuration, and using junction made "c:/windows/system32/drivers/etc/" to point to "Z:/etc" where Z is my virtual 20 MB drive. Stored that drive into img file and made it autostart on every boot from that file. Also made it to synchronize all changes from ram drive to img file on shutdown/logoff if changed anything. So everything is now automatic, no need to setup anything manually on each boot. There is already visible speed increase in HostsMan, for example when it counts entries, previously it took about 3 seconds. Now 1 second max. Editing is fast too. Everything works as expected. This is on xp, now gonna try on win10 to see if firefox will stop being slow. And yeah, there is no more hard disk activity being observed in procmon, not even on virtual drive. I put virtual drive in physical memory, not in virtual memory of course.

 

I tried previously to change this key in regedit: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\DataBasePath: "%SystemRoot%\System32\drivers\etc" to "Z:\etc" to avoid using junction but it didn't work.

Edited June 12, 2018 by elementaos

[elementaos]

Nothing works on windows 10. Sometimes its idm or svchost or firefox or malwarebytes or edge who use 95% cpu. All try to access hosts repeatedly. Whole os becomes unusable. Maybe something in windows 10 design. XP has no problems. Although constant reading of file occurs in both os-es, in xp it is slow enough and no high cpu usage. So weird yeah. Actually putting file into ram like this causes pc to be even more slow, because cpu can work faster because hdd is not slowing it down from accessing file. Good old xp. I will try to investigate this more when i have more time. I do not use win7 because it doesnt support s2 and s3 sleep state for my hardware, even after installing specific update that fixes it, or after installing all updates.

It's good trick to put something into ram to make it faster. For example easy2boot installs xp this way, and installation is very fast. I can put some game or web browser data folder to speed them up.

Edited June 12, 2018 by elementaos

[exile360]

Well that's frustrating.  Hopefully you can figure out a reasonable alternative that works so that you can use your HOSTS/block lists without the performance issues.  If I come up with any new ideas I'll definitely let you know.

[elementaos]

No, tried proxifier and proxycap. They can block using stars and questionmarks too, which is advantage. But they can accept only very little hostnames. Even if i edit their settings file manually, they just freeze and refuse to start. If I do that via app, app cuts out 99% of hostnames. I figured out on windows 10 we must not ever disable dns client. No wonder why it can't be disabled using anything but Autoruns by sysinternals. It's not meant to work that way, unlinke in previous oses. When it is not disabled, still useless, because reading of my file takes about 1 hour, and if i would edit even one symbol it would reread whole file. Luckily i may use proxifier for few most important hosts and some adblocker, maybe it will be faster if its settings or whole browser data folder in virtual drive, that would at least speed up adblockers. HostsSerer doesn't change anything.

Edited June 12, 2018 by elementaos