|Web search engines are designed to organize lists of web pages according to
key words or phrases the user might request. We all use them. We rely on them.
For most people, the hyper-popular search engines such as Google, Bing, and
Yahoo totally represent the world wide web. They are the web. Or so we
are allowed to believe.
The search engines tirelessly "crawl" the web, picking their way through every byte of every website they visit. These "crawlers" provide data that is evaluated and organized according to algorithms that are provide the kind of information that users ordinarily find when they request a search. These algorithms are designed according to criteria set by search-engine programming, and those criteria are sometimes prejudiced by corporate necessities and social whimsies. Strange as it may seem, however, there are actually websites that don't give a flip whether they appear on your search list or not, and there are even more places that send the web crawlers a clear message, "Get out and stay out!"
According to an article in the November 2, 2015, issue of Darkweb News, only 4% of the web is represented by the almighty search engines. This image here of an iceberg illustrate the point. The Visible Web is the "tip of the iceberg" that is accessible to search engines. As far as most people know, that IS the web.
But what about the 96% of the stuff below the surface--in the Deep Web? In spite of the fact that web crawlers ignore it (or are locked out), most of the sites are wide open to the public. But can you see the problem? With search engines unable to work with the Deep Web, the links have to be located and listed manually, one at at time. Free-Ed.Net (where you are at this moment) is a curator of Deep Web links that enrich the work of self-directed lifelong learners.
|David L. Heiserman, Editor||
Copyright © SweetHaven
Revised: June 06, 2015