Using the Invisible Web to Find Information

View Worksheet

During the course of conducting research via the major internet search engines, you may discover that finding the scholarly resources and information you seek is difficult, if not impossible. How do you find what you need? First, understand that the web is divided into the “visible,” or searchable web and the Invisible Web. Instead of giving up out of frustration, utilize the Invisible Web to find quality information.

What is the Invisible Web?

The Invisible Web consists of web pages that are not indexed by the major search engines. This deep, or hidden web is a wealth of information, especially in medical and scientific research topics. The major search engines (Google, Yahoo!, Bing) use spiders to crawl the content on web pages to index it in order to provide search results. These spiders, however, cannot and do not index web pages where a password is required to view information or on pages where you must enter search criteria, make selections to proceed to further pages or develop a query.

What type of information is found on the Invisible Web?

Most of the information available on the Invisible Web is found in specialized databases. These databases require you to enter a search term or phrase in order to access the information. Because of this, the information is not indexed by the major search engines. When you type in a search term, a new page is dynamically generated in response to your query, giving you specific articles and information related to your search. An exception to un-indexed specialized information lies in Google Scholar, which allows you to search scholarly journals only.

Is the information found on the Invisible Web helpful?

The information found on the Invisible Web is highly specialized and usually contained within databases. This highly specialized data allows you to find relevant, specific and scholarly information whether your research is narrowly focused or broadly focused. In addition, most of the information contained within these databases is maintained by academic institutions and written or prepared by scholars and experts in specific fields. As far as reliability and validity, the resources found on the Invisible Web are often the same as those you would find by physically browsing a large academic library’s print scholarly resources. They give you reliable, scholarly sources of information that allows you to write a stronger, better researched paper or assignment.

How to access the Invisible Web

Accessing the Invisible Web is simply of matter of knowing where to look. In some cases, you may encounter databases that restrict access to subscribed members. There are, however, plenty of free-access databases available as well. If you are a student, your school’s library likely has access to a number of subscription databases, so contact the school librarian for access information when necessary.

Within the Invisible Web, you are more likely to find statistics, research results and information gathered, examined and provided by academic institutions and researchers. You can access the Invisible Web by using the right search terms in regular search engines or by accessing and searching specialized search engines, directories and databases.

Using general search engines to access the Invisible Web

While popular search engines may not give you results that include information from the Invisible Web with a basic search, the information is accessible with the right approach. Since most information on the hidden web is contained within databases, the easiest way to access the Invisible Web with a normal search engine is to include the term “databases” after your search phrase or term. For example, if you are searching for aviation statistics, you would search for “aviation statistics databases” using Google, Yahoo! or Bing as the search engine. The same principle applies to general directories through sites such as Google or Yahoo!.

Examples of specialized search engines, directories and databases on the Invisible Web

While the size of the Invisible Web is tremendously large, there are certain search engines, directories and databases that allow you to search for more specialized, academic information. While the following list is not exhaustive, it is a great place for you to start your research on the Invisible Web.

  • [URL]CompletePlanet[http://www.completeplanet.com]—This is a large collection of searchable databases and specialty search engines.
  • [URL]The Directory of Open Access Journals [http:www.doap.org]—A searchable database of full-text journals.
  • [URL]FindArticles[http://findarticles.com]—A searchable collection of more than 10 million articles from a wide range of publications.
  • [URL]HighWire[http://highwire.stanford.edu]—Maintained by Stanford University, this site provides access to one of the internet’s largest collections of scholarly content that is free to view.
  • [URL]IncyWincy[http://www.incywincy.com/]—An Invisible Web search engine that taps into other search engines and filters the cumulative results.
  • [URL]InfoMine[http://infomine.ucr.edu/]—This site offers a large collection of online scholarly internet resources in the form of a standalone search engine, but it also offers a starting point for other references.
  • [URL]Intute[http://www.intute.ac.uk/]—A UK-based site with access to resources from some of the top academic and professional institutions. The site is not currently maintained or kept up to date (as of July 2011), but its resources are archived and searchable. The site also includes subject-specific tutorials on researching on the web at the college level.
  • [URL]Scirus[http://www.scirus.com]—A scientific search engine that searches both the visible web and the Invisible Web.
  • [URL]The WWW Virtual Library[http://vlib.org/]—The oldest catalog on the web where you can search within an alphabetical list of topics, or dive deeper into any category before searching.

While this list is just a starting point for your research, it provides insight into the type of sites you are seeking on the Invisible Web. If you are a student, your school’s library likely has direct links to specialized databases and directories to which you have access as a student. When seeking information for your research paper, remember that the Invisible Web exists, and know how to access it to ensure your sources are as relevant and of the best quality with respect to validity and reliability.