Directories

The Parts of a Search Engine

The Spider

The Index

Search Engine Software

In Short

To the typical Web user, there may be little difference between search engines and directories.  Enter a keyword, click on go, and get the search results.  In fact, about 85 percent of users find what they're looking for on the Web by using one or more of the hundreds of search engines and directories that are online.

To a Web site owner, however, knowing the difference between search engines and directories and knowing how they function is fundamental to surviving and thriving online. 

Search engines create their listings using different software programs.  These programs 'crawl' the Web like spiders, visiting the different sites that have been submitted to them.  After reviewing and reading each site, these programs index what they have found.  People, then search through what the engines have compiled, by typing in words or phrases that they want to find.  Search engines constantly have their 'spiders' crawling the Web, reading and indexing sites.  Page titles, descriptions, body text, code, and other elements all play a role in how a site is indexed by a search engine.

Millions of people use directories to search what they are looking for, and never know it.  Some of the more popular directories are Yahoo and LookSmart.  Where directories differ from search engines, is in how they index and review your site. While search engines send spiders to visit and index your site, directories take the descriptions sent to them (like the description you enter and submit using our service), and review these.  Some Directories, Like Yahoo actually have employees who read and index each site using a set of predefined categories.  Therefore, it is important that you supply an accurate description that will cover your entire site.

When a search is performed on a directory, it searches through the descriptions it has indexed, and tries to deliver the most relevant response.  Because this is the case, getting listed in a directory can be very different from getting listed in a search engine.  They are both looking at different criteria. However, there is one common theme.  If you submit a description, or submit keywords, which are not relevant to your actual site, your chances of getting listed and ranked, are very slim.  In fact, you may be blacklisted by several search engines and directories, and not be allowed to submit again.

Like most engines, a search engine is actually comprised of many parts.  And each search engine is a little different than the other search engines.  Like all companies trying to compete for business, each search engine tries to differentiate itself, through the process it uses to index and search through the information on the web.

On the most basic level however, most search engines are comprised of three main elements:

1. The Spider

2. The Index

3. Search Engine Software

Once a URL is submitted to a search engine, the spider, sometimes called a robot or a crawler is sent out from the search engine to read and review the site.  This is not an immediate process, and can take up to four weeks.  Once the spider reaches the site, it looks at specific elements.  Some search engines ask their spiders to look at META tags, others want a review of the actual site content, and others want some combination of the two.  However, all the information gathered is stored and taken back to the search engine.

When the spider returns to the search engine, it takes all the information it has gathered, and puts it in a central location.  Then all this information is sorted and cataloged into the Index.  The manner in which each search engine indexes the information varies from search engine to search engine.  However, each search engine needs an index to store and group its information for the searches that are performed.

Once all the information has been gathered by the spider, and sorted in the index, it is ready for the public to use.  When people visit a search engine, and type in the words and phrases they want to search, the search engine software delivers the results they receive.  This software recognizes the words or phrases being searched for, dives into the index, and tries to find relevant words or phrases from the information it has stored.  Once it has gathered what it feels are relevant sites, it then ranks each site and displays them for the user.  The order and formulas used to rank each site are different for each search engine. That is why you can search for the same words on several search engines, and come up with different results.

The list of sites you receive after a search is generated by search engine software.  The software searches the index or catalog to find good matches for your request. The information and data that make up this index is delivered by the search engine spider.  The spider searches different sites (like yours) on the web, to gather this information and data.