Crawler-Based Search Engines

Crawler-based search engines, such as Google, create their listings automatically. They "crawl" or "spider" the web, then people search through what they have found.

If you change your web pages, crawler-based search engines eventually find these changes, and that can affect how you are listed. Page titles, body copy and other elements all play a role.

Human-Powered Directories

A human-powered directory, such as the Open Directory, depends on humans for its listings. You submit a short description to the directory for your entire site, or editors write one for sites they review. A search looks for matches only in the descriptions submitted.

Changing your web pages has no effect on your listing. Things that are useful for improving a listing with a search engine have nothing to do with improving a listing in a directory. The only exception is that a good site, with good content, might be more likely to get reviewed for free than a poor site.

"Hybrid Search Engines" Or Mixed Results

In the web's early days, it used to be that a search engine either presented crawler-based results or human-powered listings. Today, it extremely common for both types of results to be presented. Usually, a hybrid search engine will favor one type of listings over another. For example, MSN Search is more likely to present human-powered listings from LookSmart. However, it does also present crawler-based results (as provided by Inktomi), especially for more obscure queries.

The Parts Of A Crawler-Based Search Engine

Crawler-based search engines have three major elements. First is the spider, also called the crawler. The spider visits a web page, reads it, and then follows links to other pages within the site. This is what it means when someone refers to a site being "spidered" or "crawled." The spider returns to the site on a regular basis, such as every month or two, to look for changes.

Everything the spider finds goes into the second part of the search engine, the index. The index, sometimes called the catalog, is like a giant book containing a copy of every web page that the spider finds. If a web page changes, then this book is updated with new information.

Sometimes it can take a while for new pages or changes that the spider finds to be added to the index. Thus, a web page may have been "spidered" but not yet "indexed." Until it is indexed --added to the index -- it is not available to those searching with the search engine.

Search engine software is the third part of a search engine. This is the program that sifts through the millions of pages recorded in the index to find matches to a search and rank them in order of what it believes is most relevant.

Major Search Engines: The Same, But Different

All crawler-based search engines have the basic parts described above, but there are differences in how these parts are tuned. That is why the same search on different search engines often produces different results . Information on this page has been drawn from the help pages of each search engine, along with knowledge gained from articles, reviews, books, independent research, tips from others and additional information received directly from the various search engines.

We Can Submit your site in Major Search engine in very cheap price with good Ranking. Please email us at submission@softwebsnepal.om

You don't have to pay thousands to get noticed! Our premium packages include... • Website review and custom optimization analysis • Quick listing express submissions to the top search engines • Custom keyword research • Optimized title and meta tags • Submission to 75,000+ search engines and directories • Safe follow-up re-submissions • Detailed ranking reports • Helpful professional support

In the search engine list below, Search Engine Watch provides a guide to the major search engines of the web. Why are these considered to be "major" search engines? Because they are either well-known or well-used.

For webmasters, the major search engines are the most important places to be listed, because they can potentially generate so much traffic.

For searchers, well-known, commercially-backed search engines generally mean more dependable results. These search engines are more likely to be well-maintained and upgraded when necessary, to keep pace with the growing web.

Top Choices

The search engines below are all excellent choices to start with when searching for information.

Google
http://www.google.com/

Voted three times Most Outstanding Search Engine , Google has a well-deserved reputation as the top choice for those searching the web. The crawler-based service provides both comprehensive coverage of the web along with great relevancy. It's highly recommended as a first stop in your hunt for whatever you are looking for.

Google provides the option to find more than web pages, however. Using "tabs" on the top of the search box on the Google home page, you can easily seek out images from across the web, discussions that are taking place on Usenet newsgroups, scan through human-compiled information provided from the Open Directory locate news information Also offered, though not through tabs, is catalog searching and product searching.

Google is also know for the wide range of features it offers, such as cached links that let you "resurrect" dead pages or see older versions of recently changed ones. It offers excellent spell checking, easy access to dictionary definitions, integration of stock quotes, street maps, telephone numbers and more. See Google's help page for an entire rundown on some of these features. The Google Toolbar has also won a popular following for the easy access it provides to Google and its features directly from the Internet Explorer browser.

In addition to Google's unpaid editorial results, the company also operates its own advertising programs. The cost-per-click AdWords program places ads on Google as well as some of Google's partners. Similarly, Google is also a provider of unpaid editorial results to some other search engines.

We Can Submit your site in Major Search engine in very cheap price with good Ranking. Please email us at submission@softwebsnepal.om

You don't have to pay thousands to get noticed! Our premium packages include... • Website review and custom optimization analysis • Quick listing express submissions to the top search engines • Custom keyword research • Optimized title and meta tags • Submission to 75,000+ search engines and directories • Safe follow-up re-submissions • Detailed ranking reports • Helpful professional support

AllTheWeb.com
http://www.alltheweb.com/

An excellent crawler-based search engine, All The Web provides both comprehensive coverage of the web and outstanding relevancy. If you tried Google and didn't find it, All The Web should probably be next on your list. Indeed, it's a first stop search engine, for some. In addition to web page results, AllTheWeb.com provides the ability to search for news stories, pictures, video clips, MP3s and FTP files.

Until recently, AllTheWeb.com was owned by a company called FAST and used as a showcase for that company's web search technology. That's why you sometimes may sometimes hear AllTheWeb.com also referred to as FAST or FAST Search.

Yahoo
http://www.yahoo.com/

Launched in 1994, Yahoo is the web's oldest "directory," a place where human editors organize web sites into categories. However, in October 2002, Yahoo made a giant shift to using Google's crawler-based listings for its main results.

If Yahoo is now powered by Google, then why bother using it? For one thing, you might find that the way Yahoo "enhances" Google's listings with information from its own directory may make search results more readable.

In addition, Yahoo's search results pages still show Categories links. When offered, these will take you to a list of web sites that have been reviewed and approved by a human editor.

It's also possible to do a pure search of just the human-compiled Yahoo Directory, which is how the old or "classic" Yahoo used to work. To do this, search from the Yahoo Directory home page, as opposed to the regular Yahoo.com home page. Then you'll get both directory category links ("Related Directory Categories") and "Directory Results," which are the top web site matches drawn from all categories of the Yahoo Directory.

Sites pay a fee to be included in the Yahoo Directory's commercial listings, though they must meet editor approval before being accepted. Non-commercial content is accepted for free.

Consider Yahoo any time you think you might be well served by having a list of human-reviewed web sites. It's also a good choice for popular queries, since the category listings it provides may help you narrow in and refine your query. Doing a pure Yahoo Directory search also provides an unique human view of the web.

We Can Submit your site in Major Search engine in very cheap price with good Ranking. Please email us at submission@softwebsnepal.om

You don't have to pay thousands to get noticed! Our premium packages include... • Website review and custom optimization analysis • Quick listing express submissions to the top search engines • Custom keyword research • Optimized title and meta tags • Submission to 75,000+ search engines and directories • Safe follow-up re-submissions • Detailed ranking reports • Helpful professional support

MSN Search
http://search.msn.com/

Microsoft is known for constantly reworking its software products until they get them right, and MSN Search is a shining example of the company putting that same effort into an online product. In particular, the company has its own team of editors that monitors the most popular searches being performed and then hand-picks sites that are believed to be the most relevant. After performing a search, "Popular Topics" shown below the search box on the results page are also suggestions built largely by editors to guide you into making a more refined search. When appropriate, search results may also feature links to encyclopedia content from Microsoft Encarta or news headlines, at the top of the page.

Of course, humans editors can't do everything, so MSN Search also relies on search providers for answers to many of its queries. Usually, it will be human-powered results from the LookSmart directory that dominate the page. Unlike when MSN editors are involved, these human-powered results are not hand-picked to match a query.

Overall, MSN Search provides a blend of human-powered directory information and crawler coverage different from any of the other top choices listed above. It's a high quality resource that provides its own unique view of the web and one worth checking.

AOL Search
http://aolsearch.aol.com/ (internal)
http://search.aol.com/(external)

AOL Search provides users with editorial listings that come Google's crawler-based index. Indeed, the same search on Google and AOL Search will come up with very similar matches. So, why would you use AOL Search? Primarily because you are an AOL user. The "internal" version of AOL Search provides links to content only available within the AOL online service. In this way, you can search AOL and the entire web at the same time. The "external" version lacks these links. Why wouldn't you use AOL Search? If you like Google, many of Google's features such as "cached" pages are not offered by AOL Search.

Ask Jeeves
http://www.askjeeves.com/

Ask Jeeves initially gained fame in 1998 and 1999 as being the "natural language" search engine that let you search by asking questions and responded with what seemed to be the right answer to everything.

In reality, technology wasn't what made Ask Jeeves perform so well. Behind the scenes, the company at one point had about 100 editors who monitored search logs. They then went out onto the web and located what seemed to be the best sites to match the most popular queries.

Today, Ask Jeeves instead depends on crawler-based technology to provide results to its users. These results come from the Teoma search engine that it owns.

Ask Jeeves also owns the Direct Hit service, but results from Direct Hit are no longer offered to the public directly through the Direct Hit site.

We Can Submit your site in Major Search engine in very cheap price with good Ranking. Please email us at submission@softwebsnepal.om

You don't have to pay thousands to get noticed! Our premium packages include... • Website review and custom optimization analysis • Quick listing express submissions to the top search engines • Custom keyword research • Optimized title and meta tags • Submission to 75,000+ search engines and directories • Safe follow-up re-submissions • Detailed ranking reports • Helpful professional support

Lycos
http://www.lycos.com/

Lycos is one of the oldest search engines on the web, launched in 1994. It ceased crawling the web for its own listings in April 1999 and instead uses crawler-based results provided by

"Fast Forward" lets you see search results in one side of your screen and the actual pages listed in another. Relevant categories of human-compiled information from the Open Directory appear at the bottom of the search results page. At the top of the page, Lycos will suggest other searches related to your original topic right under the search box. Perhaps you might even like the look and feel better! Whatever the reason, under the hood, Lycos provides all the same relevancy and comprehensiveness you'll find at AllTheWeb.com.

Inktomi
http://www.inktomi.com/

Among the major search engines, Inktomi is the second-oldest crawler. It briefly operated as an experimental search engine at UC Berkeley. However, the creators then formed their own company in 1996 with the same name and gained their first customer, HotBot, in the middle of that year. The company then pursued a strategy of "powering" other search engines, rather than running its own branded service for the public.

Today, Inktomi continues to crawl the web. The company had been left behind by rivals Google and AllTheWeb.com in terms of comprehensiveness, but changes made in the summer of 2002 made it much more competitive. It was purchased by Yahoo in March 2003.

We Can Submit your site in Major Search engine in very cheap price with good Ranking. Please email us at submission@softwebsnepal.om

You don't have to pay thousands to get noticed! Our premium packages include... • Website review and custom optimization analysis • Quick listing express submissions to the top search engines • Custom keyword research • Optimized title and meta tags • Submission to 75,000+ search engines and directories • Safe follow-up re-submissions • Detailed ranking reports • Helpful professional support

LookSmart
http://www.looksmart.com/

LookSmart is a human-compiled directory of web sites. The company does operate its own web site, but this really isn't intended for the public to use. Instead, similar to Inktomi, LookSmart provides its results to other search engines that need listings.

LookSmart gathers its listings in two ways. Commercial sites pay to be listed in its commercial categories, making the service very much like an electronic "Yellow Pages." However, volunteer editors at the LookSmart-owned Zeal directory also catalog sites into non-commercial categories for free. Though Zeal is a separate web site, its listings are integrated into LookSmart's results.

LookSmart launched independently in October 1996, was backed by Reader's Digest for about a year, and then company executives bought back control of the service.

Open Directory
http://dmoz.org/

The Open Directory uses volunteer editors to catalog the web. Formerly known as NewHoo, it was launched in June 1998. It was acquired by AOL Time Warner-owned Netscape in November 1998, and the company pledged that anyone would be able to use information from the directory through an open license arrangement.

While you can search at the Open Directory site itself, this is not recommended. The site has no "backup" results that kick in should there not be a match in the human-compiled database. In addition, the ranking of sites during keyword searching is poor, while alphabetical ordering is used when you choose to "browse" categories by topic.

Instead, to scan the valuable information compiled by the Open Directory, consider using the version offered by Google, the Google Directory. Here, keyword searching uses Google's refined relevancy algorithms and makes use of link analysis to better propel good pages from the human database to the top. In addition, when viewing sites by category, they will be listed in PageRank order, which means the most popular sites based on analyzing links from across the web will be listed first.

We Can Submit your site in Major Search engine in very cheap price with good Ranking. Please email us at submission@softwebsnepal.om

You don't have to pay thousands to get noticed! Our premium packages include... • Website review and custom optimization analysis • Quick listing express submissions to the top search engines • Custom keyword research • Optimized title and meta tags • Submission to 75,000+ search engines and directories • Safe follow-up re-submissions • Detailed ranking reports • Helpful professional support

AltaVista
http://www.altavista.com/

AltaVista is the oldest crawler-based search engine on the web. It opened in December 1995 and for several years was the "Google" of its day, in terms of providing relevant results and having a loyal group of users that loved the service.

Sadly, an attempt to turn AltaVista into a portal site in 1998 saw the company lose track of the importance of search. Over time, relevancy dropped, as did the freshness of AltaVista's listings and the crawler's coverage of the web.

Today, AltaVista is once again focused on search. Improvements have been made, but crawlers such as Google and AllTheWeb provide more comprehensive results. Because of this, AltaVista is probably a third-choice crawler, one to try if you haven't found what you are looking for at one of its competitors.

AltaVista does remains strong is in terms of some of the specialty searching it offers. It provides a good image search service, and you can look for video and audio clips, as well. It also has an outstanding news search service.

AltaVista was originally owned by Digital, then taken over by Compaq, when that company purchased Digital in 1998. AltaVista was later spun off into a private company, controlled by CMGI.

We Can Submit your site in Major Search engine in very cheap price with good Ranking. Please email us at submission@softwebsnepal.om

You don't have to pay thousands to get noticed! Our premium packages include... • Website review and custom optimization analysis • Quick listing express submissions to the top search engines • Custom keyword research • Optimized title and meta tags • Submission to 75,000+ search engines and directories • Safe follow-up re-submissions • Detailed ranking reports • Helpful professional support