Search Engine Optimization History

By most accounts, the honor of being the first Internet search engine goes to Archie. A pre-Web search application created in 1990 by a McGill University student named Alan Emtage. By 1990, academics and technologists were regularly using the Internet to store papers, technical specs, and other kinds of documents on machines that were publicly accessible. Unless you had the exact machine address and file name, however, it was nearly impossible to find those archives. Archie scoured Internet-based archives (hence the name “Archie”) and build an index of each file it found.

Based on the Internet's file transfer protocol (FTP) standard, Archie's architecture was similar to most moderns Web search engines – it crawled sources, built and index, and had a search interface. But the pre-web era was not very user friendly time. Only true techies and academics used Archie, though among that crowd it was quite popular. Typical users would query the engine by connecting directly to an Archie server via a command line interface. They would query Archie via keywords thought to be in a matching file's title, then receive a list of places where a particular matching file could be found.

They then connected to that machine, and rummaged around till they found what they were looking. Not particularly  robust, but. better than nothing.

The name "Archie" had a quirky appeal that seemed to fit young Internet. In 1993, students at the University of Nevada: created Veronica, a play on the comic book couple. Veronica worked much as Archie did but substituted  Gopher, another popular and  more fully featured Internet file-sharing standard, for FTP. Veronica moved search a bit to what we now now expect--the Gopher standard allowed searchers to connect directly  to the document queried as opposed to just the machine on which the document resided. Not a huge step, but progress.

Both Archie and Veronica lacked semantic abilities-they didn't  index the full text of the document, just the document's title. That meant a searcher had to know-or infer-the title of the document he or she was looking for. If you were looking for a "to do list" and its title was "Today's Tasks" you'd be out of luck, even if the document's first words were, in fact, "to-do list." With the rise of Web, Archie and Veronica soon fell out of favor.

As the Web took off, so did the basic problem of search. When, the Internet was the domain of academics and technologists, finding things was a limited problem. But from 1993 to 1996, the Web grew from130 sites to more than 600,000. Watching all this growth was Matthew Gray, a. researcher at the Massachusetts Institute of Technology and a pioneer of the earliest Web-based search engine the WWW Wanderer.

The Wanderer solved a basic problem Gray had noted with Web, namely, that it was growing faster than any human could track. "I Wrote the Wanderer to systematically traverse the Web nd collect sites,” Gary later wrote. The Wanderer was a robot that automatically created an index of sites, and Gray hacked up a search interface that allowed users to search the index. The Wanderer was soon replaced my more powerful engines. One of the first was WebCrawler, developed by University of Washington researcher Brian Pinkerton. The WebCrawler retrieved the URLs in much the same manner as a Web browser, he then created a rudimentary crawler and started indexing WebSites. WebCrawler was important to the evolution of search because it was the first to index text of the web documents it found.1




1The Search - Copyright @ John Battle, 2005 All rights reserved