Web Search Technology
The Internet EncyclopediaPublished 3 January 2004
Clement Yu, Weiyi Meng
Citations4
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This chapter describes the inner workings of search engines and metasearch engines, and describes techniques used to build search engine and metAsearch engines.
Abstract
Abstract The World Wide Web has become the largest information source in recent years and search engines are indispensable tools for finding needed information from the Web. The coverage of the Web by a single search engine may be limited. One effective way to increase the search coverage of the Web is to combine the coverage of multiple search engines. Systems that do such combination are called metasearch engines. When a metasearch engine receives a query from a user, it invokes the underlying search engines to retrieve useful information for the user and combines the returned results from multiple search engines into a single ranked list before displaying it to the user. In this chapter, we describe the inner workings of search engines and metasearch engines. Specifically, we describe techniques used to build search engines and metasearch engines.
Keywords
Computer Science
Symposium on Discrete AlgorithmsAuthoritative sources in a hyperlinked environment
1,836 Citations1998Jon Kleinberg
Journal of the American Society for Information Science and TechnologySearching the web: The public and their queries
938 Citations2001Amanda Spink, Dietmar Wolfram +2 more
It is found that most people use few search terms, few modified queries, view few Web pages, and rarely use advanced search features, and the language of Web queries is distinctive.
Reexamining the cluster hypothesis
828 Citations1996Marti A. Hearst, Jan Pedersen
This work systematically evaluates Scatter/Gather in this context and finds significant improvements over similarity search ranking alone and provides evidence validating the cluster hypothesis which states that relevant documents tend to be more similar to each other than to non-relevant documents.
Computer Networks and ISDN SystemsAutomatic resource compilation by analyzing hyperlink structure and associated text
700 Citations1998Soumen Chakrabarti, Byron Dom +4 more
An evaluation of ARC suggests that the resources found by ARC frequently fare almost as well as, and sometimes better than, lists of resources that are manually compiled or classified into a topic.
ACM SIGIR ForumSearching Distributed Collections With Inference Networks
612 Citations2017James P. Callan, Zhihong Lu +1 more
Methods of addressing each issue in the inference network model are described, their implementation in the INQUERY system is discussed, and experimental results demonstrating their effectiveness are presented.
Database and Expert Systems ApplicationsThe INQUERY Retrieval System
526 Citations1992James P. Callan, W. Bruce Croft +1 more
A retrieval system (INQUERY) that is based on a probabilistic retrieval model and provides support for sophisticated indexing and complex query formulation is described.
ComputerMining the Web's link structure
502 Citations1999Soumen Chakrabarti, Byron Dom +6 more
Clever is a search engine that analyzes hyperlinks to uncover two types of pages: authorities, which provide the best source of information on a given topic; and hubs, which provides collections of links to authorities.
Kluwer Academic Publishers eBooksDistributed Information Retrieval
387 Citations2005Jamie Callan
A broad and diverse group of experimental results is presented to demonstrate that the algorithms are effective, efficient, robust, and scalable.
ACM Computing SurveysBuilding efficient and effective metasearch engines
324 Citations2002Weiyi Meng, Clement Yu +1 more
In this article, techniques that have been proposed to tackle several underlying challenges for building a good metasearch engine are surveyed.
IEEE ExpertThe MetaCrawler architecture for resource aggregation on the Web
252 Citations1997Erik Selberg, Oren Etzioni
The paper discusses the MetaCrawler Softbot parallel Web search service that has been available at the University of Washington since June 1995 and has some sophisticated features that allow it to obtain results of much higher quality than simply regurgitating the output from each search service.
Information RetrievalMeasuring Search Engine Quality
210 Citations2001David Hawking, Nick Craswell +2 more
Performance of the engines as a group is found to be inferior to the group of participants in the TREC-8 Large Web task, although the best engines approach the median of those systems.
Kluwer Academic Publishers eBooksCombining Approaches to Information Retrieval
202 Citations2005W. Bruce Croft
It is shown that combining approaches for information retrieval can be modeled as combining the outputs of multiple classifiers based on one or more representations, and that this simple model can provide explanations for many of the experimental results.
STARTS
200 Citations1997Luis Gravano, Chen-Chuan K. Chang +2 more
This paper describes STARTS, an emerging protocol for Internet retrieval and search that facilitates the task of querying multiple document sources and discusses the process that led to its definition.
ACM Transactions on Information SystemsExperiences with selecting search engines using metasearch
196 Citations1997Daniel Dreilinger, Adele E. Howe
The efficacy of SavvySearch's incrementally acquired metaindex approach to selecting search engines is studied by analyzing the effect of time and experience on performance and how much experience is required to surpass the simple scheme.
OnlineAn information system for corporate users: wide area information servers
166 Citations1991Brewster Kahle, Art Medlar
Application of a cognitive model that clearly defines goals and expectations for ACP discussions prior to meeting with patients and families improves rates of ACP discussions.
ACM SIGMOD RecordAutomatic discovery of language models for text databases
139 Citations1999Jamie Callan, Margaret E. Connell +1 more
ProFusion: Intelligent Fusion from Multiple, Distributed Search Engines
128 Citations1996Susan Gauch, Guijun Wang +1 more
Computer Networks and ISDN SystemsInquirus, the NECI meta search engine
126 Citations1998Steve Lawrence, C. Lee Giles
The Inquirus meta search engine makes improvements over existing search engines in a number of areas, e.g.: more useful document summaries incorporating query term context, identification of both pages which no longer exist and pages which have no longer contain the query terms.
IEEE ExpertLycos: design choices in an Internet search service
99 Citations1997M.I. Mauldin
The history and precursors of the Lycos system for collecting, storing, and retrieving information about pages on the Web are outlined and some of the design choices made in building this Web indexer are discussed.
A knowledge-based approach to organizing retrieved documents
98 Citations1999Wanda Pratt, Marti A. Hearst +1 more
DynaCat is presented, a tool that dynamically categorizes search results into a hierarchical organization by using knowledge of important kinds of queries and a model of the domain terminology to help users find answers to important types of questions more quickly and easily.
Computer Networks and ISDN SystemsALIWEB - Archie-like indexing in the WEB
92 Citations1994M.B.M. de Koster
The background and objectives of ALIWEB are discussed, an overview of its functionality and implementation is given, and it is compared to other existing resource directories in the Web.
Towards a highly-scalable and effective metasearch engine
54 Citations2001Zonghuan Wu, Weiyi Meng +2 more
A highly scalable and accurate database selection method based on a theory for ranking search engines optimally is proposed and an operational prototype system has been built based on the proposed approach.
Adaptive Agents for Information Gathering from Multiple, Distributed Information Sources
52 Citations1999Yizhong Fan, Susan Gauch
An intelligent, adaptive Web search tool that can not only locate relevant information sources for the user, but also adapt to the frequent changes of the dynamic Web environment is described.
Efficient and effective metasearch for a large number of text databases
44 Citations1999Clement Yu, Weiyi Meng +3 more
This paper proposes to use a hierarchy of database representatives to improve the efficiency of metasearch engines, and provides an algorithm to search the hierarchy and shows that the retrieval effectiveness of the algorithm is the same as that of evaluating the user query against all database representatives.
ACM SIGIR ForumInfoseek's experiences searching the internet
44 Citations1998Steven Kirsch
This article recounts both amusing and useful data Infoseek has gathered over the past four years as a leading search site on the Internet, including what people ask on the web, factors for relevance ranking web pages, techniques for searching an infinite number of web pages.
A new study on using HTML structures to improve retrieval
36 Citations2003Michal Cutler, Hepu Deng +2 more
Results are presented on a methodology of using the structures and hyperlinks of HTML documents to improve the effectiveness of retrieving HTML documents and a genetic algorithm is implemented to determine a "best so far" class importance factor combination.
USENIX Symposium on Internet Technologies and SystemsThe search broker
31 Citations1997Udi Manber, Peter A. Bigot
A method of conducting search on the web that is based on a two-level search idea that strikes a balance between flat global search and specialized databases, and gives users convenient access to vast amounts of information.