Towards a highly-scalable and effective metasearch engine
Published 1 April 2001
Zonghuan Wu, Weiyi Meng, Clement Yu, Zhuogang Li
Citations54
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A highly scalable and accurate database selection method based on a theory for ranking search engines optimally is proposed and an operational prototype system has been built based on the proposed approach.
Abstract
Article Towards a highly-scalable and effective metasearch engine Share on Authors: Zonghuan Wu Department of Computer Science, SUNY at Binghamton, Binghamton, NY Department of Computer Science, SUNY at Binghamton, Binghamton, NYView Profile , Weiyi Meng Department of Computer Science, SUNY at Binghamton, Binghamton, NY Department of Computer Science, SUNY at Binghamton, Binghamton, NYView Profile , Clement Yu Department of Computer Science, University of Illinois at Chicago, Chicago, IL Department of Computer Science, University of Illinois at Chicago, Chicago, ILView Profile , Zhuogang Li Department of Computer Science, SUNY at Binghamton, Binghamton, NY Department of Computer Science, SUNY at Binghamton, Binghamton, NYView Profile Authors Info & Claims WWW '01: Proceedings of the 10th international conference on World Wide WebMay 2001 Pages 386–395https://doi.org/10.1145/371920.372093Online:01 April 2001Publication History 29citation330DownloadsMetricsTotal Citations29Total Downloads330Last 12 Months5Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access
Keywords
Computer Science
NatureAccessibility of information on the web
1,356 Citations1999Steve Lawrence, C. Lee Giles
As the web becomes a major communications medium, the data on it must be made more accessible, and search engines need to make the data more accessible.
ScienceSearching the World Wide Web
974 Citations1998Steve Lawrence, C. Lee Giles
The coverage and recency of the major World Wide Web search engines was analyzed, yielding some surprising results, including a lower bound on the size of the indexable Web of 320 million pages.
ACM SIGIR ForumReal life information retrieval: a study of user queries on the Web
621 Citations1998Bernard J. Jansen, Amanda Spink +2 more
This work analyzed transaction logs of a set of 51,473 queries posed by 18,113 users of Excite, a major Internet search service, to provide data on the number of search terms, and the use of logic and modifiers.
ACM SIGIR ForumSearching Distributed Collections With Inference Networks
612 Citations2017James P. Callan, Zhihong Lu +1 more
Methods of addressing each issue in the inference network model are described, their implementation in the INQUERY system is discussed, and experimental results demonstrating their effectiveness are presented.
Computer Networks and ISDN SystemsA technique for measuring the relative size and overlap of public Web search engines
393 Citations1998Krishna Bharat, Andrei Broder
A standardized, statistical way of measuring search engine coverage and overlap through random queries is described that can be implemented by third-party evaluators using only public query interfaces and suggests the size of the static, public Web as of November was over 200 million pages.
Searching distributed collections with inference networks
387 Citations1995James P. Callan, Zhihong Lu +1 more
Methods of addressing each issue in the inference network model are described, dkcusses their implementation in the INQUERY system, and experimental results demonstrating their effectiveness are presented.
ACM Computing SurveysBuilding efficient and effective metasearch engines
324 Citations2002Weiyi Meng, Clement Yu +1 more
In this article, techniques that have been proposed to tackle several underlying challenges for building a good metasearch engine are surveyed.
Cluster-based language models for distributed retrieval
296 Citations1999Jinxi Xu, W. Bruce Croft
A new approach to distributed retrieval based on document clustering and language modeling is proposed and it is shown that all three methods improve the effectiveness of distributed retrieval.
The Information Manifold
257 Citations1995Thomas Kirk, Alon Y. Levy +2 more
The Information Manifold is described, a system for browsing and querying of multiple networked information sources that demonstrates the viability of knowledge representation technology for retrieval and organization of information from disparate information sources.
IEEE ExpertThe MetaCrawler architecture for resource aggregation on the Web
252 Citations1997Erik Selberg, Oren Etzioni
The paper discusses the MetaCrawler Softbot parallel Web search service that has been available at the University of Washington since June 1995 and has some sophisticated features that allow it to obtain results of much higher quality than simply regurgitating the output from each search service.
Generalizing GlOSS to Vector-Space Databases and Broker Hierarchies
223 Citations1995Luis Gravano, Héctor García-Molina
ACM Transactions on Information SystemsExperiences with selecting search engines using metasearch
196 Citations1997Daniel Dreilinger, Adele E. Howe
The efficacy of SavvySearch's incrementally acquired metaindex approach to selecting search engines is studied by analyzing the effect of time and experience on performance and how much experience is required to surpass the simple scheme.
The Collection Fusion Problem.
166 Citations1994Ellen M. Voorhees, N. K. Gupta +1 more
This paper examines two collection fusion techniques that use the results of past queries to compute the number of documents to retrieve from each of a set of subcollections such that the total number of retrieved documents is equal to N, the number to be returned to the user.
Principles of Database Query Processing for Advanced Applications
163 Citations1997Clement Yu, Weiyi Meng
Effective retrieval with distributed collections
157 Citations1998Jinxi Xu, Jamie Callan
It is found that when a large number of collections are available, the retrieval effectiveness is significantly worse than that of centralized systems, mainly because typical queries are not adequate for the purpose of choosing the right collections.
ACM SIGMOD RecordAutomatic discovery of language models for text databases
139 Citations1999Jamie Callan, Margaret E. Connell +1 more
Computer Networks and ISDN SystemsInquirus, the NECI meta search engine
126 Citations1998Steve Lawrence, C. Lee Giles
The Inquirus meta search engine makes improvements over existing search engines in a number of areas, e.g.: more useful document summaries incorporating query term context, identification of both pages which no longer exist and pages which have no longer contain the query terms.
ACM Transactions on Information SystemsMethods for information server selection
121 Citations1999David Hawking, Paul B. Thistlewaite
A novel method using Lightweight Probe queries (LWP method) is compared with several methods based on data from past query processing, while Random and Optimal server rankings serve as controls.
Computer NetworksQuery routing for Web search engines: architecture and experiments
99 Citations2000Atsushi Sugiura, Oren Etzioni
Q-Pilot is described, an automatic query routing system that attempts to dynamically route each user query to the appropriate specialized search engines, based on an off-line component that creates an approximate model of each specialized search engine's topic.
A probabilistic solution to the selection and fusion problem in distributed information retrieval
71 Citations1999Christoph Baumgarten
A model for optimal information retrieval over a distributed document collection is described and experimentally evaluated and part of the model is a selection criterion for limiting the ranking process to a subset of subcollections.
Merging Ranks from Heterogeneous Internet Sources
64 Citations1997Luis Gravano, Héctor García-Molina
This paper presents a condition that a source must satisfy so that a meta-broker can extract the top objects for a query from the source without examining its entire contents, and shows an efficient algorithm to extract theTop objects from sources that satisfy the given condition.
Discovering the representative of a search engine
60 Citations2001King-Lup Liu, Adrain Santoso +2 more
This paper shows that the needed information can be estimated from an uncooperative search engine with good accuracy and presents techniques for the estimation of the number of documents indexed by the search engine and the maximum weight of each term.
A probabilistic model for distributed information retrieval
55 Citations1997Christoph Baumgarten
Adaptive Agents for Information Gathering from Multiple, Distributed Information Sources
52 Citations1999Yizhong Fan, Susan Gauch
An intelligent, adaptive Web search tool that can not only locate relevant information sources for the user, but also adapt to the frequent changes of the dynamic Web environment is described.
Estimating the usefulness of search engines
52 Citations1999Weiyi Meng, King-Lup Liu +3 more
A statistical method to estimate the usefulness of a search engine for any given query, which can be used by a metasearch engine to choose local search engines to invoke.
IEEE Transactions on Knowledge and Data EngineeringA methodology to retrieve text documents from multiple databases
49 Citations2002C. Yu, King-Lup Liu +3 more
It is shown that the effectiveness of the best-search algorithm is the same as that of evaluating the user query against all database representatives, and will guarantee the retrieval of the n most similar documents for the query.
Efficient and effective metasearch for text databases incorporating linkages among documents
44 Citations2001Clement Yu, Weiyi Meng +2 more
The importance (rank) of each document as determined by the linkages is integrated in each database representative to facilitate the selection of databases for each given query.
Efficient and effective metasearch for a large number of text databases
44 Citations1999Clement Yu, Weiyi Meng +3 more
This paper proposes to use a hierarchy of database representatives to improve the efficiency of metasearch engines, and provides an algorithm to search the hierarchy and shows that the retrieval effectiveness of the algorithm is the same as that of evaluating the user query against all database representatives.
ACM SIGIR ForumInfoseek's experiences searching the internet
44 Citations1998Steven Kirsch
This article recounts both amusing and useful data Infoseek has gathered over the past four years as a leading search site on the Internet, including what people ask on the web, factors for relevance ranking web pages, techniques for searching an infinite number of web pages.
IEEE Transactions on Knowledge and Data EngineeringA statistical method for estimating the usefulness of text databases
39 Citations2002King-Lup Liu, C. Yu +3 more
A statistical method to estimate the usefulness of a search engine for any given query is presented and results indicate that the estimation method is much more accurate than existing methods.
USENIX Symposium on Internet Technologies and SystemsThe search broker
31 Citations1997Udi Manber, Peter A. Bigot
A method of conducting search on the web that is based on a two-level search idea that strikes a balance between flat global search and specialized databases, and gives users convenient access to vast amounts of information.
Detection of heterogeneities in a multiple text database environment
24 Citations1999Weiyi Meng, Clement Yu +1 more
This work first analyzes the impact of various heterogeneities on building a metasearch engine, then presents some techniques that can be used to detect the most prominentheterogeneities among multiple search engines.
Finding the most similar documents across multiple text databases
24 Citations2003Clement Yu, King-Lup Liu +3 more
A methodology for finding the n most similar documents across multiple text databases for any given query and for any positive integer n and a number of strategies are presented to retrieve documents from the identified databases.