Using high performance systems to build collections for a digital library
Published 25 June 2003
Donna Bergmark
Citations1
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work describes the experience with a highly parallel Web crawling system (Mercator) to construct - automatically - collections of scientific resources for the National Science Digital Library.
Abstract
Nothing is more distributed than the Web, with its content spread across thousands of servers. High performance hardware and software is essential for an effective download, analysis, and organization of this content. We describe our experience with a highly parallel Web crawling system (Mercator) to construct - automatically - collections of scientific resources for the National Science Digital Library.
Keywords
Computer Science
Journal of the ACMAuthoritative sources in a hyperlinked environment
9,060 Citations1999Jon Kleinberg
This work proposes and test an algorithmic formulation of the notion of authority, based on the relationship between a set of relevant authoritative pages and the set of “hub pages” that join them together in the link structure, and has connections to the eigenvectors of certain matrices associated with the link graph.
Computer NetworksFocused crawling: a new approach to topic-specific Web resource discovery
1,492 Citations1999Soumen Chakrabarti, Martin van den Berg +1 more
A new hypertext resource discovery system called a Focused Crawler that is robust against large perturbations in the starting set of URLs, and capable of exploring out and discovering valuable resources that are dozens of links away from the start set, while carefully pruning the millions of pages that may lie within this same radius.
Lecture notes in computer scienceThe Web as a Graph: Measurements, Models, and Methods
1,009 Citations1999Jon Kleinberg, Ravi Kumar +3 more
This paper describes two algorithms that operate on the Web graph, addressing problems from Web search and automatic community discovery, and proposes a new family of random graph models that point to a rich new sub-field of the study of random graphs, and raises questions about the analysis of graph algorithms on the Internet.
World Wide WebMercator: A scalable, extensible Web crawler
578 Citations1999Allan Heydon, Marc Najork
This paper describes Mercator, a scalable, extensible Web crawler written entirely in Java, and comments on Mercator's performance, which is found to be comparable to that of other crawlers for which performance numbers have been published.
ACM Transactions on Information SystemsSALSA
359 Citations2001Ronny Lempel, Shlomo Moran
It is proved that SALSA is quivalent to a weighted in degree analysis of the link-sturcutre of WWW subgraphs, making it computationally more efficient than the Mutual reinforcement approach, and comparisions reveal a topological Phenomenon called the TKC effect which prevents the Mutual Reinforcement approach from identifying meaningful authorities.
Design and implementation of a high-performance distributed Web crawler
328 Citations2003Vladislav Shkapenyuk, Torsten Suel
This paper describes the design and implementation of a distributed Web crawler that runs on a network of workstations that scales to several hundred pages per second, is resilient against system crashes and other events, and can be adapted to various crawling applications.
Massive computingHigh-Performance Web Crawling
100 Citations2002Marc Najork, Allan Heydon
This chapter describes the experience building and operating a high-performance crawler, which is an important component of many web services and uses data structures far too large to fit in main memory to access and update them efficiently.
Lecture notes in computer scienceFocused Crawls, Tunneling, and Digital Libraries
97 Citations2002Donna Bergmark, Carl Lagoze +1 more
A combination of focused crawling and tunneling could be an effective tool for building digital libraries by evaluating the current crawl direction on the fly to determine when to terminate a tunneling activity.
Core services in the architecture of the national science digital library (NSDL)
64 Citations2002Carl Lagoze, William Y. Arms +13 more
The first phase of the interoperability infrastructure including the metadata repository, search and discovery services, rights management services, and user interface portal facilities are described.
Collection synthesis
48 Citations2002Donna Bergmark
This paper reports on the use of one such crawler to synthesize document collections on various topics in science, mathematics, engineering and technology that could be part of a digital library.
arXiv (Cornell University)Core Services in the Architecture of the National Digital Library for Science Education (NSDL)
34 Citations2002Carl Lagoze, William Y. Arms +13 more
The first phase of the interoperability infrastructure including the metadata repository, search and discovery services, rights management services, and user interface portal facilities are described.
ACM SIGIR ForumCompiling document collections from the Internet
15 Citations2000Vitaliy Kluev
Results suggest that the rough estimation of precision and recall calculated in this study offer great promise, and that the time required for manual analysis of document content by the crawler was significantly reduced.