A web crawler design for data mining
Journal of Information SciencePublished 1 October 2001
Mike Thelwall
Citations144
SJR quartileQ1
SJR score0.63
SNIP1.31
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A distributed design is proposed in order to effectively use idle computing resources and to help information scientists avoid the need to employ dedicated equipment.
Abstract
The content of the web has increasingly become a focus for academic research. Computer programs are needed in order to conduct any large-scale processing of web pages, requiring the use of a web crawler at some stage in order to fetch the pages to be analysed. The processing of the text of web pages in order to extract information can be expensive in terms of processor time. Consequently a distributed design is proposed in order to effectively use idle computing resources and to help information scientists avoid the need to employ dedicated equipment. A system developed using the model is examined and the advantages and limitations of the approach are discussed.
Keywords
Computer Science
Computer Networks and ISDN SystemsThe anatomy of a large-scale hypertextual Web search engine
15,828 Citations1998Sergey Brin, Lawrence M. Page
This paper provides an in-depth description of Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext and looks at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want.
Journal of the ACMAuthoritative sources in a hyperlinked environment
9,060 Citations1999Jon Kleinberg
This work proposes and test an algorithmic formulation of the notion of authority, based on the relationship between a set of relevant authoritative pages and the set of “hub pages” that join them together in the link structure, and has connections to the eigenvectors of certain matrices associated with the link graph.
Computer NetworksGraph structure in the Web
2,760 Citations2000Andrei Broder, Ravi Kumar +6 more
The study of the web as a graph yields valuable insight into web algorithms for crawling, searching and community discovery, and the sociological phenomena which characterize its evolution.
World Wide WebMercator: A scalable, extensible Web crawler
578 Citations1999Allan Heydon, Marc Najork
This paper describes Mercator, a scalable, extensible Web crawler written entirely in Java, and comments on Mercator's performance, which is found to be comparable to that of other crawlers for which performance numbers have been published.
Journal of DocumentationThe calculation of web impact factors
466 Citations1998Peter Ingwersen
This case study reports the investigations into the feasibility and reliability of calculating impact factors for web sites, called Web Impact Factors (Web‐IF), and demonstrates that Web‐IFs are calculable with high confidence for national and sector domains whilst institutional Web‐ifs should be approached with caution.
Computer Networks and ISDN SystemsSPHINX: a framework for creating personal, site-specific Web crawlers
151 Citations1998Robert C. Miller, Krishna Bharat
SPHINX is a Java toolkit and interactive development environment for Web crawlers geared towards developing crawlers that are Web-site-specific, personally customized, and relocatable.
ScientometricsData collection methods on the Web for infometric purposes — A review and analysis
133 Citations2001Judit Bar‐Ilan
IEEE Transactions on Software EngineeringFinding idle machines in a workstation-based distributed system
104 Citations1989Marvin Theimer, Keith A. Lantz
Journal of DocumentationWeb impact factors and search engine coverage
99 Citations2000Mike Thelwall
The results indicate that search engine coverage, even of large national domains is extremely uneven and would be likely to lead to misleading calculations.
Journal of Information SciencePresenting a model for the structure and content of a university World Wide Web site
77 Citations1999Iain Middleton, Mike McConnell +1 more
A case is made for a non-static approach to Web design which identifies users and makes extensive use of themes to promote currently relevant information at the top level and is not constrained by traditional boundaries between departments.
Journal of DocumentationCan search engines be used as tools for web‐link analysis? A critical view
74 Citations1999Herbert Snyder, Howard Rosenbaum
The research finds that search engines are highly variable in the results they produce, are limited in the search functions they offer, have poorly and/or incorrectly documented functions, use search logics that are opaque, and change the search function they offer over time.
ACM Transactions on Mathematical SoftwareNEOS and Condor
67 Citations2000Michael C. Ferris, Michael P. Mesnier +1 more
The use of Condor, a distributed resource management system, is discussed as a provider of computational resources for NEOS, an environment for solving optimization problems over the Internet.
Journal of DocumentationResults from a web impact factor crawler
64 Citations2001Mike Thelwall
The principal findings were that with certain restrictions, WIFs can be calculated reliably, but do not correlate with accepted research rankings owing to the variety of material hosted on university servers.
Journal of the American Society for Information ScienceThe Web as an information source on informetrics? A content analysis
51 Citations2000Judit Bar‐Ilan
Analysis of the Web pages retrieved by the major search engines on a particular date, as a result of the query “informetrics OR informetric,” indicates that valuable, freely available data is hidden in the Web waiting to be extracted from the millions of Web pages.
Journal of the American Society for Information ScienceReaders, authors, and page structure: A discussion of four questions arising from a content analysis of web pages
50 Citations2000Stephanie W. Haas, Erika S. Grams
Internet ResearchCommercial Web site links
43 Citations2001Mike Thelwall
A survey of the links published by a relatively random collection of business Web sites indicates that around 66 percent of Web sites do carry external links, most of which are targeted at a specific purpose, but that about 17 percent publish general links, with implications for those designing and marketing Web sites.
Interacting with ComputersHow did university departments interweave the Web: A study of connectivity and underlying factors
33 Citations1998Chaomei Chen, Julian Newman +2 more
The relationship between the organisational profile of a university department in Scotland and its structural connectivity on the WWW and a content survey of WWW-based education activities in American land-grant universities to investigate successful applications of these enabling techniques in education.
Computer NetworksMining the Web for relations
27 Citations2000Neel Sundaresan, Jeonghee Yi
This paper defines and formalizes the general duality problem of relations on the Web, and solves the problem of identifying acronyms and their expansions through patterns of occurrences of (acronym, expansion) pairs as they occur in Web pages.
Online and CD-Rom ReviewWorld wide web robots: an overview
20 Citations1999Tham Yoke Chun
This paper traces the development of World Wide Web Robots and provides an overview of their main functions and workings, focusing on search robots and illustrations will be drawn from two major search engines: AltaVista and Excite.
Computer NetworksMy personal search engine
13 Citations1999Mark A.C.J. Overmeer
A modular approach to constructing spiders will facilitate more development, hopefully resulting in new information retrieval techniques, and improve access to data on the Internet, which is crucial in the authors' information driven society.