Impact of search engines on page popularity
Published 17 May 2004
Junghoo Cho, Sourashis Roy
Citations285
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper analytically estimates how much longer it takes for a new page to attract a large number of Web users when search engines return only popular pages at the top of search results and shows that search engines can have an immensely worrisome impact on the discovery of new Web pages.
Abstract
Recent studies show that a majority of Web page accesses are referred by search engines. In this paper we study the widespread use of Web search engines and its impact on the ecology of the Web. In particular, we study how much impact search engines have on the popularity evolution of Web pages. For example, given that search engines return currently popular" pages at the top of search results, are we somehow penalizing newly created pages that are not very well known yet? Are popular pages getting even more popular and new pages completely ignored? We first show that this unfortunate trend indeed exists on the Web through an experimental study based on real Web data. We then analytically estimate how much longer it takes for a new page to attract a large number of Web users when search engines return only popular pages at the top of search results. Our result shows that search engines can have an immensely worrisome impact on the discovery of new Web pages.
Keywords
Computer SciencePhysics and Astronomy
ScienceEmergence of Scaling in Random Networks
36,366 Citations1999Albert-Ĺaszló Barabási, Réka Albert
A model based on these two ingredients reproduces the observed stationary scale-free distributions, which indicates that the development of large networks is governed by robust self-organizing phenomena that go beyond the particulars of the individual systems.
Computer Networks and ISDN SystemsThe anatomy of a large-scale hypertextual Web search engine
15,828 Citations1998Sergey Brin, Lawrence M. Page
This paper provides an in-depth description of Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext and looks at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want.
The PageRank Citation Ranking : Bringing Order to the Web
12,645 Citations1999Lawrence M. Page, Sergey Brin +2 more
This paper describes PageRank, a mathod for rating Web pages objectively and mechanically, effectively measuring the human interest and attention devoted to them, and shows how to efficiently compute PageRank for large numbers of pages.
Journal of the ACMAuthoritative sources in a hyperlinked environment
9,060 Citations1999Jon Kleinberg
This work proposes and test an algorithmic formulation of the notion of authority, based on the relationship between a set of relevant authoritative pages and the set of “hub pages” that join them together in the link structure, and has connections to the eigenvectors of certain matrices associated with the link graph.
NatureDiameter of the World-Wide Web
4,285 Citations1999Réka Albert, Hawoong Jeong +1 more
The World-Wide Web becomes a large directed graph whose vertices are documents and whose edges are links that point from one document to another, which determines the web's connectivity and consequently how effectively the authors can locate information on it.
Lecture notes in computer scienceThe Web as a Graph: Measurements, Models, and Methods
1,009 Citations1999Jon Kleinberg, Ravi Kumar +3 more
This paper describes two algorithms that operate on the Web graph, addressing problems from Web search and automatic community discovery, and proposes a new family of random graph models that point to a rich new sub-field of the study of random graphs, and raises questions about the analysis of graph algorithms on the Internet.
ScienceStrong Regularities in World Wide Web Surfing
761 Citations1998Bernardo A. Huberman, Peter Pirolli +2 more
A model that assumes that users make a sequence of decisions to proceed to another page, continuing as long as the value of the current page exceeds some threshold, yields the probability distribution for the number of pages that a user visits within a given Web site.
The Evolution of the Web and Implications for an Incremental Crawler
498 Citations2000Junghoo Cho, Héctor García-Molina
An architecture for the incremental crawler is proposed, which combines the best design choices, which can improve the ``freshness'' of the collection significantly and bring in new pages in a more timely manner.
Proceedings of the National Academy of SciencesWinners don't take all: Characterizing the competition for links on the web
461 Citations2002David M. Pennock, Gary William Flake +3 more
A simple generative model quantifies the degree to which the rich nodes grow richer, and how new (and poorly connected) nodes can compete, and accurately accounts for the true connectivity distributions of category-specific web pages, the web as a whole, and other social networks.
World Wide WebChanges in Web client access patterns: Characteristics and caching implications
377 Citations1999Paul Barford, Azer Bestavros +2 more
This study compares two measurements of Web client workloads separated in time by three years, both captured from the same computing facility at Boston University and finds that for the computing facility represented by traces between 1995 and 1998, the benefits of using size‐based caching policies have diminished and the potential for caching requested files in the network has declined.
A large-scale study of the evolution of web pages
344 Citations2003Dennis Fetterly, Mark S. Manasse +2 more
Rate of change and other metrics: a live study of the world wide web
318 Citations1997Fred Douglis, Anja Feldmann +2 more
The potential benefit of a shared proxy-caching server in a large environment is quantified by using traces that were collected at the Internet connection points for two large corporations, representing significant numbers of references.
Computer NetworksHow dynamic is the Web?
290 Citations2000Brian Brewington, George Cybenko
Using empirical models and a novel analytic metric of `up-to-dateness', the rate at which Web search engines must re-index the Web to remain current is estimated.
Adaptive on-line page importance computation
216 Citations2003Serge Abiteboul, Mihaí Preda +1 more
A new algorithm OPIC is introduced that works on-line, and uses much less resources, and does not require storing the link matrix, and is used to focus crawling to the most interesting pages.
Software Practice and ExperienceA large‐scale study of the evolution of Web pages
202 Citations2004Dennis Fetterly, Mark S. Manasse +2 more
Computer NetworksWebBase: a repository of Web pages
187 Citations2000Jun Hirai, Sriram Raghavan +2 more
This paper focuses on the storage manager module, and illustrates how traditional techniques for storage and indexing can be tailored to meet the requirements of a web repository.
Predictive caching and prefetching of query results in search engines
173 Citations2003Ronny Lempel, Shlomo Moran
PDC (probability driven cache), a novel scheme tailored for caching search results, that is based on a probabilistic model of search engine users is presented, and prefetching can increase cache hit ratios by 50% for large caches, and can double the hit ratios of small caches.
Optimal crawling strategies for web search engines
149 Citations2002Joel L. Wolf, Mark S. Squillante +3 more
A two-part scheme, based on network flow theory, that determines the (nearly) optimal crawling frequencies, as well as the theoretically optimal times to crawl each web page, within an extremely general stochastic framework.
Page quality
128 Citations2005Junghoo Cho, Sourashis Roy +1 more
The proposed quality estimator has the potential to alleviate the rich-get-richer phenomenon and help new and high-quality pages get the attention that they deserve and is derived through a careful analysis of a reasonable web user model.
A new paradigm for ranking pages on the world wide web
74 Citations2003John A. Tomlin
This paper describes a new paradigm for modeling traffic levels on the world wide web using a method of entropy maximization, and specifically applies the primal and dual solutions of this model to the (static) ranking of web sites.
Computer NetworksTowards a better understanding of Web resources and server responses for improved caching
63 Citations1999Craig E. Wills, Михаил Михайлов
Results from the work indicate that there is potential to reuse more cached resources than is currently being realized due to inaccurate and nonexistent cache directives, and that separating out the dynamic portions of a page into their own resources allows relatively static portions to be cached.
Web Traffic Logs: An Imperfect Resource for Evaluation
43 Citations1999Brian D. Davison
This paper enumerates a number of deficiencies in most common, publicly available request logs and investigates some of the additional insights available from logs containing full content, and how they might apply to content-based prefetching proxies.