Detecting phrase-level duplication on the world wide web

Published 15 August 2005

Dennis Fetterly, Mark S. Manasse, Marc Najork

Citations127

TL;DR

The algorithms used to discover a number of other instances of large-scale phrase-level replication within the two data sets collected in December 2002 and June 2004 are described.

Abstract

Two years ago, we conducted a study on the evolution of web pages over time. In the course of that study, we discovered a large number of machine-generated "spam" web pages emanating from a handful of web servers in Germany. These spam web pages were dynamically assembled by stitching together grammatically well-formed German sentences drawn from a large collection of sentences. This discovery motivated us to develop techniques for finding other instances of such "slice and dice" generation of web pages, where pages are automatically generated by stitching together phrases drawn from a limited corpus. We applied these techniques to two data sets, a set of 151 million web pages collected in December 2002 and a set of 96 million web pages collected in June 2004. We found a number of other instances of large-scale phrase-level replication within the two data sets. This paper describes the algorithms we used to discover this type of replication, and highlights the results of our data mining.

Keywords

Computer Science

Computer NetworksGraph structure in the Web

2,760 Citations2000Andrei Broder, Ravi Kumar +6 more

The study of the web as a graph yields valuable insight into web algorithms for crawling, searching and community discovery, and the sociological phenomena which characterize its evolution.

Computer Networks and ISDN SystemsSyntactic clustering of the Web

1,347 Citations1997Andrei Broder, S. Glassman +2 more

An efficient way to determine the syntactic similarity of files is developed and applied to every document on the World Wide Web, and a clustering of all the documents that are syntactically similar is built.

A large-scale study of the evolution of web pages

344 Citations2003Dennis Fetterly, Mark S. Manasse +2 more

Spam, damn spam, and statistics

293 Citations2004Dennis Fetterly, Mark S. Manasse +1 more

This paper proposes that some spam web pages can be identified through statistical analysis, and examines a variety of properties, including linkage structure, page content, and page evolution, and finds that outliers in the statistical distribution of these properties are highly likely to be caused by web spam.

Some applications of Rabin’s fingerprinting method

197 Citations1993Andrei Broder

This paper presents an implementation and several applications of BenRabin's fingerprinting scheme that take considerable advantage of its algebraic properties.

Who links to whom: mining linkage between Web sites

168 Citations2002Krishna Bharat, Bay-Wei Chang +2 more

A modified "copy model" to explain observed link weight distributions as a function of subgraph size is discussed and a data mining application of the hostgraph is described: a related host finding algorithm which achieves a precision of 0.65 at rank 3.

View PDF

Recognizing Nepotistic Links on the Web

150 Citations2000Brian D. Davison

High accuracy in initial experiments is reported to show the potential for using a machine learning tool to automatically recognize and eliminate nepotistic links— links between pages that are present for reasons other than merit.

The connectivity sonar

108 Citations2003Einat Amitay, David Carmel +3 more

This research argues that sites of similar role exhibit similar structural patterns, as the functionality of a site naturally induces a typical hyperlinked structure and typical connectivity patterns to and from the rest of the Web.