Detecting and Modeling Local Text Reuse
Published 4 December 2015
David A. Smith, Ryan Cordell, Elizabeth Maddock Dillon, Nick Stramp, John Wilkerson
Citations28
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
Texts propagate through many social networks and provide evidence for their structure. We describe and evaluate ef-ficient algorithms for detecting clusters of reused passages embedded within longer documents in large collections. We apply these techniques to two case studies: analyzing the culture of free reprinting in the nineteenth-century United States and the development of bills into legislation in the U.S. Congress. Using these divergent case studies, we eval-uate both the efficiency of the approximate local text reuse detection methods and the accuracy of the results. These techniques allow us to explore how ideas spread, which ideas spread, and which subgroups shared ideas. 1.
Keywords
Computer Science
PubMedRegularization Paths for Generalized Linear Models via Coordinate Descent.
13,977 Citations2010Jerome H. Friedman, Trevor Hastie +1 more
Algorithms on strings, trees, and sequences: computer science and computational biology
3,529 Citations1997Dan Gusfield
Cambridge University Press eBooksAlgorithms on Strings, Trees and Sequences
3,045 Citations1997Dan Gusfield
Ukkonen’s method is the method of choice for most problems requiring the construction of a suffix tree, and it will be presented first because it is easier to understand.
Algorithms on strings, trees, and sequences
1,723 Citations1997Dan Gusfield
Meme-tracking and the dynamics of the news cycle
1,547 Citations2009Jure Leskovec, Lars Bäckström +1 more
This work develops a framework for tracking short, distinctive phrases that travel relatively intact through on-line text; developing scalable algorithms for clustering textual variants of such phrases, and identifies a broad class of memes that exhibit wide spread and rich variation on a daily basis.
Finding near-duplicate web pages
424 Citations2006Monika Henzinger
A combined algorithm is presented which achieves precision 0.79 with 79% of the recall of the other algorithms, and since Charikar's algorithm finds more near-duplicate pairs on different sites, it achieves a better precision overall than Broder et al.'s algorithm.
Cambridge University Press eBooksCongress and the Politics of Problem Solving
278 Citations2013E. Scott Adler, John Wilkerson
Pairwise document similarity in large collections with MapReduce
204 Citations2008Tamer Elsayed, Jimmy Lin +1 more
This paper presents a MapReduce algorithm for computing pairwise document similarity in large document collections that exhibits linear growth in running time and space in terms of the number of documents.
Local text reuse detection
95 Citations2008Jangwon Seo, W. Bruce Croft
A new approach to detecting local text reuse is introduced and compared to other approaches and this comparison involves a study of the amount and type of reuse that occurs in real documents, including TREC newswire and blog collections.
Lecture notes in computer scienceA Scalable System for Identifying Co-derivative Documents
73 Citations2004Yaniv Bernstein, Justin Zobel
Spex is presented, a novel hash-based algorithm for extracting duplicated chunks from a document collection and deco, a prototype system that makes use of spex, is described.
Digital Studies / Le champ numériqueSomething Borrowed: Sequence Alignment and the Identification of Similar Passages in Large Text Collections
40 Citations2011Mark Olsen, Russell Horton +1 more
A simple technique to identify lexically-similar passages in large collections of text using sequence alignment algorithms is described, which is hoped to broaden the discussion concerning the relationship of Enlightenment thought to previous intellectual traditions as well as its reuse in the centuries that followed.
Generating links by mining quotations
35 Citations2008Okan Kolak, Bill N. Schilit
A scalable algorithm for mining repeated word sequences from extremely large text corpora and a new user interface for navigating across and within works in the collection using quotation links are presented.
Collective graph identification
32 Citations2011Galileo Namata, Stanley Kok +1 more
This paper introduces the problem of graph identification, i.e., the discovery of the true graph structure underlying an observed network, and empirically demonstrates that C3 is superior, in terms of both predictive accuracy and runtime, to state-of-the-art probabilistic approaches on three real-world problems.
Efficient indexing of repeated n-grams
26 Citations2011Samuel Huston, Alistair Moffat +1 more
The proposed method for finding the repeated n-grams in text corpora provides an alternative to previous approaches that scales almost linearly in the length of the sequence, is largely independent of n, and provides a uniform workload balance across the set of available processors.
Partial duplicate detection for large book collections
26 Citations2011İsmet Zeki Yalnız, Ethem F. Can +1 more
Experiments on several datasets show that DUPNIQ is more accurate than traditional methods for duplicate detection such as shingling and is fast.
Homomorphic fingerprints under misalignments
18 Citations2013Alexandr Andoni, Assaf Goldberger +2 more
This paper presents the first linear sketch that is robust to a small number of alignment errors and can be used to determine whether two files are within a small Hamming distance of being a cyclic shift of each other.
Finding and exploring memes in social media
12 Citations2012Hohyon Ryu, Matthew Lease +1 more
A new form of technological assistance for critical literacy which automatically discovers and displays underlying memes: ideas represented by similar phrases which occur across diýerent information sources by surfacing these memes to users, creating a rich hypertext representation in which underlying memes can be explored in context.