TextTiling: segmenting text into multi-paragraph subtopic passages
Published 1 March 1997
Marti A. Hearst
Citations1,253
SJR quartileQ1
SJR score1.15
SNIP4.16
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The algorithm is fully implemented and is shown to produce segmentation that corresponds well to human judgments of the subtopic boundaries of 12 texts, which should be useful for many text analysis tasks, including information retrieval and summarization.
Abstract
TextTiling is a technique for subdividing texts into multi-paragraph units that represent passages, or subtopics. The discourse cues for identifying major subtopic shifts are patterns of lexical co-occurrence and distribution. The algorithm is fully implemented and is shown to produce segmentation that corresponds well to human judgments of the subtopic boundaries of 12 texts. Multi-paragraph subtopic segmentation should be useful for many text analysis tasks, including information retrieval and summarization. 1.
Keywords
Computer ScienceArts and Humanities
Cohesion in English
5,806 Citations2014Μ. Α. Κ. Halliday, Ruqaiya Hasan
International Journal of LexicographyIntroduction to WordNet: An On-line Lexical Database<sup>*</sup>
4,207 Citations1990George A. Miller, Richard Beckwith +3 more
Standard alphabetical procedures for organizing lexical information put together words that are spelled alike and scatter words with similar or related meanings haphazardly through the list.
Computational LinguisticsAttention, intentions, and the structure of discourse
2,606 Citations1986Barbara J. Grosz, Candace L. Sidner
A new theory of discourse structure that stresses the role of purpose and processing in discourse is explored and various properties of discourse are described, and explanations for the behavior of cue phrases, referring expressions, and interruptions are explored.
arXiv (Cornell University)Using Information Content to Evaluate Semantic Similarity in a Taxonomy
2,147 Citations1995Philip Resnik
arXiv (Cornell University)Assessing agreement on classification tasks: the kappa statistic
2,092 Citations1996Jean Carletta
A trainable document summarizer
1,349 Citations1995Julian Kupiec, Jan Pedersen +1 more
The trends in the results are in agreement with those of Edmundson who used a subjectively weighted combination of features as opposed to training the feature weights using a corpus, which suggests that even shorter extracts may be useful indicative summmies.
Lexical cohesion computed by thesaural relations as an indicator of the structure of text
941 Citations1991Jane Morris, Graeme Hirst
RHETORICAL STRUCTURE THEORY: A THEORY OF TEXT ORGANIZATION
894 Citations1987William C. Mann
This paper establishes a new definitional foundation for RST, Definitions are made more systematic and explicit, they introduce a new functional element, and incidentally reflect more experience in text analysis.
Word-sense disambiguation using statistical models of Roget's categories trained on large corpora
670 Citations1992David Yarowsky
A program that disambiguates English word senses in unrestricted text using statistical models of the major Roget's Thesaurus categories, enabling training on unrestricted monolingual text without human intervention.
The Thread of Discourse
585 Citations1975Joseph E. Grimes
Multi-paragraph segmentation of expository text
558 Citations1994Marti A. Hearst
TextTiling, an algorithm for partitioning expository texts into coherent multi-paragraph discourse units which reflect the subtopic structure of the texts, is described and shown to produce segmentation that corresponds well to human judgments of the major subtopic boundaries of thirteen lengthy texts.
TileBars
554 Citations1995Marti A. Hearst
One sense per discourse
524 Citations1992William A. Gale, Kenneth Church +1 more
An experiment confirmed the hypothesis that if a polysemous word such as sentence appears two or more times in a well-written discourse, it is extremely likely that they will all share the same sense and found that the tendency to share sense in the same discourse is extremely strong.
arXiv (Cornell University)Multi-Paragraph Segmentation of Expository Text
392 Citations1994Marti A. Hearst
Journal of the American Society for Information ScienceHow effective is suffixing?
388 Citations1991Donna Harman
Three measures were selected which evaluate performance at given rank cutoff points, such as those cor- responding to a screenful of document titles,such as those responding to the lists of the top ranked documents.
Approaches to passage retrieval in full text information systems
387 Citations1993Gerard Salton, James Allan +1 more
New approaches are described in this study for implementing selective passage retrieval systems, and identifying text passages responsive to particular user needs.
Information Processing & ManagementConstructing literature abstracts by computer: Techniques and prospects
385 Citations1990Chris D. Paice
The question of how to achieve proper balance in auto-abstracts is a problem which has hardly been addressed hitherto; it appears that assignment of selected textual material into ‘abstract-frames’ may offer a solution.
Empirical Studies on the Disambiguation of Cue Phrases
372 Citations1991Julia Hirschberg, Diane Litman
This paper reports results of empirical studies on discourse and sentential uses of cue phrases, in which both text-based and prosodic features were examined for disambiguating power.
Journal of the American Society for Information ScienceStemming algorithms: A case study for detailed evaluation
371 Citations1996David A. Hull
A case study of stemming algorithms is described which describes a number of novel approaches to evaluation and demonstrates their value.
Subtopic structuring for full-length document access
330 Citations1993Marti A. Hearst, Christian Plaunt
It is argued that the advent of large volumes of full-length text, as opposed to short texts like abstracts and newswire, should be accompanied by corresponding new approaches to information access and a partition of the text into coherent multi-paragraph units that represent the pattern of subtopics that comprise the text.
Edinburgh Research ExplorerA problem for RST: the need for multi-level discourse analysis
231 Citations1992Johanna D. Moore, Martha E. Pollack
Two levels of analysis are focused on, which involve the relation between the information conveyed in consecutive elements of a coherent discourse and the fact that discourses are produced to effect changes in the mental state of the discourse participants.
Mixed initiative in dialogue
219 Citations1990Marilyn Walker, Steve Whittaker
A set of rules for the transfer of control is applied to 4 sets of dialogues and it is found that both allocation of control and the manner in which control is transferred is radically different for the two dialogue types.
Text segmentation based on similarity between words
203 Citations1993Hideki Kozima
Comparison with the text segments marked by a number of subjects shows that LCP closely correlates with the human judgments, which may provide valuable information for resolving anaphora and ellipsis.
A prosodic analysis of discourse segments in direction-giving monologues
191 Citations1996Julia Hirschberg, Christine H. Nakatani
The effects of speaking style and of discourse segmentation method (text-alone versus text-and-speech) on the nature of this relationship between intonational variation and discourse structure are examined.
Automatic text decomposition using text segments and text themes
176 Citations1996Gerard Salton, Amit Singhal +2 more
The interaction between text segments and text themes is used to characterize text structure, and to formulate specifications for information retrieval, text traversal, and text summarization.
Estimating upper and lower bounds on the performance of word-sense disambiguation programs
176 Citations1992William A. Gale, Kenneth Church +1 more
This paper will attempt to establish upper and lower bounds on the level of performance that can be expected in an evaluation, and finds that the upper bound is very dependent on the instructions given to the judges.
Overview of the first TREC conference
175 Citations1993Donna Harman
There was a large variety of retrieval techniques reported on, including methods using automatic thesaurii, sophisticated term weighting, natural language techniques, relevance feedback, and advanced pattern matching.
Intention-based segmentation
153 Citations1993Rebecca J. Passonneau, Diane Litman
A two part study evaluates the statistical reliability of human segmentation of a corpus of spontaneous, narrative monologues, where speaker intention is the segmentation criterion.
An automatic method of finding topic boundaries
140 Citations1994Jeffrey C. Reynar
A new method of locating discourse boundaries based on lexical cohesion and a graphical technique called dotplotting is outlined, which can be performed either manually, by examining a graph, or automatically, using an optimization algorithm.
Combining multiple knowledge sources for discourse segmentation
100 Citations1995Diane Litman, Rebecca J. Passonneau
This work predicts discourse segment boundaries from linguistic features of utterances, using a corpus of spoken narratives as data to develop segmentation algorithms from training data using hand tuning and machine learning.
Communications of the ACMInformedia Digital Video Library
98 Citations1995Mike Christel, Takeo Kanade +5 more
The Informedia Digital Video Library Project is developing new technologies for creating full-content search and retrieval digital video libraries and is creating a testbed that will enable K-12 students to access, explore, and retrieve science and mathematics materials from the digital video library.
eScholarship (California Digital Library)Structural Complexity in Fairy Tales
95 Citations1972George Lakoff
LanguageA New Tool for Discourse Analysis: The Vocabulary-Management Profile
91 Citations1991Gilbert Youmans
Discourse deixis
84 Citations1988Bonnie Webber
This paper proposes and argues for a process-based account of subsequent reference via deictic expressions, and attributes distinct mental reality to units of text often called discourse segments, a reality that is distinct from that of the entities described therein.
Redundancy in collaborative dialogue
77 Citations1992Marilyn Walker
A model of the role of informationally redundant utterances in providing evidence to support inferences about mutual understanding and acceptance is proposed and contributes to a theory of dialogue by showing how mutual beliefs can be inferred in the absence of the master-slave assumption.
The use of emphasis to automatically summarize a spoken discourse
76 Citations1992F.R. Chen, M. Withgott
The authors describe a method for exploiting prosodic information in natural, conventional speed for the purpose of automatically creating an audio summary based on identifying emphasized speech and then using proximity measures on the emphasized regions to select summarizing excerpts.
Selective text utilization and text traversal
57 Citations1993Gerard Salton, James F. Allen
Global text comparison methods are used to identify similarities between text elements, followed by local context-checking operations that resolve ambiguities and distinguish superficially similar texts from texts that actually cover identical topics.
Digital Access to Scholarship at Harvard (DASH) (Harvard University)Instructions for annotating discourse
55 Citations1995Christine H. Nakatani, Barbara J. Grosz +2 more
This guide contains instructions for segmenting discourse that are meant to be self-contained and can be used by "naive" subjects, i.e., segmenters who have not studied discourse theory or discourse processing methods.
Tense trees as the "fine structure" of discourse
52 Citations1992Chung Hee Hwang, Lenhart K. Schubert
A new compositional tense-aspect deindexing mechanism that makes use of tense trees as components of discourse contexts that allows reference episodes to be correctly identified even for embedded clauses and for discourse that involves shifts in temporal perspective.
Towards new measures of information retrieval evaluation
48 Citations1995William Hersh, Diane L. Elliot +3 more
A new approach to assessing information obtained is utilized, based on a short-answer test given to senior medical students, which showed that pre-searching scores on the test were low but that searching yielded a high proportion of answers with both systems.
International Journal of Man-Machine StudiesMulti-window displays for readers of lengthy texts
46 Citations1987Jo W. Tombaugh, Ann Lickorish +1 more
Two experiments explore whether it would help readers re-locate information in an “electronic book” if different windows on the screen were used to display specific sections of the text and suggest ways in which the display of lengthy electronic texts may be improved.
Retrieval of Partial Documents.
46 Citations1993Alistair Moffat, Ron Sacks‐Davis +2 more
How to break documents into parts, how to implement retrieval of parts; and the impact of division on retrieval effectiveness are considered.
The interpretation of tense in discourse
45 Citations1987Bonnie Webber
This paper gives an account of the role tense plays in the listener's reconstruction of the events and situations a speaker has chosen to describe and several new ideas are presented: that tense is better viewed by analogy with definite NPs than with pronouns.
Context and structure in automated full-text information access
45 Citations1994Martha Alice Hearst
A graphical interface is described, called Cougar, that displays retrieved documents in terms of interactions among their automatically-assigned main topics, thus allowing users to familiarize themselves with the topics and terminology of a text collection.
TextTiling: A Quantitative Approach to Discourse
43 Citations1993Marti A. Hearst
Replicability of Transaction and Action Coding in the Map Task Corpus
30 Citations1995Amy Isard, Jean Carletta
A way of identifying transactions and their associated actions for HCRC Map Task dialogues is devised, and the replicability of the coding scheme is tested using naive subjects.
ACM SIGMOD RecordMetadata for mixed-media access
27 Citations1994Francine Chen, Marti A. Hearst +3 more
A grammatico-statistical approach to discourse partitioning
20 Citations1994Tadashi Nomoto, Yoshihiko Nitta
The present approach to text segmentation is quite successful in recovering articles from the unstructured corpus and makes use of coherence measure based on tf-idf, a standard information retrieval measurement.
arXiv (Cornell University)Stylistic Variation in an Information Retrieval Experiment
18 Citations1996Jussi Karlgren
The generation of high-level structure for extended explanations
12 Citations1990David J. Mooney, Sandra Carberry +1 more
A computational model is presented based on the hypothesis that high-level structure can be determined by bottom-up processes that attempt to satisfy speaker, listener, and compositional goals, after which top-down strategies can be used to organize the material about the selected framework.
Text, speech and language technologyComparative Discourse Analysis of Parallel Texts
8 Citations1999Pim van der Eijk
In a parallel corpus, similar representations can be derived for versions of a text in various languages and used for parallel segmentation and as an alternative measure of text-translation similarity.