TopCat: data mining for topic identification in a text corpus
IEEE Transactions on Knowledge and Data EngineeringPublished 1 August 2004
Chris Clifton, Robert Cooley, Jason D. M. Rennie
Citations121
SJR quartileQ1
SJR score2.57
SNIP3.30
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
TopCat (topic categories) is a technique for identifying topics that recur in articles in a text corpus. Natural language processing techniques are used to identify key entities in individual articles, allowing us to represent an article as a set of items. This allows us to view the problem in a database/data mining context: Identifying related groups of items. We present a novel method for identifying related items based on traditional data mining techniques. Frequent itemsets are generated from the groups of items, followed by clusters formed with a hypergraph partitioning scheme. We present an evaluation against a manually categorized ground truth news corpus; it shows this technique is effective in identifying topics in collections of news articles.
Keywords
Computer Science
Mining association rules between sets of items in large databases
14,720 Citations1993Rakesh Agrawal, Tomasz Imieliński +1 more
An efficient algorithm is presented that generates all significant association rules between items in the database of customer transactions and incorporates buffer management and novel estimation and pruning techniques.
Fast algorithms for mining association rules
10,739 Citations1998Rakesh Agrawal, Ramakrishnan Srikant
Program electronic library and information systemsAn algorithm for suffix stripping
8,136 Citations1980Martin Porter
An algorithm for suffix stripping is described, which has been implemented as a short, fast program in BCPL, and performs slightly better than a much more elaborate system with which it has been compared.
Lecture notes in computer scienceText categorization with Support Vector Machines: Learning with many relevant features
7,925 Citations1998Thorsten Joachims
SVMs achieve substantial improvements over the currently best performing methods and behave robustly over a variety of di-erent learning tasks, eliminating the need for manual parameter tuning.
A Comparative Study on Feature Selection in Text Categorization
4,766 Citations1997Yiming Yang, Jan Pedersen
DF thresholding, the simplest method with the lowest cost in computation, can be reliably used instead of IG or CHI when the computation of these measures are too expensive, and strong correlations between the DF, IG and CHI values of a term are found.
ACM SIGMOD RecordMining association rules between sets of items in large databases
4,475 Citations1993Rakesh Agrawal, Tomasz Imieliński +1 more
International Journal of LexicographyIntroduction to WordNet: An On-line Lexical Database<sup>*</sup>
4,207 Citations1990George A. Miller, Richard Beckwith +3 more
Standard alphabetical procedures for organizing lexical information put together words that are spelled alike and scatter words with similar or related meanings haphazardly through the list.
Word association norms, mutual information, and lexicography
3,674 Citations1990Kenneth Church, Patrick Hanks
Future Generation Computer SystemsMining generalized association rules
1,617 Citations1997Ramakrishnan Srikant, Rakesh Agrawal
A new interest-measure for rules which uses the information in the taxonomy is presented, and given a user-specified “minimum-interest-level”, this measure prunes a large number of redundant rules.
Beyond market baskets
1,287 Citations1997Sergey Brin, Rajeev Motwani +1 more
This work develops the notion of mining rules that identify correlations (generalizing associations), and proposes measuring significance of associations via the chi-squared test for correlation from classical statistics, enabling the mining problem to reduce to the search for a border between correlated and uncorrelated itemsets in the lattice.
Word association norms, mutual information, and lexicography
1,066 Citations1989Kenneth Church, Patrick Hanks
The proposed measure, the association ratio, estimates word association norms directly from computer readable corpora, making it possible to estimate norms for tens of thousands of words.
Multilevel hypergraph partitioning
718 Citations1997George Karypis, Rajat Aggarwal +2 more
The experiments show that the multilevel hypergraph partitioning algorithm produces high quality partitioning in relatively small amount of time and outperforms other schemes (in hyperedge cut) quite consistently with larger margins.
Computer NetworksGrouper: a dynamic clustering interface to Web search results
716 Citations1999Oren Zamir, Oren Etzioni
This paper introduces Grouper, an interface to the results of the HuskySearch meta-search engine, which dynamically groups the search results into clusters labeled by phrases extracted from the snippets, and reports on the first empirical comparison of user Web search behavior on a standard ranked-list presentation versus a clustered presentation.
Word sense disambiguation using Conceptual Density
387 Citations1996Eneko Agirre, Germán Rigau
The method relies on the use of the wide-coverage noun taxonomy of WordNet and the notion of conceptual distance among concepts, captured by a Conceptual Density formula developed for this purpose, for the resolution of lexical ambiguity of nouns and its automatic evaluation over the Brown Corpus.
Terminology International Journal of Theoretical and Applied Issues in Specialized CommunicationMethods of automatic term recognition
376 Citations1996Kyo Kageura, Bin Umino
This paper tries to give an overview of the principles and methods of automatic term recognition and two major trends are examined, i.e., studies in automatic recognition of significant elements for indexing mainly carried out in information-retrieval circles and current research in automaticterm recognition in the field of computational linguistics.
Data Mining and Knowledge DiscoveryBeyond Market Baskets: Generalizing Association Rules to Dependence Rules
374 Citations1998Craig Silverstein, Sergey Brin +1 more
This work develops the notion of dependence rules that identify statistical dependence in both the presence and absence of items in itemsets in the lattice and develops pruning strategies based on the closure property that lead to an efficient algorithm for discovering dependence rules.
Proceedings of the 34th Design Automation ConferenceMultilevel Hypergraph Partitioning: Application In Vlsi Domain
293 Citations2005George Karypis, Ritesh Aggarwal +2 more
Communications of the ACMNatural language processing for information retrieval
281 Citations1996David Lewis, Karen Spärck Jones
The paper considers the new opportunities and challenges presented by the user’s ability to search full text directly (rather than e.g. titles and abstracts), and suggests appropriate approaches to doing this, with a focus on the potential role of natural language processing.
Fast and intuitive clustering of web documents
267 Citations1997Oren Zamir, Oren Etzioni +2 more
It is shown that word-intersection clustering produces superior clusters and does so faster than standard techniques, and that the O(n log n) time phrase-intersections clustering method produces comparable clusters anddoes so more than two orders of magnitude faster than all methods tested.
Evaluating text categorization
234 Citations1991David Lewis
The VLDB JournalScalable feature selection, classification and signature generation for organizing large text databases into hierarchical topic taxonomies
230 Citations1998Soumen Chakrabarti, Byron Dom +2 more
An automatic system that starts with a small sample of the corpus in which topics have been assigned by hand, and then updates the database with new documents as the corpus grows, assigning topics to these new documents with high speed and accuracy is described.
Communications of the ACMAutomatic structuring and retrieval of large text files
224 Citations1994Gerard Salton, James Allan +1 more
An alternative approach is introduced which uses the document collections themselves as a basis for the text analysis, together with sophisticated text matching operations carried out at several levels of detail.
An efficient algorithm for the incremental updation of association rules in large databases
218 Citations1997Shiby Thomas, Sreenath Bodagala +2 more
This paper proposes an incremental updating technique based on negative borders, for the maintenance of association rules when new transaction data is added to or deleted from a transaction database.
Clustering transactions using large items
217 Citations1999Ke Wang, Xu Chu +1 more
This work proposes the notion of large items, i.e., items contained in some minimum fraction of transactions in a cluster, to measure the similarity of a cluster of transactions.
Discovering trends in text databases
207 Citations1997Brian Lent, Rakesh Agrawal +1 more
This work addresses the problem of discovering trends in text databases by defining a trend, a specific subsequence of the history of a phrase that satisfies the users’ query over the histories.
ACM Transactions on Information SystemsInformation extraction as a basis for high-precision text classification
206 Citations1994Ellen Riloff, Wendy G. Lehnert
An approach to text classification that represents a compromise between traditional word-based techniques and in-depth natural language processing and an automated method for empirically deriving appropriate threshold values is described.
University of Minnesota Digital Conservancy (University of Minnesota)Clustering Based on Association Rule Hypergraphs
176 Citations1997Eui-Hong Han, George Karypis +2 more
Query flocks
159 Citations1998Dick Tsur, Jeffrey D. Ullman +5 more
The general idea, called “query flocks,” is a generate-and-test model for data-mining problems and is shown how the idea can be used either in a general-purpose mining system or in a next generation of conventional query optimizers.
A method for word sense disambiguation of unrestricted text
145 Citations1999Rada Mihalcea, Dan Moldovan
This paper presents a method that attempts to disambiguate all the nouns, verbs, adverbs and adjectives in a text, using the senses provided in WordNet, and reports an average accuracy of 80% for the first ranked sense, and 91%.
Mixed-initiative development of language processing systems
141 Citations1997David Day, John Aberdeen +4 more
A new set of integrated tools, collectively called the Alembic Workbench, that uses a mixed-initiative approach to "bootstrapping" the manual tagging process, with the goal of reducing the overhead associated with corpus development.
An investigation of linguistic features and clustering algorithms for topical document clustering
130 Citations2000Vasileios Hatzivassiloglou, Luis Gravano +1 more
A statistical model for combining similarity information from multiple sources is described and applied to DARPA's Topic Detection and Tracking phase 2 (TDT2) data, alleviating the need for extensive search in order to determine optimal weights for combining input features.
A WordNet-based algorithm for word sense disambiguation
101 Citations1995Xiaobin Li, Stan Śzpakowicz +1 more
An algorithm is presented for automatic word sense disambiguation based on lexical knowledge contained in WordNet and on the results of surface-syntactic analysis that results indicate that even on a relatively small text the proposed method produces correct noun meaning more than 72% of the time.
Topic Detection and Tracking using idf-Weighted Cosine Coefficient
97 Citations1999Johannes Schultz
This research achieved very competitive results for tracking using a very simple method of feature selection, without word stemming and without a score normalization scheme, based on the well-known idf -weighted cosine coefficient similarity metric.
Lecture notes in computer scienceLink Based Clustering of Web Search Results
72 Citations2001Yitong Wang, Masaru Kitsuregawa
A new approach to cluster search results returned from Web search engine using link analysis based on common links shared by pages using co-citation and coupling analysis, which is based on document clustering algorithms in IR.
Knowledge Discovery and Data MiningMaximal association rules: a new tool for mining for keyword co-occurrences in document collections
65 Citations1997Ronen Feldman, Yonatan Aumann +3 more
A new method for computing co-occurrence frequencies of the various keywords labeling the documents based on computing maximal association rules is described, which is useful for efficiently finding association rules that include negated attributes.
Communications of the ACMMachine learning of event segmentation for news on demand
62 Citations2000Stanley Boykin, Andrew Merlino
There are many different types of multimedia videos found in the world today—consider home videos, surveillance camera videos, television broadcasts as general categories; commercial users, government personnel, and home consumers all have specific requirements to search these videos for topics and/or events.
Journal of Intelligent Information SystemsExploiting Background Information in Knowledge Discovery from Text
55 Citations1997Ronen Feldman, Haym Hirsh
FACT takes a query-centered view of knowledgediscovery, in which a discovery request is viewed as a query over the implicit set of possible results supported by a collection of documents, and background knowledge is used to specify constraints on the desired results of this query process.
International Journal of Intelligent SystemsLanguage-oriented information retrieval
43 Citations1989David Lewis, W. Bruce Croft +1 more
The view is taken that IR is an inference task, and that natural language processing (NLP) techniques can produce text representations that enable more accurate inferences about document content.
Efficient mining of association rules in text databases
41 Citations1999John D. Holt, Soon M. Chung
Two new algorithms for mining association rules between words in text databases are proposed, named Multipass-Apriori (M-A Priori) and Multipassing-DHP (M -DHP).
Generating association rules from semi-structured documents using an extended concept hierarchy
34 Citations1997Lisa Singh, Peter Scheuermann +1 more
An approach for generatirrg association rules that relates structured data values fo concepts extracted from unstructured data and an implementation of an extended concept hierarchy to maintain parent, child, and sibling relaGonships between concepts are suggested.
Lecture notes in computer scienceMining in the phrasal frontier
25 Citations1997Helena Ahonen-Myka, Oskari Heinonen +2 more
The first experiments in applying general methods of data mining to discovering phrases and co-occurring terms show that data mining methods — with appropriate preprocessing — can be used in text processing, and that by shifting the focus the process can beused to obtain results for various purposes.
GeoNODE: An End-to-End System from Research Components
4 Citations2001Chris Clifton, John Griffith +1 more
The GeoNODE system is demonstrated, the lessons learned are discussed, issues in infrastructure “awareness” that affect future research projects, and benefits to research from participating in this project are discussed.