Data-intensive analytics for predictive modeling
IBM Journal of Research and DevelopmentPublished 1 January 2003
Chid Apte, S. J. Hong, Ramesh Natarajan, Edwin Pednault, Fateh A. Tipu, Sabine Weiß
Citations35
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An overview of the Data Abstraction Research Group's major technical accomplishments is presented, which include advances in methods for feature analysis, rule-based pattern discovery, and probabilistic modeling, and novel solutions for insurance risk management, targeted marketing, and text mining.
Abstract
The Data Abstraction Research Group was formed in the early 1990s, to bring focus to the work of the Mathematical Sciences Department in the emerging area of knowledge discovery and data mining (KD & DM). Most activities in this group have been performed in the technical area of predictive modeling, roughly at the intersection of machine learning, statistical modeling, and database technology. There has been a major emphasis on using business and industrial problems to motivate the research agenda. Major accomplishments include advances in methods for feature analysis, rule-based pattern discovery, and probabilistic modeling, and novel solutions for insurance risk management, targeted marketing, and text mining. This paper presents an overview of the group's major technical accomplishments.
Keywords
Computer Science
IEEE Transactions on Information TheoryNearest neighbor pattern classification
16,134 Citations1967Thomas M. Cover, Peter E. Hart
The nearest neighbor decision rule assigns to an unclassified sample point the classification of the nearest of a set of previously classified points, so it may be said that half the classification information in an infinite sample set is contained in the nearest neighbor.
Experiments with a new boosting algorithm
7,585 Citations1996Yoav Freund, Robert E. Schapire
Neural NetworksStacked generalization
7,478 Citations1992David H. Wolpert
The conclusion is that for almost any real-world generalization problem one should use some version of stacked generalization to minimize the generalization error rate.
Cambridge University Press eBooksPattern Recognition and Neural Networks
6,468 Citations1996B. D. Ripley
Communications of the ACMUsing collaborative filtering to weave an information tapestry
4,040 Citations1992David Theo Goldberg, David M. Nichols +2 more
Tapestry is intended to handle any incoming stream of electronic documents and serves both as a mail filter and repository; its components are the indexer, document store, annotation store, filterer, little box, remailer, appraiser and reader/browser.
International Encyclopedia of Statistical ScienceAnalysis of Variance
1,429 Citations1987Gudmund R. Iversen
ACM Transactions on Information SystemsAutomated learning of decision rules for text categorization
867 Citations1994Chidanand Apté, Fred J. Damerau +1 more
It is shown that machine-generated decision rules appear comparable to human performance, while using the identical rule-based representation, and compared with other machine-learning techniques.
IEEE Intelligent Systems and their ApplicationsMaximizing text-mining performance
211 Citations1999Sabine Weiß, Chid Apte +5 more
IEEE Transactions on Knowledge and Data EngineeringUse of contextual information for feature ranking and discretization
129 Citations1997Se June Hong
A new approach to classification rules or decision trees from examples by finding each feature's "obligation" to the class discrimination in the context of other features, which is a powerful alternative to the traditional methods.
Active learning using adaptive resampling
91 Citations2000Vijay S. Iyengar, Chidanand Apté +1 more
An active learning method is presented that uses adaptive resampling in a natural way to signi cantly reduce the size of the required labeled set and generates a classi cation model that achieves the high accuracies possible with current adaptive Resampling methods.
Lightweight Rule Induction
65 Citations2000Sholom M. Weiss, Nitin Indurkhya
Experimental results on large benchmark datasets demonstrate that predictive performance can rival the best reported results in the literature.
Segmentation-based modeling for advanced targeted marketing
42 Citations2001Chid Apte, Eric Bibelnieks +5 more
An evaluation of ATM-SE's modeling capabilities using data from Fingerhut's catalog mailings is presented, demonstrating the capabilities of the IBM Advanced Targeted Marketing-Single EventsTM (IBM ATM-SETM) solution.
IEEE Transactions on Knowledge and Data EngineeringR-MINI: An iterative approach for generating minimal rules from examples
40 Citations1997Se June Hong
R-MINI is presented, an adaptation of a well-established heuristic-switching-function-minimization technique, MINI that reduces the number of rules by repeated application of generalization and specialization operations to the rule set while maintaining completeness and consistency.
IEEE Intelligent Systems and their ApplicationsProbabilistic estimation-based data mining for discovering insurance risks
36 Citations1999Chid Apte, Emiliano Grossman +4 more
IBM's underwriting profitability analysis application mines property and casualty insurance policy and claims data to construct predictive models for insurance risks and UPA uses the ProbE data-mining kernel to discover risk-characterization rules by analyzing large, noisy data sets.
Knowledge Discovery and Data MiningPredicting equity returns from securities data with minimal rule generation
29 Citations1994Chidanand Apté, Se June Hong
It is demonstrated that the domain can be effectively modeled by classification rules induced from available historical data for the purpose of making gainful predictions for equity investments, and that new techniques developed at IBM Research, including minimal rule generation (R-MINI) and contextual feature analysis, are robust enough to consistently extract useful information from noisy domains such as financial markets.
Solving regression problems with rule-based ensemble classifiers
27 Citations2001Nitin Indurkhya, Sholom M. Weiss
A lightweight learning method that induces an ensemble of decision-rule solutions for regression problems that discretizes the variable by k-means clustering and solves the resultant classification problem.
Predicting equity returns from securities data
27 Citations1996Chidanand Apté, Se June Hong
The rationale for the minimal rule generation technique, the motivation for the use of contextual information in analyzing features, and the results of correlations and simulated managed investment based on classi cation rules generated by R-MINI are introduced.
IBM Systems JournalA probabilistic estimation framework for predictive modeling analytics
19 Citations2002Chid Apte, Ramesh Natarajan +2 more
IEEE Intelligent Systems and their ApplicationsLightweight document matching for help-desk applications
18 Citations2000S.M. Weiss, Brian White +2 more
A completely automated Java based document matcher that accepts an unlimited-length textural structure as input and employs a fast matching algorithm to produce, like a search engine, a ranked list of relevant documents.
Lecture notes in computer scienceLightweight Collaborative Filtering Method for Binary-Encoded Data
18 Citations2001Sholom M. Weiss, Nitin Indurkhya
A lightweight method for collaborative filtering is described that processes binary encoded data for compactly storing data, computing similarities between new and stored records, and making recommendations tailored to an individual.
IBM Systems JournalAutomated generation of model cases for help-desk applications
8 Citations2002S. M. Weiss, Chid Apte
Lecture notes in computer scienceA New Approach For Item Choice Recommendations
4 Citations2001Se June Hong, Ramesh Natarajan +1 more
A statistical model is developed to compute the probability of a new item becoming the next choice given the current partial basket, and make a recommendation list based on a ranking of this probability.