Data Cleansing: A Prelude to Knowledge Discovery
Published 1 January 2009
Jonathan I. Maletic, Andrian Marcus
Citations45
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This chapter analyzes the problem of data cleansing and the identification of potential errors in data sets and presents a set of general methods that can be used to address the problem.
Abstract
This chapter analyzes the problem of data cleansing and the identification of potential errors in data sets. The differing views of data cleansing are surveyed and reviewed and a brief overview of existing data cleansing tools is given. A general framework of the data cleansing process is presented as well as a set of general methods that can be used to address the problem. The applicable methods include statistical outlier detection, pattern matching,clustering, and Data Mining techniques. The experimental results of applying these methods to a real world data set are also given. Finally, research directions necessary to further address the data cleansing problem are discussed.
Keywords
Computer ScienceDecision Sciences
Mining association rules between sets of items in large databases
14,720 Citations1993Rakesh Agrawal, Tomasz Imieliński +1 more
An efficient algorithm is presented that generates all significant association rules between items in the database of customer transactions and incorporates buffer management and novel estimation and pruning techniques.
BiometricsApplied Multivariate Statistical Analysis.
11,424 Citations1988Andrea Johnson, Dean W. Wichern
Choice Reviews OnlineFinding groups in data: an introduction to cluster analysis
10,614 Citations1991
Journal of Management Information SystemsBeyond Accuracy: What Data Quality Means to Data Consumers
4,330 Citations1996Richard Y. Wang, Diane M. Strong
Using this framework, IS managers were able to better understand and meet their data consumers' data quality needs and this research provides a basis for future studies that measure data quality along the dimensions of this framework.
Data Mining and Knowledge Discovery Handbook
2,202 Citations2010Oded Maimon, Lior Rokach
Information Visualization in Data Mining and Knowledge DiscoveryMagnetic Bubble TechnologyStatistical Data AnalyticsUrban InformaticsAdvances in Knowledge Discovery and Data Mining data mining and knowledge discovery technologies.
ACM SIGMOD RecordEfficient algorithms for mining outliers from large data sets
1,871 Citations2000Sridhar Ramaswamy, Rajeev Rastogi +1 more
Journal of the Royal Statistical Society Series A (Statistics in Society)Outliers in Statistical Data.
1,698 Citations1995Anthony C. Atkinson, V. Barnett +1 more
Outliers in Statistical Data, 3rd edition by V. Barnett and T. Lewis.
Algorithms for Mining Distance-Based Outliers in Large Datasets
1,530 Citations1998Edwin M. Knorr, Raymond T. Ng
This paper provides formal and empirical evidence showing the usefulness of DB-outliers and presents two simple algorithms for computing such outliers, both having a complexity of O(k N’), k being the dimensionality and N being the number of objects in the dataset.
Mining quantitative association rules in large relational tables
1,484 Citations1996Ramakrishnan Srikant, Rakesh Agrawal
This work deals with quantitative attributes by fine-partitioning the values of the attribute and then combining adjacent partitions as necessary and introduces measures of partial completeness which quantify the information lost due to partitioning.
Communications of the ACMData quality in context
1,232 Citations1997Diane M. Strong, Yang W. Lee +1 more
A new study reveals businesses are defining data quality with the consumer in mind, and within this larger context of information systems, data is collected from multiple data sources and stored in databases.
The VLDB JournalDistance-based outliers: algorithms and applications
1,196 Citations2000Edwin M. Knorr, Raymond T. Ng +1 more
Outlier detection can be done efficiently for large datasets, and for k-dimensional datasets with large values of k, and it is shown that outlier detection is a meaningful and important knowledge discovery task.
Efficient algorithms for mining outliers from large data sets
1,150 Citations2000Sridhar Ramaswamy, Rajeev Rastogi +1 more
A novel formulation for distance-based outliers that is based on the distance of a point from its kth nearest neighbor is proposed and the top n points in this ranking are declared to be outliers.
The Computer JournalA Survey of Recent Advances in Hierarchical Clustering Algorithms
1,047 Citations1983Fionn Murtagh
A general framework for hierarchical, agglomerative clustering algorithms is discussed here, which opens up the prospect of much improvement on current, widely-used algorithms.
Outlier detection for high dimensional data
1,034 Citations2001Charų C. Aggarwal, Philip S. Yu
New techniques for outlier detection are discussed which find the outliers by studying the behavior of projections from the data set by studying the behavior of projections from the data set.
Clustering Methods
878 Citations2006Lior Rokach, Oded Maimon
This paper examines MDS, that attempted to find a coordinate system to map items into based on their (dis)similarity, but what happens if you don’t know the identity, but still want to come up with bottom-up categories?
Data Mining and Knowledge DiscoveryReal-world Data is Dirty: Data Cleansing and The Merge/Purge Problem
813 Citations1998Mauricio A. Hernández, Salvatore J. Stolfo
This paper develops a system for accomplishing this Data Cleansing task and demonstrates its use for cleansing lists of names of potential customers in a direct marketing-type application and reports on the successful implementation for a real-world database that conclusively validates results previously achieved for statistically generated data.
Mining association rules with item constraints
769 Citations1997Ramakrishnan Srikant, Quoc Vu +1 more
This work considers the problem of integrating constraints that are Boolean expressions over the presence or absence of items into the association discovery algorithm and presents three integrated algorithms for mining association rules with item constraints and discusses their tradeoffs.
Communications of the ACMThe impact of poor data quality on the typical enterprise
721 Citations1998Thomas C. Redman
The typical executive is already besieged by too many problems, low customer satisfaction, high costs, a data warehouse project that is late, and so forth, so this article aims to increase awareness by providing a summary of the impacts of poor data quality on a typical enterprise.
Lecture notes in computer scienceOutlier Detection Using Replicator Neural Networks
711 Citations2002Simon Hawkins, Hongxing He +2 more
Replicator neural networks (RNNs) are used to provide a measure of the outlyingness of data records and the effectiveness of the RNNs for outlier detection is demonstrated on two publicly available databases.
Coding and Information Theory
681 Citations1980R. W. Hamming
IEEE Transactions on Knowledge and Data EngineeringA framework for analysis of data quality research
678 Citations1995R.Y. Wang, Veda C. Storey +1 more
Using an analogy between product manufacturing and data manufacturing, this paper develops a framework for analyzing data quality research, and uses it as the basis for organizing the data quality literature.
Potter's Wheel: An Interactive Data Cleaning System
491 Citations2001Vijayshankar Raman, Joseph L. Hellerstein
Potter’s Wheel is presented, an interactive data cleaning system that tightly integrates transformation and discrepancy detection, and users can gradually build a transformation as discrepancies are found, and clean the data without writing complex programs or enduring long delays.
Robust and efficient fuzzy match for online data cleaning
456 Citations2003Surajit Chaudhuri, Kris Ganjam +2 more
A new similarity function is proposed which overcomes limitations of commonly used similarity functions, and an efficient fuzzy match algorithm is developed which can effectively clean an incoming tuple if it fails to match exactly with any tuple in the reference relation.
ACM SIGMOD RecordOutlier detection for high dimensional data
436 Citations2001Charų C. Aggarwal, Philip S. Yu
Data Quality
343 Citations2000Richard Y. Wang, Mostapha Ziad +1 more
IEEE Intelligent Systems and their ApplicationsLearning approaches for detecting and tracking news events
338 Citations1999Yi Yang, Jaime Carbonell +4 more
The authors extend existing supervised-learning and unsupervised-clustering algorithms to allow document classification based on the information content and temporal aspects of news events to be classified using manually segmented documents.
Data Mining and Knowledge DiscoveryA Taxonomy of Dirty Data
300 Citations2003Won Bae Kim, Byoungju Choi +3 more
A comprehensive classification of dirty data is developed for use as a framework for understanding how dirty data arise, manifest themselves, and may be cleansed to ensure proper construction of data warehouses and accurate data analysis.
Communications of the ACMEnhancing data quality in data warehouse environments
268 Citations1999Donald P. Ballou, Giri Kumar Tayi
A conceptual framework for enhancing data quality in data warehouse environments is offered and the factors that should be considered, such as the current level of data quality, the levels of quality needed by the relevant decision processes, and the potential benefits of projects designed to enhance data quality are explored.
Information Processing & ManagementThe notion of data and its quality dimensions
256 Citations1994Chris Fox, Anany Levitin +1 more
The most important dimensions of data quality: accuracy, completeness, consistency and currentness are defined and several related dimensions are discussed and defined and discussed in detail.
Discovering informative patterns and data cleaning
233 Citations1996Isabelle Guyon, Nada Matic +1 more
A method for discovering informative patterns from data that can be reduced to only a few representative data entries and an attractive candidate for new applications in knowledge discovery is presented.
Knowledge and Information SystemsFindOut : Finding Outliers in Very Large Datasets
200 Citations2002Dantong Yu, Gholamhosein Sheikholeslami +1 more
A novel deviation (or outlier) detection approach, termed FindOut, based on wavelet transform is introduced, which can successfully identify outliers from large datasets.
IntelliClean
116 Citations2000Mong Li Lee, Tok Wang Ling +1 more
A new method to compute transitive closure under uncertainty which handles the merging of groups of inexact duplicate records and a generic knowledge-based frame- work for effective data cleaning that implements existing cleaning strategies and more.
ACM SIGKDD Explorations NewsletterSummary from the KDD-03 panel
86 Citations2003Usama M. Fayyad, Gregory Piatetsky-Shapiro +1 more
This panel was an attempt to address the possible future directions for Data Mining and KDD.
FigshareRatio Rules: A New Paradigm for Fast, Quantifiable Data Mining
52 Citations2018Flip Korn, Alexandros Labrinidis +2 more
Lecture notes in computer scienceImproving Supervised Learning by Feature Decomposition
45 Citations2002Oded Maimon, Lior Rokach
The results achieved in the empirical comparison testing with well-known learning algorithms indicate the superiority of the feature decomposition approach in learning tasks that contains high number of features and moderate numbers of tuples.
Information and Software TechnologyA model of the data (life) cycles with application to quality
44 Citations1993Anany Levitin, Thomas C. Redman
A new model of the data life-cycle is presented, needed to clarify activities involving data, from its creation through use, and to establish the relationships of these activities to one another.
Robust and efficient fuzzy match for online data cleaning
43 Citations2003Surajit Chaudhuri, Kris Ganjam +2 more
Ordinal association rules for error identification in data sets
39 Citations2001Andrian Marcus, Jonathan I. Maletic +1 more
A method that finds these rules and identifies potential errors in data is proposed, and one use for ordinal rules is to identify possible errors inData.
Information and Software TechnologyIntegrity analysis
24 Citations1988MI Svanks
Integrity analysis principles are applicable to quality assurance, EDP audit, systems design, operations, and management, as well as to the design of data dictionaries, database management systems, and generalized software for data inspection.
A fast filtering scheme for large database cleansing
22 Citations2002Sam Yuan Sung, Zhao Li +1 more
This paper proposes a simple and fast comparison method, TI-Similarity, which reduces the time for each comparison, and proposes a new detection method, RAR, to further reduce the number of comparisons.
Data quality through knowledge engineering
22 Citations2003Tamraparni Dasu, Gregg Vesonder +1 more
The approach is to impement business rules as constraints on data in a classical expert system formalism sometimes called production rules, which works by allowing good data to pass through a system of constraints unchecked.
Using Recon for data cleaning
21 Citations1995Evangelos Simoudis, Brian Livezey +1 more
Recon incorporates several data mining modules into a single, uniform framework: data visualization, deductive databases, and rule induction, which yields superior error detection over the application of any single data mining technique.
Lecture notes in computer scienceA New Efficient Data Cleansing Method
12 Citations2002Zhao Li, Sung Sam Yuan +2 more
This paper proposes a new comparison method, LCSS, based on the longest common subsequence, and shows that it possesses the desired properties and proposes two new detection methods, SNM-IN and SNM -INOUT, which are variances of the popular detection method SNM.
ACM SIGKDD Explorations NewsletterA data cleaning solution by Perl scripts for the KDD Cup 2003 task 2
7 Citations2003Martine Cadot, Joseph Di Martino
The solution is based on a data cleaning methodology using Perl scripts that contain regular expression for automatically extracting relevant information from the 35472 LaTeX texts and has permitted to obtain 144,087 associations.