Interfacing knowledge discovery algorithms to large database management systems

Information and Software TechnologyPublished 1 June 1999

Simon Lavington, Neil Dewhurst, Edwin G. Wilkins, Alex A. Freitas

DOI10.1016/s0950-5849(99)00024-5

Source URL

Citations15

SJR quartileQ1

SJR score1.04

SNIP2.15

TL;DR

A family of generic, set-based, primitive operations for Knowledge Discovery in Databases (KDD), drawn from paradigms such as Bayesian classifiers, Rule-Induction/Decision Tree algorithms, Instance-Based Learning methods, and Genetic Programming are presented.

Abstract

The efficient mining of large, commercially credible, databases requires a solution to at least two problems: (a) better integration between existing Knowledge Discovery algorithms and popular DBMS; (b) ability to exploit opportunities for computational speedup such as data parallelism. Both problems need to be addressed in a generic manner, since the stated requirements of end-users cover a range of data mining paradigms, DBMS, and (parallel) platforms. In this paper we present a family of generic, set-based, primitive operations for Knowledge Discovery in Databases (KDD). We show how a number of well-known KDD classification metrics, drawn from paradigms such as Bayesian classifiers, Rule-Induction/Decision Tree algorithms, Instance-Based Learning methods, and Genetic Programming, can all be computed via our generic primitives. We then show how these primitives may be mapped into SQL and, where appropriate, optimised for good performance in respect of practical factors such as client-server communication overheads. We demonstrate how our primitives can support C4.5, a widely-used rule induction system. Performance evaluation figures are presented for commercially available parallel platforms, such as the IBM SP/2.

Keywords

Computer Science

C4.5: Programs for Machine Learning

23,665 Citations1992J. R. Quinlan

A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.

Medical Entomology and ZoologyGenetic Programming: On the Programming of Computers by Means of Natural Selection

13,257 Citations1992John R. Koza

This book discusses the evolution of architecture, primitive functions, terminals, sufficiency, and closure, and the role of representation and the lens effect in genetic programming.

Data Mining and Knowledge DiscoveryData Cube: A Relational Aggregation Operator Generalizing Group-By, Cross-Tab, and Sub-Totals

2,134 Citations1997Jim Gray, Surajit Chaudhuri +6 more

This paper explains the cube and roll-up operators, shows how they fit in SQL, explains how users can define new aggregatefunctions for cubes, and discusses efficient techniques to compute the cube.

Implementing data cubes efficiently

1,166 Citations1996Venky Harinarayan, Anand Rajaraman +1 more

This paper investigates the issue of which cells (views) to materialize when it is too expensive to materialized all views, and presents greedy algorithms that work off this lattice and determine a good set of views to materializing.

Computer systems that learn: classification and prediction methods from statistics, neural nets, machine learning, and expert systems

906 Citations1991Sholom M. Weiss, Casimir A. Kulikowski

This book discusses how to Estimate the True Performance of a Learning System, and the Importance of Unbiased Error Rate Estimation, and Machine Learning: Easily Understood Decision Rules.

Lecture notes in computer scienceSLIQ: A fast scalable classifier for data mining

753 Citations1996Manish Mehta, Rakesh Agrawal +1 more

Issues in building a scalable classifier are discussed and the design of SLIQ, a new classifier that uses a novel pre-sorting technique in the tree-growth phase to enable classification of disk-resident datasets is presented.

The MIT Press eBooksOn the Computation of Multidimensional Aggregates

480 Citations1999Sameet Agarwal, Rakesh Agarwal +5 more

This paper presents fast algorithms for computing a collection of group bys using sort-based and hashbased grouping methods with several .optimizations, like combining common operations across multiple groupbys, caching, and using pre-computed group-by8 for computing other groupbys.

Lecture notes in computer scienceOn changing continuous attributes into ordered discrete attributes

432 Citations1991Jason Catlett

This paper describes how continuous attributes can be converted economically into ordered discrete attributes before being given to the learning system, and suggests this change of representation does not often result in a significant loss of accuracy, but offers large reductions in learning time.

Mining Very Large Databases with Parallel Processing

205 Citations2000Alex A. Freitas, Simon Lavington

Mining Very Large Databases with Parallel Processing addresses the problem of large-scale data mining, describing advances in the integration of three computer science areas, namely: "intelligent" (machine learning-based) data mining techniques; relational databases and parallel processing.

Elsevier eBooksA Comparative Evaluation of Voting and Meta-learning on Partitioned Data

148 Citations1995Philip K. Chan, Salvatore J. Stolfo

This paper evaluates different techniques for learning from partitioned data and the meta-learning approach is empirically compared with techniques in the literature that aim to combine multiple evidence to arrive at one prediction.