FERNN: An Algorithm for Fast Extraction of Rules from Neural Networks
Applied IntelligencePublished 1 January 2000
Rudy Setiono, Wee Kheng Leow
Citations143
SJR quartileQ2
SJR score0.93
SNIP1.21
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Experimental results show that the size and the predictive accuracy of the tree generated are comparable to those extracted by another method which prunes and retrains the network.
Abstract
Before symbolic rules are extracted from a trained neural network, the network is usually pruned so as to obtain more concise rules. Typical pruning algorithms require retraining the network which incurs additional cost. This paper presents FERNN, a fast method for extracting rules from trained neural networks without network retraining. Given a fully connected trained feedforward network with a single hidden layer, FERNN first identifies the relevant hidden units by computing their information gains. For each relevant hidden unit, its activation values is divided into two subintervals such that the information gain is maximized. FERNN finds the set of relevant network connections from the input units to this hidden unit by checking the magnitudes of their weights. The connections with large weights are identified as relevant. Finally, FERNN generates rules that distinguish the two subintervals of the hidden activation values in terms of the network inputs. Experimental results show that the size and the predictive accuracy of the tree generated are comparable to those extracted by another method which prunes and retrains the network.
Keywords
Computer Science
C4.5: Programs for Machine Learning
23,665 Citations1992J. R. Quinlan
A complete guide to the C4.5 system as implemented in C for the UNIX environment, which starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting.
Society for Industrial and Applied Mathematics eBooksNumerical Methods for Unconstrained Optimization and Nonlinear Equations
7,651 Citations1996J. E. Dennis, Robert B. Schnabel
The Caltech Institute Archives (California Institute of Technology)Second order derivatives for network pruning: Optimal Brain Surgeon
1,489 Citations1992Babak Hassibi, David G. Stork
Of OBS, Optimal Brain Damage, and magnitude-based methods, only OBS deletes the correct weights from a trained XOR network in every case, and thus yields better generalization on test data.
Knowledge-Based SystemsSurvey and critique of techniques for extracting rules from trained artificial neural networks
1,230 Citations1995Robert Andrews, Joachim Diederich +1 more
This survey focuses on mechanisms, procedures, and algorithms designed to insert knowledge into ANNs, extract rules from trained ANNs (rule extraction), and utilise ANNs to refine existing rule bases (rule refinement).
Neural ComputationFirst- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method
1,193 Citations1992Roberto Battiti
First- and second-order optimization methods for learning in feedforward neural networks are reviewed to illustrate the main characteristics of the different methods and their mutual relations.
Communications of the ACMConnectionist expert systems
665 Citations1988Stephen I. Gallant
Connectionist networks can be used as expert system knowledge bases and can be constructed from training examples by machine learning techniques, giving a way to automate the generation of expert systems for classification problems.
Machine LearningExtracting Refined Rules from Knowledge-Based Neural Networks
587 Citations1993Geoffrey G. Towell, Jude Shavlik
This article proposes and empirically evaluates a method for the final, and possibly most difficult, step of the refinement of existing knowledge and demonstrates that neural networks can be used to effectively refine symbolic knowledge.
Machine LearningExtracting refined rules from knowledge-based neural networks
490 Citations1993Geoffrey G. Towell, Jude Shavlik
This article proposes and empirically evaluates a method for the final, and possibly most difficult, step of the refinement of existing knowledge and demonstrates that neural networks can be used to effectively refine symbolic knowledge.
Neural NetworksImproving the convergence of the back-propagation algorithm
417 Citations1992Arjen van Ooyen, Bernard Nienhuis
The modified back-propagation method consists of a simple change in the total error-of-performance function that is to be minimized by the algorithm, and the final approach to the desired response function is accelerated by an amount that can be predicted analytically.
IEEE Transactions on Neural NetworksAn iterative pruning algorithm for feedforward neural networks
299 Citations1997Giovanna Castellano, Anna Maria Fanelli +1 more
A new pruning method is developed, based on the idea of iteratively eliminating units and adjusting the remaining weights in such a way that the network performance does not worsen over the entire training set.
ScholarlyCommons (University of Pennsylvania)Learning Algorithms for Connectionist Networks: Applied Gradient Methods of Nonlinear Optimization
239 Citations1988Raymond L. Watrous
It is shown that in plateau regions of relatively constant gradient, the momentum term acts to increase the step size by a factor of 1/1-μ, where μ is the momentumTerm, and in valley regions with steep sides,The momentum constant acts to focus the search direction toward the local minimum by averaging oscillations in the gradient.
Neural ComputationA Penalty-Function Approach for Pruning Feedforward Neural Networks
192 Citations1997Rudy Setiono
ComputerSymbolic representation of neural networks
188 Citations1996Rudy Setiono, Huan Liu
This algorithm uses symbolic rules to represent the network decision process and extracts rules from a neural network with discretized hidden unit activation values to preserve network accuracy and explain the prediction process.
Extracting Rules from Artificial Neural Networks with Distributed Representations
172 Citations1994Sebastian Thrun
National Conference on Artificial IntelligenceRule learning by searching on adapted nets
161 Citations1991LiMin Fu
The KT algonhm is a novel algonthm for generating rules from an adapted net efficiently that is able to deal with both single-layer and multi-layer networks, and can learn both confirming and disconfirming rules.
Neural ComputationExtracting Rules from Neural Networks by Pruning and Hidden-Unit Splitting
155 Citations1997Rudy Setiono
An algorithm for extracting rules from a standard three-layer feedforward neural network is proposed that can extract reasonably compact rule sets that have high predictive accuracy rates and is shown to work using real-world data arising from molecular biology and signal processing.
Connection ScienceRule Extraction: From Neural Architecture to Symbolic Representation
153 Citations1995Gail A. Carpenter, Ah‐Hwee Tan
This paper shows how knowledge, in the form of fuzzy rules, can be derived from a supervised learning neural network called fuzzy ARTMAP.
NeurocomputingNeuroLinear: From neural networks to oblique decision rules
124 Citations1997Rudy Setiono, Huan Liu
NeuroLinear is a system for extracting oblique decision rules from neural networks that have been trained for classification of patterns that is effective in extracting compact and comprehensible rules with high predictive accuracy from Neural networks.
Elsevier eBooksAUTOMATED KNOWLEDGE ACQUISITION
117 Citations1991
This paper discusses theoretical considerations the extraction of rules and concepts using a single-layered Hebbian neural network BRAINNE, and automated knowledge acquisition using multi-layering neural network BRINNE in the real world.
NeurocomputingA simple and effective method for removal of hidden units and weights
77 Citations1994Masafumi Hagiwara
According to computer simulations using the mirror symmetry problem, the Weights power method has shown the best performance with respect to size reduction, generalization performance, and the amount of computation required.
X2R: A fast rule generator
54 Citations1995Huan Liu, Sun Teck Tan
X2R is introduced, a simple and fast algorithm that can be applied to both numeric and discrete data, and generate rules from datasets, like season-classification and golf-playing that contain continuous and/or discrete data.
Lecture notes in computer scienceGlobal data analysis and the fragmentation problem in decision tree induction
29 Citations1997Ricardo Vilalta, Gunnar Blix +1 more
This study investigates an inherent limitation of top-down decision tree induction in which the continuous partitioning of the instance space progressively lessens the statistical support of every partial hypothesis and shows how the fragmentation problem adversely affects predictive accuracy as variation ∇ increases.
Connection ScienceA Neural Network Construction Algorithm which Maximizes the Likelihood Function
23 Citations1995Rudy Setiono
A new method for constructing a feedforward neural network is proposed that starts with a single hidden unit and more units are added to the hidden layer one at a time until a network that is suitable for inference is constructed.
Cambridge University Press eBooksA review of techniques for extracting rules from trained artificial neural networks
5 Citations2001Robert Andrews, Alan Tickle +1 more