Novelty detection: a review—part 1: statistical approaches
Signal ProcessingPublished 17 September 2003
M. Markou, Sameer Singh
Citations1,414
SJR quartileQ1
SJR score0.96
SNIP1.51
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
There are a multitude of applications where novelty detection is extremely important including signal processing, computer vision, pattern recognition, data mining, and robotics.
Abstract
Novelty detection is the identification of new or unknown data or signal that a machine learning system is not aware of during training. Novelty detection is one of the fundamental requirements of a good classification or identification system since sometimes the test data contains information about objects that were not known at the time of training the model. In this paper we provide state-of-the-art review in the area of novelty detection based on statistical approaches. The second part paper details novelty detection using neural networks. As discussed, there are a multitude of applications where novelty detection is extremely important including signal processing, computer vision, pattern recognition, data mining, and robotics.
Keywords
Computer ScienceMedicine
IEEE Transactions on Information TheoryNearest neighbor pattern classification
16,134 Citations1967Thomas M. Cover, Peter E. Hart
The nearest neighbor decision rule assigns to an unclassified sample point the classification of the nearest of a set of previously classified points, so it may be said that half the classification information in an infinite sample set is contained in the nearest neighbor.
Advances in industrial controlPattern Classification
7,884 Citations2000Evan L. Russell, Leo H. Chiang +1 more
Computers & GeosciencesFCM: The fuzzy c-means clustering algorithm
6,572 Citations1984James C. Bezdek, Robert Ehrlich +1 more
A FORTRAN-IV coding of the fuzzy c -means (FCM) clustering program is transmitted, which generates fuzzy partitions and prototypes for any set of numerical data.
Mathematical Proceedings of the Cambridge Philosophical SocietyLimiting forms of the frequency distribution of the largest or smallest member of a sample
3,308 Citations1928Ronald Aylmer Fisher, L. H. C. Tippett
A re-examination of text categorization methods
2,660 Citations1999Yiming Yang, Xin Liu
The results show that SVM, kNN and LLSF signi cantly outperform NNet and NB when the number of positive training instances per category are small, and that all the methods perform comparably when the categories are over 300 instances.
Statistical Pattern Recognition
1,754 Citations2001
Journal of the Royal Statistical Society Series A (Statistics in Society)Outliers in Statistical Data.
1,698 Citations1995Anthony C. Atkinson, V. Barnett +1 more
Outliers in Statistical Data, 3rd edition by V. Barnett and T. Lewis.
Self-nonself discrimination in a computer
1,325 Citations2002Stephanie Forrest, Alan S. Perelson +2 more
A method for change detection which is based on the generation of T cells in the immune system is described, which reveals computational costs of the system and preliminary experiments illustrate how the method might be applied to the problem of computer viruses.
The VLDB JournalDistance-based outliers: algorithms and applications
1,196 Citations2000Edwin M. Knorr, Raymond T. Ng +1 more
Outlier detection can be done efficiently for large datasets, and for k-dimensional datasets with large values of k, and it is shown that outlier detection is a meaningful and important knowledge discovery task.
IEEE Transactions on Information TheoryOn optimum recognition error and reject tradeoff
826 Citations1970Chee Lap Chow
An optimum rejection rule is described and a general relation between the error and reject probabilities and some simple properties of the tradeoff in the optimum recognition system are presented.
A study of retrospective and on-line event detection
731 Citations1998Yiming Yang, Tom Pierce +1 more
This paper applied hierarchical and non-hierarchical document clustering algorithms to a corpus of 15,836 stories, focusing on the exploitation of both content and temporal information, and found the resulting cluster hierarchies highly informative for retrospective detection of previously unidentified events.
Novelty Detection and Neural Network Validation
439 Citations1993Chris Bishop
This paper provides a quantitative procedure for measuring novelty, and its performance is demonstrated using an application involving the monitoring of oil flow in multi-phase pipelines.
A novelty detection approach to classification
393 Citations1995Nathalie Japkowicz, Catherine E. Myers +1 more
A particular Novelty Detection approach to classification that uses a Redundancy Compression and Non-Redundancy Differentiation technique based on the hippocampus, a part of the brain critically involved in learning and memory, is presented.
IEE Proceedings - Vision Image and Signal ProcessingNovelty detection and neural network validation
389 Citations1994Chris Bishop
Pattern RecognitionHost-based intrusion detection using dynamic and static behavioral models
346 Citations2002Dit‐Yan Yeung, Yuxin Ding
This paper adopts an anomaly detection approach by detecting possible intrusions based on program or user profiles built from normal usage data using a scheme that can be justified from the perspective of hypothesis testing.
Novelty detection for the identification of masses in mammograms
342 Citations1995Lionel Tarassenko
An alternative approach is explored in which a description of normality is attempted using the large number of available mammograms which do not show any evidence of mass-like structures to try and identify candidate masses in previously unseen images analysis and interpretation.
Pattern Recognition LettersTwo-phase clustering process for outliers detection
342 Citations2001Miao Jiang, S. S. Tseng +1 more
A two-phase clustering algorithm for outliers detection is proposed, which first modify the traditional k-means algorithm in Phase 1 by using a heuristic “if one new input pattern is far enough away from all clusters' centers, then assign it as a new cluster center”.
IEEE Communications MagazineNetwork intrusion and fault detection: a statistical anomaly approach
255 Citations2002C.N. Manikopoulos, Symeon Papavassiliou
The proposed methodology can reliably detect attacks with traffic anomaly intensity as low as 3-5 percent of the typical background traffic intensity, thus promising to generate an effective early warning.
Parzen-window network intrusion detectors
250 Citations2003Dit-Yan Yeung, Calvin Chow
This work proposes to take a nonparametric density estimation approach based on Parzen-window estimators with Gaussian kernels to build an intrusion detection system using normal data only, which compares favorably with the KDD Cup winner which is based on an ensemble of decision trees with bagged boosting.
Topic-conditioned novelty detection
234 Citations2002Yiming Yang, Jian Zhang +2 more
This paper proposes a new approach which addresses this problem in two stages: using a supervised learning algorithm to classify the on-line document stream into pre-defined broad topic categories, and performing topic-conditioned novelty detection for documents in each topic.
On-line unsupervised outlier detection using finite mixtures with discounting learning algorithms
221 Citations2000Kenji Yamanishi, Jun’ichi Takeuchi +2 more
Knowledge and Information SystemsUsing artificial anomalies to detect unknown and known network intrusions
220 Citations2004W. Fan, Matthew L. Miller +3 more
An algorithm to generate artificial anomalies to coerce the inductive learner into discovering an accurate boundary between known classes (normal connections and known intrusions) and anomalies is proposed.
IEEE Transactions on Systems Science and CyberneticsThe Nearest Neighbor Classification Rule with a Reject Option
214 Citations1970Martin E. Hellman
Here the (k,k?) nearest neighbor rule with a reject option is examined, which looks at the k nearest neighbors and rejects if less than k? of these are from the same class; if k? or more are from one class, a decision is made in favor of that class.
IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews)To reject or not to reject: that is the question-an answer in case of neural classifiers
211 Citations2000Claudio De Stefano, Carlo Sansone +1 more
A method defining a reject option that is applicable to a given 0-reject classifier and a function P characterizing the reject option's adequacy to the domain has been introduced, showing that P can be expressed as a function of /spl sigma/ and the optimal value for /spl Sigma/ is defined as the one which maximizes the function P.
Informal identification of outliers in medical data
209 Citations2000Jorma Laurikkala, Martti Juhola +1 more
The removal of outliers increased the descriptive classification accuracy of discriminant analysis functions and nearest neighbour method, while the predictive ability of these methods reduced somewhat.
IEE Proceedings - Vision Image and Signal ProcessingNovelty detection using extreme value statistics
192 Citations1999Stephen Roberts
It is shown that a more principled approach may be taken using extreme value statistics in the area of novelty detection, and points that lie outside of the range of expected extreme values may be flagged as outliers.
A Linear Programming Approach to Novelty Detection
192 Citations2000Colin Campbell, Kristin P. Bennett
A simpler kernel method for estimating the support based on linear programming that is easy to implement and can learn large datasets rapidly is proposed and demonstrated on medical and fault detection datasets.
Neural ComputationA Probabilistic Resource Allocating Network for Novelty Detection
186 Citations1994Stephen Roberts, Lionel Tarassenko
A robust method for novelty detection is developed, which aims to minimize the number of heuristically chosen thresholds in the novelty decision process by growing a gaussian mixture model to form a representation of a training set of normal system states.
Studies in fuzziness and soft computingStatistical pattern recognition
182 Citations2000Ludmila I. Kuncheva
Pattern recognition is about those problems that are still not algorithmically clear-cut, but which humans easily identify faces, speakers, smells — tasks that are still a challenge for a computer.
Pattern RecognitionReject option with multiple thresholds
167 Citations2000Giorgio Fumera, Fabio Roli +1 more
The formulation of the best error-reject trade-off and the related optimal reject rule was given by Chow and can be obtained using the so-called “reject” option.
Neural ComputationStatistical Independence and Novelty Detection with Information Preserving Nonlinear Maps
140 Citations1996Lucas C. Parra, Gustavo Deco +1 more
This work applies the criterion of minimal mutual information to the real world problem of electrical motor fault detection treated as a novelty detection task, and generalizes to nonlinear transformations by only demanding perfect transmission of information.
IEEE Transactions on Knowledge and Data EngineeringAn approach to novelty detection applied to the classification of image regions
116 Citations2004Sameer K. Singh, M. Markou
The proposed framework for novelty detection evaluates neural networks as adaptive classifiers that are capable of novelty detection and retraining on the basis of newly discovered information and applies this model to the application area of object recognition in video.
Proceedings of the Institution of Mechanical Engineers Part C Journal of Mechanical Engineering ScienceApplications of probability density estimation to the detection of abnormal conditions in engineering
109 Citations1998M.J. Desforges, Philip Jacob +1 more
The combined data compression and PDF estimation approach for novelty detection is applied to data measured from a gearbox with a progressive fault and to radar data corresponding to six military targets, where abnormal situations are clearly identified on the basis of novel data inputs.
IEEE Transactions on Neural NetworksA method for improving classification reliability of multilayer perceptrons
106 Citations1995L.P. Cordella, Claudio De Stefano +2 more
Criteria for evaluating the classification reliability of a neural classifier and for accordingly making a reject option are proposed, and the method has been tested with a neuralclassifier devised for handprinted and multifont printed characters, by using a database of about 300000 samples.
Detection, synthesis and compression in mammographic image analysis with a hierarchical image probability model
102 Citations2002Clay D. Spence, Lucas C. Parra +1 more
A probability model over image spaces that employs a pyramid representation to factor images across scale and a tree-structured set of hidden variables to capture long-range spatial dependencies and is a hierarchical mixture of conditional probabilities, similar to a hidden Markov model on a tree.
A comparison of negative and positive selection algorithms in novel pattern detection
101 Citations2002Dipankar Dasgupta, Fernando Niño
This paper describes a probabilistic method that uses a negative selection scheme (complement pattern space) to detect any changes in the normal behavior of monitored data patterns to detect anomaly detection.
Anomaly detection in multidimensional data using negative selection algorithm
99 Citations2003Dipankar Dasgupta, Nivedita Majumdar
Anomaly detection results with single and multidimensional data sets using the negative selection algorithm developed by Forrest et al. (1994) are reported.
Extreme value statistics for novelty detection in biomedical signal processing
95 Citations2000Stephen Roberts
Extreme-value theory (EVT) and its associated statistics provide a complete and principled approach to the problem of screening data for novel segments which avoids the need for heuristic methods.
Systems & Control LettersPole-assignment robustness in a specified disk
87 Citations1991Jyh‐Horng Chou
Integrated Computer-Aided EngineeringA System for the Analysis of Jet Engine Vibration Data
86 Citations1999A. Nairac, N. Townsend +4 more
A system to extract diagnostic information from jet engine carcass vibration data using neural network techniques, which provides good separation between usual and unusual vibration signatures but, given the small number of examples of normal engines, the resulting representation of normality may be overfitting the training data.
Lecture notes in computer scienceOutlier detection using classifier instability
85 Citations1998David M. J. Tax, Robert P. W. Duin
A new method is presented which is based on the instability of the output of simple classifiers on new objects which detects outlier detection methods in a handwritten digit recognition problem.
Artificial Intelligence in EngineeringOn-line control chart pattern detection and discrimination—a neural network approach
78 Citations1999Ruey‐Shiang Guh, Farbod Zorriassatine +2 more
An artificial neural network based model, which employs a pattern discrimination algorithm to recognise unnatural control chart patterns and is capable of superior ARL performance while the type of the unnatural pattern can also be accurately identified.
Open Systems & Information Dynamics
75 Citations1997Lars Kai Hansen, Christian Liisberg +1 more
It is argued that universality in error-reject tradeoff curves for widely differing algorithms classifying handwritten characters is in fact to be expected for general classification problems and extended to classifiers working from finite samples on a broad, albeit limited, class of problems.
IEEE Transactions on Energy ConversionElliptical novelty grouping for on-line short-turn detection of excited running rotors
67 Citations1999S.E. Guttormsson, Robert J. Marks +2 more
Data description in subspaces
56 Citations2002David M. J. Tax, Robert P. W. Duin
This paper proposes a simple method based on nearest neighbor distances which is capable of finding data boundaries in these low sample sizes and appears to be especially useful when the data is distributed in subspaces.
Pattern Recognition LettersRejection based classifier for face detection
51 Citations2002Michael Elad, Yacov Hel-Or +1 more
This paper proposes a new classifier, called the maximal rejection classifier (MRC), which is based on linear successive rejection operations and an application of detecting faces in images is demonstrated using the MRC.
Choosing an appropriate model for novelty detection
46 Citations1997A. Nairac
For novelty detection, a description of normality is learnt by fitting a model to the set of normal examples, and previously unseen patterns are then tested by comparing their novelty score (as defined by the model) against some threshold.
Lecture notes in computer scienceA Mixture Approach to Novelty Detection Using Training Data with Outliers
41 Citations2001Martin Lauer
This paper explicitly model the existence of outliers in the training data using a widespread outlier distribution, and uses Gaussian mixture models to describe the patterns' distribution.
The use of novelty detection techniques for monitoring high-integrity plant
40 Citations2003Stephen King, Dennis King +4 more
This paper describes the practical implementation of novelty detection schemes that aim to overcome the limitations of data driven approaches to fault detection.
Classification and novelty detection using linear models and a class dependent-elliptical basis function neural network
37 Citations2002T. Brotherton, Travis S. Johnson +1 more
The development and application of a class-dependent elliptical basis function (CD-EBF) neural net developed to solve novelty detection problems and application to passive detection and classification of EM signals is shown.
Novelty detection in jet engines
36 Citations1999Lionel Tarassenko
The principle of novelty detection offers an alternative approach to the problem of fault detection by learning a statistical description of normality and identifying anomalies which have not been seen in the training data.
Artificial Intelligence in MedicineEvIdent™: a functional magnetic resonance image analysis system
35 Citations2001Nicolino J. Pizzi, Rodrigo Vivanco +1 more
EvIdent (EVent IDENTification) is a user-friendly, algorithm-rich, exploratory data analysis software for quickly detecting, investigating, and visualizing novel events in a set of images as they evolve in time and/or frequency.
Pattern RecognitionMulticlassification: reject criteria for the Bayesian combiner
35 Citations1999Pasquale Foggia, Carlo Sansone +2 more
A method for determining the best trade-off between error rate and reject rate for a multi-expert system (MES) using the Bayesian combining rule, based on the estimation of the reliability of each classification act and the evaluation of the convenience of rejecting the input sample when the reliability is under a threshold.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIELong-term stability of normal condition data for novelty detection
31 Citations2000G. Manson, Gareth Pierce +4 more
Modeling text with generalizable Gaussian mixtures
30 Citations2002Lars Kai Hansen, Snævar Sigurðsson +4 more
It is shown that the generalizability of these models depends on the dimensionality of the representation and the sample size, and the relation between supervised and unsupervised learning in the test data is discussed.
Pattern recognition by immunocomputing
25 Citations2003Alexander O. Tarakanov, Victor A. Skormin
It is shown that intrusion detection in computer networks presents a possible implementation of immunocomputing and its application to pattern recognition.
On the long-term stability of normal condition for damage detection in a composite panel
23 Citations2001G. Manson, Gareth Pierce +1 more
Proceedings of International Conference on Neural Networks (ICNN'97)ARTMAP-FD: familiarity discrimination applied to radar target recognition
20 Citations2002Gail A. Carpenter, Mark A. Rubin +1 more
ARTMAP-FD extends fuzzy ARTMAP to perform familiarity discrimination, where the network learns to abstain from meaningless guesses on patterns not belonging to a class represented in the training set.
Neural networks for novelty detection in airframe strain data
9 Citations2000Simon Hickinbotham, J. Austin
A method of automating detection of sensor faults is reported, believed to be the only known method that is capable of detecting such faults.