Invariant operators, small samples, and the bias-variance dilemma
Published 12 November 2004
Xiaojin Shi, Roberto Manduchi
Citations6
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An analysis of this interesting behavior based on the bias-variance dilemma is proposed, and the concept of "randomized invariants" for training is introduced, which can be used to mitigate the effect of small sample size.
Abstract
Invariant features or operators are often used to shield the recognition process from the effect of "nuisance" parameters, such as rotations, foreshortening, or illumination changes. From an information-theoretic point of view, imposing invariance results in reduced (rather than improved) system performance. In fact, in the case of small training samples, the situation is reversed, and invariant operators may reduce the misclassification rate. We propose an analysis of this interesting behavior based on the bias-variance dilemma, and present experimental results confirming our theoretical expectations. In addition, we introduce the concept of "randomized invariants" for training, which can be used to mitigate the effect of small sample size.
Keywords
Computer Science
Springer series in statisticsThe Elements of Statistical Learning
24,344 Citations2001Trevor Hastie, J. Friedman +1 more
IEEE ExpertData mining and knowledge discovery: making sense out of data
4,643 Citations1996U.M. Feyyad
Find loads of the data mining and knowledge discovery making sense out of data book catalogues in this site as the choice of you visiting this page.
Neural ComputationNeural Networks and the Bias/Variance Dilemma
3,563 Citations1992Stuart Geman, Elie Bienenstock +1 more
It is suggested that current-generation feedforward neural networks are largely inadequate for difficult problems in machine perception and machine learning, regardless of parallel-versus-serial hardware or other implementation issues.
Data Mining and Knowledge DiscoveryOn Bias, Variance, 0/1—Loss, and the Curse-of-Dimensionality
1,029 Citations1997Jerome H. Friedman
This work candramatically mitigate the effect of the bias associated with some simpleestimators like “naive” Bayes, and the bias induced by the curse-of-dimensionality on nearest-neighbor procedures.
Neural Information Processing SystemsEfficient Pattern Recognition Using a New Transformation Distance
464 Citations1992Patrice Simard, Yann LeCun +1 more
A new distance measure which can be made locally invariant to any set of transformations of the input and can be computed efficiently is proposed.
Elsevier eBooksError-Correcting Output Coding Corrects Bias and Variance
357 Citations1995Eun Bae Kong, Thomas G. Dietterich
An investigation of why the ECOC technique works, particularly when employed with decision-tree learning algorithms, shows that it can reduce the variance of the learning algorithm.
Lecture notes in computer scienceIncorporating invariances in support vector learning machines
280 Citations1996Bernhard Schölkopf, Chris Burges +1 more
This work presents a method of incorporating prior knowledge about transformation invariances by applying transformations to support vectors, the training examples most critical for determining the classification boundary.
International Journal of Pattern Recognition and Artificial IntelligenceBOOSTING PERFORMANCE IN NEURAL NETWORKS
184 Citations1993Harris Drucker, Robert E. Schapire +1 more
The boosting algorithm is used to construct an ensemble of neural networks that significantly improves performance (compared to a single network) in optical character recognition (OCR) problems and improved performance significantly, and, in some cases, dramatically.
Bias, Variance, and Arcing Classifiers
171 Citations1996Leo Breiman
A Unified Bias-Variance Decomposition for Zero-One and Squared Loss
150 Citations2000Pedro Domingos
This paper defines bias and variance for an arbitrary loss function, and shows that the resulting decomposition specializes to the standard one for the squared-loss case, and to a close relative of Kong and Dietterich’ s (1995)One for the zero-one case.
Image and Vision ComputingVision and Lie's approach to invariance
81 Citations1995Luc Van Gool, Theo Moons +2 more
The theory of Lie groups is described, a systematic approach to the problem of extracting invariants from Lie groups that helps in tackling problems such as calibration-less vision, efficient matching, shape-from-motion, grouping, and several other problems considered crucial to intelligent vision.
IEEE Transactions on Pattern Analysis and Machine IntelligenceOn dimensionality, sample size, and classification error of nonparametric linear classification algorithms
61 Citations1997Šarūnas Raudys
Two nonparametric linear classification algorithms - the zero empirical error classifier and the maximum margin classifier - with parametric linear classifiers designed to classify multivariate Gaussian populations are compared.
A Study on Bayes Feature Fusion for Image Classification
47 Citations2003Xiaojin Shi, Roberto Manduchi
The results suggest that the difference between the correct classification rates using Bayes fusion and using the joint feature distribution is a function of the conditional dependence of the features (measured in terms of mutual information), however: for small training data size,Bayes fusion performs almost as well as the classifier on the joint distribution.
eScholarship (California Digital Library)Small Sample Statistics for Classification Error Rates I: Error Rate Measurements
43 Citations1995J. Kent Martin, D. S. Hirschberg