Dimensionality Reduction: A Comparative Review
Published 1 January 2009
Laurens van der Maaten, Eric Postma, H.J. van den Herik
Citations2,121
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The results of the experiments reveal that nonlinear techniques perform well on selected artificial tasks, but that this strong performance does not necessarily extend to real-world tasks.
Abstract
In recent years, a variety of nonlinear dimensionality reduction techniques have been proposed that aim to address the limitations of traditional techniques such as PCA and classical scaling. The paper presents a review and systematic comparison of these techniques. The performances of the nonlinear techniques are investigated on artificial and natural tasks. The results of the experiments reveal that nonlinear techniques perform well on selected artificial tasks, but that this strong performance does not necessarily extend to real-world tasks. The paper explains these results by identifying weaknesses of current nonlinear techniques, and suggests how the performance of nonlinear dimensionality reduction techniques may be improved. Dimensionality Reduction: A Comparative Review Laurens van der Maaten Eric Postma Jaap van den Herik
Keywords
Computer ScienceEngineering
Journal of the Royal Statistical Society Series B (Statistical Methodology)Maximum Likelihood from Incomplete Data Via the <i>EM</i> Algorithm
49,657 Citations1977A. P. Dempster, N. M. Laird +1 more
ScienceOptimization by Simulated Annealing
44,600 Citations1983Scott Kirkpatrick, C. D. Gelatt +1 more
A detailed analogy with annealing in solids provides a framework for optimization of the properties of very large and complex systems.
Journal of Machine Learning ResearchVisualizing Data using t-SNE
35,654 Citations2008Laurens van der Maaten, Geoffrey E. Hinton
A new technique called t-SNE that visualizes high-dimensional data by giving each datapoint a location in a two or three-dimensional map, a variation of Stochastic Neighbor Embedding that is much easier to optimize, and produces significantly better visualizations by reducing the tendency to crowd points together in the center of the map.
The MIT Press eBooksAdaptation in Natural and Artificial Systems
35,568 Citations1992John H. Holland
Initially applying his concepts to simply defined artificial systems with limited numbers of parameters, Holland goes on to explore their use in the study of a wide range of complex, naturally occuring processes, concentrating on systems having multiple factors that interact in nonlinear ways.
Cambridge University Press eBooksConvex Optimization
31,266 Citations2004Stephen Boyd, Lieven Vandenberghe
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
IEEE Transactions on Pattern Analysis and Machine IntelligenceNormalized cuts and image segmentation
15,696 Citations2000Jianbo Shi, Jitendra Malik
ScienceNonlinear Dimensionality Reduction by Locally Linear Embedding
15,035 Citations2000Sam T. Roweis, Lawrence K. Saul
Locally linear embedding (LLE) is introduced, an unsupervised learning algorithm that computes low-dimensional, neighborhood-preserving embeddings of high-dimensional inputs that learns the global structure of nonlinear manifolds.
Annals of EugenicsTHE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS
14,727 Citations1936Ronald Aylmer Fisher
ScienceA Global Geometric Framework for Nonlinear Dimensionality Reduction
13,740 Citations2000Joshua B. Tenenbaum, Vin de Silva +1 more
An approach to solving dimensionality reduction problems that uses easily measured local metric information to learn the underlying global geometry of a data set and efficiently computes a globally optimal solution, and is guaranteed to converge asymptotically to the true structure.
The London Edinburgh and Dublin Philosophical Magazine and Journal of ScienceLIII. <i>On lines and planes of closest fit to systems of points in space</i>
11,670 Citations1901Karl Pearson
This paper is concerned with the construction of planes of closest fit to systems of points in space and the relationships between these planes and the planes themselves.
Elsevier eBooksIntroduction to Statistical Pattern Recognition
11,121 Citations1990Keinosuke Fukunaga
A system to determine the sex of fish from their length, as accurately as possible, using a set of labelled data, together with the class label (male or female), as a training set.
Neural ComputationAn Information-Maximization Approach to Blind Separation and Blind Deconvolution
9,194 Citations1995Anthony J. Bell, Terrence J. Sejnowski
It is suggested that information maximization provides a unifying framework for problems in "blind" signal processing and dependencies of information transfer on time delays are derived.
Springer series in information sciencesSelf-Organization and Associative Memory
8,770 Citations1989Teuvo Kohonen
Neural ComputationNonlinear Component Analysis as a Kernel Eigenvalue Problem
8,078 Citations1998Bernhard Schölkopf, Alexander J. Smola +1 more
A new method for performing a nonlinear form of principal component analysis by the use of integral operator kernel functions is proposed and experimental results on polynomial feature extraction for pattern recognition are presented.
On Spectral Clustering: Analysis and an algorithm
7,751 Citations2001Andrew Y. Ng, Michael I. Jordan +1 more
A simple spectral clustering algorithm that can be implemented using a few lines of Matlab is presented, and tools from matrix perturbation theory are used to analyze the algorithm, and give conditions under which it can be expected to do well.
PsychometrikaMultidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis
7,398 Citations1964Joseph B. Kruskal
The fundamental hypothesis is that dissimilarities and distances are monotonically related, and a quantitative, intuitively satisfying measure of goodness of fit is defined to this hypothesis.
Extracting and composing robust features with denoising autoencoders
7,343 Citations2008Pascal Vincent, Hugo Larochelle +2 more
This work introduces and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern.
Cambridge University Press eBooksKernel Methods for Pattern Analysis
6,598 Citations2004John Shawe‐Taylor, Nello Cristianini
This book provides an easy introduction for students and researchers to the growing field of kernel-based pattern analysis, demonstrating with examples how to handcraft an algorithm or a kernel for a new specific application, and covering all the necessary conceptual and mathematical tools to do so.
Face recognition using eigenfaces
5,211 Citations2002Matthew Turk, Alex Pentland
An approach to the detection and identification of human faces is presented, and a working, near-real-time face recognition system which tracks a subject's head and then recognizes the person by comparing characteristics of the face to those of known individuals is described.
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
The MIT Press eBooksLaplacian Eigenmaps and Spectral Techniques for Embedding and Clustering
4,535 Citations2002Mikhail Belkin, Partha Niyogi
The algorithm provides a computationally efficient approach to nonlinear dimensionality reduction that has locality preserving properties and a natural connection to clustering.
Journal of Machine Learning ResearchDistance Metric Learning for Large Margin Nearest Neighbor Classification
4,253 Citations2009Kilian Q. Weinberger, Lawrence K. Saul
This paper shows how to learn a Mahalanobis distance metric for kNN classification from labeled examples in a globally integrated manner and finds that metrics trained in this way lead to significant improvements in kNN Classification.
SIAM ReviewSemidefinite Programming
4,021 Citations1996Lieven Vandenberghe, Stephen Boyd
A survey of the theory and applications of semidefinite programs and an introduction to primaldual interior-point methods for their solution are given.
Communications of the ACMAlgorithm 97: Shortest path
3,962 Citations1962Robert W. Floyd
The procedure was originally programmed in FORTRAN for the Control Data 160 desk-size computer and was limited to te t ra t ion because subroutine recursiveness in CONTROL Data 160 FORTRan has been held down to four levels in the interests of economy.
Locality Preserving Projections
3,955 Citations2003Xiaofei He, Partha Niyogi
These are linear projective maps that arise by solving a variational problem that optimally preserves the neighborhood structure of the data set by finding the optimal linear approximations to the eigenfunctions of the Laplace Beltrami operator on the manifold.
TechnometricsGeneralized Cross-Validation as a Method for Choosing a Good Ridge Parameter
3,770 Citations1979Gene H. Golub, Michael T. Heath +1 more
The method of generalized cross-validation (GCV) for choosing a good value for λ from the data is studied, which can be used in subset selection and singular value truncation methods for regression, and even to choose from among mixtures of these methods.
IEEE Transactions on ComputersA Nonlinear Mapping for Data Structure Analysis
3,381 Citations1969John W. Sammon
An algorithm for the analysis of multivariate data is presented along with some experimental results that is based upon a point mapping of N L-dimensional vectors from the L-space to a lower-dimensional space such that the inherent data "structure" is approximately preserved.
IEEE Transactions on Pattern Analysis and Machine IntelligenceFace recognition using Laplacianfaces
3,276 Citations2005Xiaofei He, Shuicheng Yan +3 more
Experimental results suggest that the proposed Laplacianface approach provides a better representation and achieves lower error rates in face recognition.
Super-resolution through neighbor embedding
1,953 Citations2004Hong Chang, Dit‐Yan Yeung +1 more
This paper proposes a novel method for solving single-image super-resolution problems, given a low-resolution image as input, and recovers its high-resolution counterpart using a set of training examples, inspired by recent manifold teaming methods.
Neural ComputationMixtures of Probabilistic Principal Component Analyzers
1,903 Citations1999Michael E. Tipping, Chris Bishop
PCA is formulated within a maximum likelihood framework, based on a specific form of gaussian latent variable model, which leads to a well-defined mixture model for probabilistic principal component analyzers, whose parameters can be determined using an expectation-maximization algorithm.
TechnometricsIntroduction to Multivariate Analysis
1,789 Citations1983Jack C. Lee, C. Chatfield +1 more
Quarterly of Applied MathematicsThe principle of minimized iterations in the solution of the matrix eigenvalue problem
1,770 Citations1951Walter E. Arnoldi
Neighbourhood Components Analysis
1,729 Citations2004Jacob Goldberger, Geoffrey E. Hinton +2 more
A novel method for learning a Mahalanobis distance measure to be used in the KNN classification algorithm that directly maximizes a stochastic variant of the leave-one-out KNN score on the training set.
Proceedings of the National Academy of SciencesHessian eigenmaps: Locally linear embedding techniques for high-dimensional data
1,696 Citations2003David L. Donoho, Carrie Grimes
The Hessian-based locally linear embedding method for recovering the underlying parametrization of scattered data (mi) lying on a manifold M embedded in high-dimensional Euclidean space is described, where the isometric coordinates can be recovered up to a linear isometry.
Neural ComputationGeneralized Discriminant Analysis Using a Kernel Approach
1,679 Citations2000G. Baudat, F. Anouar
A new method that is close to the support vector machines insofar as the GDA method provides a mapping of the input vectors into high-dimensional feature space to deal with nonlinear discriminant analysis using kernel function operator.
Information science and statisticsNonlinear Dimensionality Reduction
1,530 Citations2007John A. Lee, Michel Verleysen
The purpose of the book is to summarize clear facts and ideas about well-known methods as well as recent developments in the topic of nonlinear dimensionality reduction, which encompasses many of the recently developed methods.
Neighborhood preserving embedding
1,510 Citations2005Xiaofei He, Deng Cai +2 more
This paper proposes a novel subspace learning algorithm called neighborhood preserving embedding (NPE), which aims at preserving the local neighborhood structure on the data manifold and is less sensitive to outliers than principal component analysis (PCA).
SIAM Journal on Scientific ComputingPrincipal Manifolds and Nonlinear Dimensionality Reduction via Tangent Space Alignment
1,471 Citations2004Zhenyue Zhang, Hongyuan Zha
Stochastic Neighbor Embedding
1,461 Citations2002Geoffrey E. Hinton, Sam T. Roweis
This probabilistic framework makes it easy to represent each object by a mixture of widely separated low-dimensional images, which allows ambiguous objects, like the document count vector for the word "bank", to have versions close to the images of both "river" and "finance" without forcing the image of outdoor concepts to be located close to those of corporate concepts.
Neural ComputationGTM: The Generative Topographic Mapping
1,389 Citations1998Chris Bishop, Markus Svensén +1 more
A form of nonlinear latent variable model called the generative topographic mapping, for which the parameters of the model can be determined using the expectation-maximization algorithm, is introduced.
Springer eBooksSelf-organization and associative memory: 3rd edition
1,230 Citations1989T. Kohonen
This work prospectively examined the association between magnesium intake and incidence of Type 2 diabetes in a general Japanese population.
Journal of Machine Learning ResearchExploring Strategies for Training Deep Neural Networks
1,114 Citations2009Hugo Larochelle, Yoshua Bengio +2 more
Applied and Computational Harmonic AnalysisDiffusion maps, spectral clustering and reaction coordinates of dynamical systems
1,016 Citations2006Boaz Nadler, Stéphane Lafon +2 more
A unifying view of a family of diffusion maps, defined as the embedding of complex data onto a low dimensional Euclidean space, via the eigenvectors of suitably defined random walks defined on the given datasets, is provided.
Probabilistic Non-linear Principal Component Analysis with Gaussian Process Latent Variable Models
948 Citations2005Neil D. Lawrence
Scaling learning algorithms towards AI
927 Citations2007Yoshua Bengio, Yann LeCun
It is argued that deep architectures have the potential to generalize in non-local ways, i.e., beyond immediate neighbors, and that this is crucial in order to make progress on the kind of complex tasks required for artificial intelligence.
Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering
867 Citations2003Yoshua Bengio, Jean-françcois Paiement +4 more
A unified framework for extending Local Linear Embedding, Isomap, Laplacian Eigenmaps, Multi-Dimensional Scaling as well as for Spectral Clustering is provided.
ScienceThe Isomap Algorithm and Topological Stability
844 Citations2002Mukund Balasubramanian, Eric L. Schwartz
An algorithm for computing a quasi-isometric, low-dimensional embedding of a set of high-dimensional data points and the basic approach presented is not new, having been presented previously.
IEEE Transactions on Evolutionary ComputationDimensionality reduction using genetic algorithms
842 Citations2000Michael L. Raymer, William F. Punch +3 more
This work presents a new approach to feature extraction in which feature selection and extraction and classifier training are performed simultaneously using a genetic algorithm, and employs this technique in combination with the k nearest neighbor classification rule.
Kernel PCA and De-Noising in Feature Spaces
828 Citations1998Sebastian Mika, Bernhard Schölkopf +4 more
This work presents ideas for finding approximate pre-images, focusing on Gaussian kernels, and shows experimental results using these pre- images in data reconstruction and de-noising on toy examples as well as on real world data.
Machine LearningSemi-Supervised Learning on Riemannian Manifolds
807 Citations2004Mikhail Belkin, Partha Niyogi
An algorithmic framework to classify a partially labeled data set in a principled manner and models the manifold using the adjacency graph for the data and approximates the Laplace-Beltrami operator by the graph Laplacian.
Neural Information Processing SystemsGlobal Versus Local Methods in Nonlinear Dimensionality Reduction
798 Citations2002Vin D. Silva, Joshua B. Tenenbaum
This work presents two variants of Isomap which combine the advantages of the global approach with what have previously been exclusive advantages of local methods: computational sparsity and the ability to invert conformal maps.
Pattern RecognitionKernel PCA for novelty detection
752 Citations2006H. Hoffmann
This study introduces and investigates the use of kernel PCA for novelty detection and demonstrated a competitive performance on two-dimensional synthetic distributions and on two real-world data sets: handwritten digits and breast-cancer cytology.
EM Algorithms for PCA and SPCA
732 Citations1997Sam T. Roweis
An expectation-maximization (EM) algorithm for principal component analysis (PCA) which allows a few eigenvectors and eigenvalues to be extracted from large collections of high dimensional data and defines a proper density model in the data space.
Neural ComputationDimension Reduction by Local Principal Component Analysis
723 Citations1997Nandakishore Kambhatla, Todd K. Leen
A local linear approach to dimension reduction that provides accurate representations and is fast to compute is developed and it is shown that the local linear techniques outperform neural network implementations.
Segmentation using eigenvectors: a unifying view
722 Citations1999Yaakov Weiss
A unified treatment of eigenvectors of block matrices based on eigendecompositions in the context of segmentation is given, and close connections between them are shown while highlighting their distinguishing features.
Maximum Likelihood Estimation of Intrinsic Dimension
709 Citations2004Elizaveta Levina, Peter J. Bickel
A new method for estimating intrinsic dimension of a dataset derived by applying the principle of maximum likelihood to the distances between close neighbors is proposed, which has the best overall performance compared with two other intrinsic dimension estimators.
Metric Learning by Collapsing Classes
663 Citations2005Amir Globerson, Sam T. Roweis
An algorithm for learning a quadratic Gaussian metric (Mahalanobis distance) for use in classification tasks and discusses how the learned metric may be used to obtain a compact low dimensional feature representation of the original input space, allowing more efficient classification with very little reduction in performance.
IEEE Transactions on Neural NetworksCurvilinear component analysis: a self-organizing neural network for nonlinear mapping of data sets
651 Citations1997Pierre Demartines, J. Hérault
A self-organized neural network performing two tasks: vector quantization of the submanifold in the data set (input space) and nonlinear projection of these quantizing vectors toward an output space, providing a revealing unfolding of theSub manifold.
IEEE Transactions on Pattern Analysis and Machine IntelligenceDiffusion maps and coarse-graining: a unified framework for dimensionality reduction, graph partitioning, and data set parameterization
619 Citations2006Stéphane Lafon, A.B. Lee
It is shown that the quantization distortion in diffusion space bounds the error of compression of the operator, thus giving a rigorous justification for k-means clustering in diffusionspace and a precise measure of the performance of general clustering algorithms.
Scientific Repository (Petra Christian University)The EM algorithm for mixtures of factor analyzers
585 Citations1996Zoubin Ghahramani, GE Hinton
This work presents an exact Expectation{Maximization algorithm for determining the parameters of this mixture of factor analyzers which concurrently performs clustering and dimensionality reduction, and can be thought of as a reduced dimension mixture of Gaussians.
A kernel view of the dimensionality reduction of manifolds
543 Citations2004Jihun Ham, Daniel D. Lee +2 more
Isomap, graph Laplacian eigenmap, and locally linear embedding all utilize local neighborhood information to construct a global embedding of the manifold and it is shown how all three algorithms can be described as kernel PCA on specially constructed Gram matrices.
Learning a kernel matrix for nonlinear dimensionality reduction
521 Citations2004Kilian Q. Weinberger, Fei Sha +1 more
This work investigates how to learn a kernel matrix for high dimensional data that lies on or near a low dimensional manifold and shows how to discover a mapping that "unfolds" the underlying manifold from which the data was sampled.
Journal of Machine Learning ResearchLearning a Mahalanobis Metric from Equivalence Constraints
515 Citations2005Aharon Bar-Hillel, Tomer Hertz +2 more
This work presents the Relevant Component Analysis algorithm, which is a simple and efficient algorithm for learning a Mahalanobis metric, and shows that RCA is the solution of an interesting optimization problem, founded on an information theoretic basis.
IEEE Signal Processing LettersFace recognition using kernel principal component analysis
514 Citations2002Kwang In Kim, Keechul Jung +1 more
Through adopting a polynomial kernel, the principal components can be computed within the space spanned by high-order correlations of input pixels making up a facial image, thereby producing a good performance.
Linear and Multilinear AlgebraEigenvalues of the Laplacian of a graph<sup>∗</sup>
480 Citations1985William N. Anderson, T. D. Morley
Learning a Nonlinear Embedding by Preserving Class Neighbourhood Structure
438 Citations2007Ruslan Salakhutdinov, Geoffrey E. Hinton
Exponential Family Harmoniums with an Application to Information Retrieval
436 Citations2004Max Welling, Michal Rosen‐Zvi +1 more
An alternative two-layer model based on exponential family distributions and the semantics of undirected models is proposed, which performs well on document retrieval tasks and provides an elegant solution to searching with keywords.
IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews)Supervised classification in high-dimensional space: geometrical, statistical, and asymptotical properties of multivariate data
419 Citations1998Luís Ortiz Jiménez, D. A. Landgrebe
High-dimensional space properties are investigated using Euclidean and Cartesian geometry and their implication for high-dimensional data and its analysis is studied in order to illuminate the differences between conventional spaces and hyperdimensional space.
Charting a Manifold
374 Citations2002Matthew Brand
A nonlinear mapping from a high-dimensional sample space to a low-dimensional vector space is constructed, effectively recovering a Cartesian coordinate system for the manifold from which the data is sampled, and is pseudo-invertible.
Neural ComputationLearning Eigenfunctions Links Spectral Embedding and Kernel PCA
345 Citations2004Yoshua Bengio, Olivier Delalleau +4 more
A direct relation is shown between spectral embedding methods and kernel principal components analysis and how both are special cases of a more general learning problem: learning the principal eigenfunctions of an operator defined from a kernel and the unknown data-generating density.
Finding nearest neighbors in growth-restricted metrics
342 Citations2002David R. Karger, Matthias Rühl
This paper develops an efficient dynamic data structure for nearest neighbor queries in growth-constrained metrics that satisfy the property that for any point q and number r the ratio between numbers of points in balls of radius 2r and r is bounded by a constant.
Linear discriminant analysis for improved large vocabulary continuous speech recognition
320 Citations1992Reinhold Haeb‐Umbach, Hermann Ney
The interaction of linear discriminant analysis (LDA) and a modeling approach using continuous Laplacian mixture density HMM is studied experimentally and the largest improvements in speech recognition could be obtained when the classes for the LDA transform were defined to be sub-phone units.
SIAM Journal on Scientific ComputingJacobi--Davidson Style QR and QZ Algorithms for the Reduction of Matrix Pencils
319 Citations1998Diederik R. Fokkema, Gérard L. G. Sleijpen +1 more
Two algorithms, JDQZ for the generalized eigen problem and JDQR for the standard eigenproblem, that are based on the iterative construction of a (generalized) partial Schur form are presented, suitable for the efficient computation of several eigenvalues and the corresponding eigenvectors near a user-specified target value in the complex plane.
IEEE Transactions on Pattern Analysis and Machine IntelligenceOrthogonal Neighborhood Preserving Projections: A Projection-Based Dimensionality Reduction Technique
314 Citations2007Effrosyni Kokiopoulou, Yousef Saad
This paper proposes a method, named orthogonal neighborhood preserving projections, which works by first building an "affinity" graph for the data in a way that is similar to the method of locally linear embedding (LLE); in contrast with the standard LLE, ONPP employs an explicit linear mapping between the input and the reduced spaces.
IEEE Transactions on Pattern Analysis and Machine IntelligenceIncremental nonlinear dimensionality reduction by manifold learning
285 Citations2006M.H.C. Law, Anil K. Jain
An incremental version of ISOMAP, one of the key manifold learning algorithms, is described and it is demonstrated that this modified algorithm can maintain an accurate low-dimensional representation of the data in an efficient manner.
Lecture notes in computer scienceSupervised Locally Linear Embedding
279 Citations2003Dick de Ridder, Olga Kouropteva +3 more
Two extensions of LLE to supervised feature extraction were independently proposed by the authors of this paper and are unified in a common framework and applied to a number of benchmark data sets.
The MIT Press eBooksSpectral Methods for Dimensionality Reduction
271 Citations2006Lawrence K. Saul, Kilian Q. Weinberger +3 more
Mapping a Manifold of Perceptual Observations
251 Citations1997Joshua B. Tenenbaum
The isometric feature mapping procedure, or isomap, is able to reliably recover low-dimensional nonlinear structure in realistic perceptual data sets, such as a manifold of face images, where conventional global mapping methods find only local minima.
NeurocomputingLinear local tangent space alignment and application to face recognition
249 Citations2006Tianhao Zhang, Jie Yang +2 more
Since images of faces often belong to a manifold of intrinsically low dimension, the LLTSA algorithm for effective face manifold learning and recognition is developed, which achieves much higher recognition rates than a few competing methods.
Journal of Molecular ModelingNew quantitative descriptors of amino acids based on multidimensional scaling of a large number of physical?chemical properties
243 Citations2001Werner Braun, Mathura S. Venkatarajan
It is shown that a five- dimensional property space can be constructed such that the amino acids are in a similar spatial distribution to that in the original high-dimensional property space.
Non-Linear Dimensionality Reduction
240 Citations1992David DeMers, Garrison W. Cottrell
The commonly used technique of autoassociation is extended to allow non-linear representations, and an objective function which penalizes activations of individual hidden units is shown to result in minimum dimensional encodings with respect to allowable error in reconstruction.
Structure preserving embedding
211 Citations2009Blake Shaw, Tony Jebara
SPE provides significant improvements in terms of visualization and lossless compression of graphs, outperforming popular methods such as spectral embedding and Laplacian eigen-maps and introducing structure preserving constraints into dimensionality reduction algorithms produces more accurate representations of high-dimensional data.
Physics in Medicine and BiologyComputer-aided classification of mammographic masses and normal tissue: linear discriminant analysis in texture feature space
209 Citations1995Heang‐Ping Chan, Datong Wei +5 more
The results demonstrate the feasibility of using linear discriminant analysis in the texture feature space for classification of true and false detections of masses on mammograms in a computer-aided diagnosis scheme.
Machine LearningOn a Connection between Kernel PCA and Metric Multidimensional Scaling
205 Citations2002Christopher K. I. Williams
The kernel PCA algorithm of Schölkopf, Smola, and Müller can be interpreted as a form of metric multidimensional scaling (MDS) when the kernel function k(x, y) is isotropic, i.e. it depends only on ‖x − y‖.
ScholarlyCommons (University of Pennsylvania)Global Coordination of Local Linear Models
177 Citations2001Sam T. Roweis, Lawrence K. Saul +1 more
The regularizer takes the form of a Kullback-Leibler divergence and illustrates an unexpected application of variational methods: not to perform approximate inference in intractable probabilistic models, but to learn more useful internal representations in tractable ones.
The use of genetic algorithms and neural networks to approximate missing data in database
174 Citations2006M. Abdella, Tshilidzi Marwala
It is observed that there is no significant reduction in accuracy of results as the number of missing cases in a single record increases, and it is found that results obtained using RBF are superior to MLP.
Journal of Computational ChemistryStochastic proximity embedding
172 Citations2003Dimitris K. Agrafiotis
Stochastic proximity embedding is introduced, a novel self‐organizing algorithm for producing meaningful underlying dimensions from proximity data that scales linearly with respect to sample size, and can be applied to very large data sets that are intractable by conventional embedding procedures.
IEEE Transactions on Signal ProcessingAdaptive Principal component EXtraction (APEX) and applications
170 Citations1994Sun‐Yuan Kung, Konstantinos Diamantaras +1 more
A neural network model (APEX) for multiple principal component extraction that is applicable to the constrained PCA problem where the signal variance is maximized under external orthogonality constraints and the exponential convergence of the network is formally proved.
Deriving action and behavior primitives from human motion data
165 Citations2003Odest Chadwicke Jenkins, Maja J. Matarić
This work presents a data-driven method for deriving perceptual-motor action and behavior primitives from human motion capture data using a spatio-temporal non-linear dimension reduction technique on a set of motion segments.
Pattern RecognitionRobust kernel Isomap
160 Citations2006Heeyoul Choi, Seungjin Choi
A method is presented which relates the Isomap to Mercer kernel machines, so that the generalization property naturally emerges, through kernel principal component analysis.
Nonlinear Dimensionality Reduction by Semidefinite Programming and Kernel Matrix Factorization.
154 Citations2005Kilian Q. Weinberger, Benjamin Packer +1 more
It is shown that the full kernel matrix can be very well approximated by a product of smaller matrices, which leads to order-of-magnitude reductions in computation time and makes it possible to study much larger problems in manifold learning.
Intelligent Data AnalysisFast dimensionality reduction and simple PCA
153 Citations1998Mike Partridge, Rafael A. Calvo
Sparse Kernel Principal Component Analysis
144 Citations2000Michael E. Tipping
By approximating the covariance matrix in feature space by a reduced number of example vectors, using a maximum-likelihood approach, it is shown that a highly sparse form of kernel PCA can be obtained without loss of effectiveness.
…