Big Data Deep Learning: Challenges and Perspectives
IEEE AccessPublished 1 January 2014Open access
Xuewen Chen, Xiaotong Lin
Citations1,232
SJR quartileQ1
SJR score0.85
SNIP1.50
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A brief overview of deep learning is provided, and current research efforts and the challenges to big data, as well as the future trends are highlighted.
Abstract
Deep learning is currently an extremely active research area in machine learning and pattern recognition society. It has gained huge successes in a broad area of applications such as speech recognition, computer vision, and natural language processing. With the sheer size of data available today, big data brings big opportunities and transformative potential for various sectors; on the other hand, it also presents unprecedented challenges to harnessing data and information. As the data keeps getting bigger, deep learning is coming to play a key role in providing big data predictive analytics solutions. In this paper, we provide a brief overview of deep learning, and highlight current research efforts and the challenges to big data, as well as the future trends.
Keywords
Computer Science
Communications of the ACMImageNet classification with deep convolutional neural networks
75,709 Citations2017Alex Krizhevsky, Ilya Sutskever +1 more
A large, deep convolutional neural network was trained to classify the 1.2 million high-resolution images in the ImageNet LSVRC-2010 contest into the 1000 different classes and employed a recently developed regularization method called "dropout" that proved to be very effective.
Proceedings of the IEEEGradient-based learning applied to document recognition
58,219 Citations1998Yann LeCun, Léon Bottou +2 more
This paper reviews various methods applied to handwritten character recognition and compares them on a standard handwritten digit recognition task, and Convolutional neural networks are shown to outperform all other techniques.
NatureLearning representations by back-propagating errors
30,885 Citations1986David E. Rumelhart, Geoffrey E. Hinton +1 more
Back-propagation repeatedly adjusts the weights of the connections in the network so as to minimize a measure of the difference between the actual output vector of the net and the desired output vector, which helps to represent important features of the task domain.
IEEE Transactions on Knowledge and Data EngineeringA Survey on Transfer Learning
23,164 Citations2009Sinno Jialin Pan, Qiang Yang
The relationship between transfer learning and other related machine learning techniques such as domain adaptation, multitask learning and sample selection bias, as well as covariate shift are discussed.
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
IEEE Transactions on Pattern Analysis and Machine IntelligenceStochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images
17,980 Citations1984Stuart Geman, Donald Geman
The analogy between images and statistical mechanics systems is made and the analogous operation under the posterior distribution yields the maximum a posteriori (MAP) estimate of the image given the degraded observations, creating a highly parallel ``relaxation'' algorithm for MAP estimation.
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
Statistical Comparisons of Classifiers over Multiple Data Sets
11,198 Citations2006Janez Demšar
A set of simple, yet safe and robust non-parametric tests for statistical comparisons of classifiers is recommended: the Wilcoxon signed ranks test for comparison of two classifiers and the Friedman test with the corresponding post-hoc tests for comparisons of more classifiers over multiple data sets.
IEEE Signal Processing MagazineDeep Neural Networks for Acoustic Modeling in Speech Recognition: The Shared Views of Four Research Groups
10,313 Citations2012Geoffrey E. Hinton, Li Deng +9 more
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
8,612 Citations2011John C. Duchi, Elad Hazan +1 more
Extracting and composing robust features with denoising autoencoders
7,343 Citations2008Pascal Vincent, Hugo Larochelle +2 more
This work introduces and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern.
The Journal of PhysiologyReceptive fields and functional architecture of monkey striate cortex
6,636 Citations1968David H. Hubel, T. N. Wiesel
The striate cortex was studied in lightly anaesthetized macaque and spider monkeys by recording extracellularly from single units and stimulating the retinas with spots or patterns of light, with response properties very similar to those previously described in the cat.
Combining labeled and unlabeled data with co-training
5,604 Citations1998Avrim Blum, Tom M. Mitchell
Big data: The next frontier for innovation, competition, and productivity
5,228 Citations2011James Manyika
The amount of data in the authors' world has been exploding, and analyzing large data sets will become a key basis of competition, underpinning new waves of productivity growth, innovation, and consumer surplus, according to research by MGI and McKinsey.
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Neural ComputationTraining Products of Experts by Minimizing Contrastive Divergence
5,017 Citations2002Geoffrey E. Hinton
A product of experts (PoE) is an interesting candidate for a perceptual system in which rapid inference is vital and generation is unnecessary because it is hard even to approximate the derivatives of the renormalization term in the combination rule.
The MIT Press eBooksGreedy Layer-Wise Training of Deep Networks
4,704 Citations2007Yoshua Bengio, Pascal Lamblin +2 more
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
Natural Language Processing (almost) from Scratch
3,987 Citations2011Ronan Collobert, Jason Weston +4 more
IEEE Transactions on Audio Speech and Language ProcessingContext-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition
3,052 Citations2011George E. Dahl, Dong Yu +2 more
A pre-trained deep neural network hidden Markov model (DNN-HMM) hybrid architecture that trains the DNN to produce a distribution over senones (tied triphone states) as its output that can significantly outperform the conventional context-dependent Gaussian mixture model (GMM)-HMMs.
Large Scale Distributed Deep Networks
2,916 Citations2012Jay B. Dean, Greg S. Corrado +10 more
This paper considers the problem of training a deep network with billions of parameters using tens of thousands of CPU cores and develops two algorithms for large-scale distributed training, Downpour SGD and Sandblaster L-BFGS, which increase the scale and speed of deep network training.
Best practices for convolutional neural networks applied to visual document analysis
2,815 Citations2005P. Simard, Don Steinkraus +1 more
A set of concrete bestpractices that document analysis researchers can use to get good results with neural networks, including a simple "do-it-yourself" implementation of convolution with a flexible architecture suitable for many visual document problems.
Lecture notes in computer scienceA Practical Guide to Training Restricted Boltzmann Machines
2,802 Citations2012Geoffrey E. Hinton
This guide is an attempt to share expertise at training restricted Boltzmann machines with other machine learning researchers.
The MIT Press eBooksEfficient sparse coding algorithms
2,393 Citations2007Honglak Lee, Alexis Battle +2 more
These algorithms are applied to natural images and it is demonstrated that the inferred sparse codes exhibit end-stopping and non-classical receptive field surround suppression and, therefore, may provide a partial explanation for these two phenomena in V1 neurons.
The American StatisticianExplaining the Gibbs Sampler
2,355 Citations1992George Casella, Edward I. George
A simple explanation of how and why the Gibbs sampler works is given and analytically establish its properties in a simple case and insight is provided for more complicated cases.
Multimodal Deep Learning
2,290 Citations2011Jiquan Ngiam, Aditya Khosla +4 more
Adaptive Algorithms and Stochastic Approximations
2,044 Citations1990Albert Benveniste, Michel Métivier +1 more
The juxtaposition of these two expressions in the title reflects the ambition of the authors to produce a reference work, both for engineers who use adaptive algorithms and for probabilists or statisticians who would like to study stochastic approximations in terms of problems arising from real applications.
IEEE Transactions on Pattern Analysis and Machine Intelligence80 Million Tiny Images: A Large Data Set for Nonparametric Object and Scene Recognition
1,976 Citations2008Antonio Torralba, Rob Fergus +1 more
For certain classes that are particularly prevalent in the dataset, such as people, this work is able to demonstrate a recognition performance comparable to class-specific Viola-Jones style detectors.
IEEE Transactions on Neural NetworksA Fast and Accurate Online Sequential Learning Algorithm for Feedforward Networks
1,974 Citations2006Nanying Liang, Guang-Bin Huang +2 more
The results show that the OS-ELM is faster than the other sequential algorithms and produces better generalization performance on benchmark problems drawn from the regression, classification and time series prediction areas.
Restricted Boltzmann machines for collaborative filtering
1,882 Citations2007Ruslan Salakhutdinov, Andriy Mnih +1 more
This paper shows how a class of two-layer undirected graphical models, called Restricted Boltzmann Machines (RBM's), can be used to model tabular data, such as user's ratings of movies, and demonstrates that RBM's can be successfully applied to the Netflix data set.
Building high-level features using large scale unsupervised learning
1,650 Citations2013Quoc V. Le
Contrary to what appears to be a widely-held intuition, the experimental results reveal that it is possible to train a face detector without having to label images as containing a face or not.
Domain adaptation for large-scale sentiment classification: A deep learning approach
1,563 Citations2011Xavier Glorot, Antoine Bordes +1 more
A deep learning approach is proposed which learns to extract a meaningful representation for each review in an unsupervised fashion and clearly outperform state-of-the-art methods on a benchmark composed of reviews of 4 types of Amazon products.
Self-taught learning
1,533 Citations2007Rajat Raina, Alexis Battle +3 more
An approach to self-taught learning that uses sparse coding to construct higher-level features using the unlabeled data to form a succinct input representation and significantly improve classification performance.
Mathematical ProgrammingPegasos: primal estimated sub-gradient solver for SVM
1,524 Citations2010Shai Shalev‐Shwartz, Yoram Singer +2 more
Lecture notes in computer scienceEvaluation of Pooling Operations in Convolutional Architectures for Object Recognition
1,518 Citations2010Dominik Scherer, Andreas Müller +1 more
The aim is to gain insight into different functions by directly comparing them on a fixed architecture for several common object recognition tasks, and empirical results show that a maximum pooling operation significantly outperforms subsampling operations.
The MIT Press eBooksMap-Reduce for Machine Learning on Multicore
1,253 Citations2007Cheng-Tao Chu, Sang Kyun Kim +5 more
This work shows that algorithms that fit the Statistical Query model can be written in a certain "summation form," which allows them to be easily parallelized on multicore computers and shows basically linear speedup with an increasing number of processors.
Adaptive deconvolutional networks for mid and high level feature learning
1,249 Citations2011Matthew D. Zeiler, Graham W. Taylor +1 more
A hierarchical model that learns image decompositions via alternating layers of convolutional sparse coding and max pooling, relying on a novel inference scheme that ensures each layer reconstructs the input, rather than just the output of the layer directly beneath, as is common with existing hierarchical approaches.
now publishers, Inc. eBooksOnline Learning and Online Convex Optimization
1,219 Citations2011Shai Shalev‐Shwartz
Foundations and Trends® in Machine LearningOnline Learning and Online Convex Optimization
1,210 Citations2012Shai Shalev‐Shwartz
A modern overview of online learning is provided to give the reader a sense of some of the interesting ideas and in particular to underscore the centrality of convexity in deriving efficient online learning algorithms.
International Joint Conference on Artificial IntelligenceFlexible, high performance convolutional neural networks for image classification
1,203 Citations2011Dan Cireşan, Ueli Meier +3 more
Journal of Machine Learning ResearchExploring Strategies for Training Deep Neural Networks
1,114 Citations2009Hugo Larochelle, Yoshua Bengio +2 more
Neural ComputationDeep, Big, Simple Neural Nets for Handwritten Digit Recognition
1,064 Citations2010Dan Claudiu Cireşan, Ueli Meier +2 more
Good old online backpropagation for plain multilayer perceptrons yields a very low 0.35 error rate on the MNIST handwritten digits benchmark.
Pegasos
982 Citations2007Shai Shalev‐Shwartz, Yoram Singer +1 more
A simple and effective stochastic sub-gradient descent algorithm for solving the optimization problem cast by Support Vector Machines, which is particularly well suited for large text classification problems, and demonstrates an order-of-magnitude speedup over previous SVM learning methods.
IEEE Transactions on Neural NetworksIncremental Learning of Concept Drift in Nonstationary Environments
898 Citations2011Ryan Elwell, Robi Polikar
An ensemble of classifiers-based approach for incremental learning of concept drift, characterized by nonstationary environments (NSEs), where the underlying data distributions change over time, which indicates that Learn++.NSE can track the changing environments very closely, regardless of the type of concept Drift.
Deep Learning of Representations for Unsupervised and Transfer Learning.
893 Citations2012Yoshua Bengio
Why unsupervised pre-training of representations can be useful, and how it can be exploited in the transfer learning scenario, where the authors care about predictions on examples that are not from the same distribution as the training distribution.
The American StatisticianExplaining the Gibbs Sampler
873 Citations1992George Casella, Edward I. George
Lecture notes in computer scienceEfficient BackProp
867 Citations1998Yann LeCun, Léon Bottou +2 more
Many undesirable behaviors of backprop can be avoided with tricks that are rarely exposed in serious technical publications, and this paper gives some of those tricks, and offers explanations of why they work.
Large-scale deep unsupervised learning using graphics processors
729 Citations2009Rajat Raina, Anand Madhavan +1 more
It is argued that modern graphics processors far surpass the computational capabilities of multicore CPUs, and have the potential to revolutionize the applicability of deep unsupervised learning methods.
Neural Information Processing SystemsSparse Feature Learning for Deep Belief Networks
715 Citations2007Marc’Aurelio Ranzato, Y-Lan Boureau +1 more
This work proposes a simple criterion to compare and select different unsupervised machines based on the trade-off between the reconstruction error and the information content of the representation, and describes a novel and efficient algorithm to learn sparse representations.
Deep learning via Hessian-free optimization
714 Citations2010James Martens
A 2nd-order optimization method based on the "Hessian-free" approach is developed, and applied to training deep auto-encoders, and results superior to those reported by Hinton & Salakhutdinov (2006) are obtained.
Improving the speed of neural networks on CPUs
674 Citations2011Vincent Vanhoucke, Andrew Senior +1 more
This paper uses speech recognition as an example task, and shows that a real-time hybrid hidden Markov model / neural network (HMM/NN) large vocabulary system can be built with a 10× speedup over an unoptimized baseline and a 4× speed up over an aggressively optimized floating-point baseline at no cost in accuracy.
International Conference on Machine LearningBuilding high-level features using large scale unsupervised learning
667 Citations2012Marc’Aurelio Ranzato, Rajat Monga +6 more
Multimodal Learning with Deep Boltzmann Machines
626 Citations2012Nitish Srivastava, Ruslan Salakhutdinov
Stochastic Gradient Learning in Neural Networks
565 Citations1991Léon Bottou
This chapter discusses connectionist learning algorithms, which consists of minimizing a cost of the form C(w) = E(J(z,w)dP(z) where dP is an unknown probability distribution that characterizes the problem to learn, and J, the loss function, defines the learning system itself.
Proceedings of the VLDB EndowmentAn architecture for parallel topic models
431 Citations2010Alexander J. Smola, Shravan Narayanamurthy
This paper describes a high performance sampling architecture for inference of latent topic models on a cluster of workstations and shows that this architecture is entirely general and that it can be extended easily to more sophisticated latent variable models such as n-grams and hierarchies.
Deep learning via semi-supervised embedding
387 Citations2008Jason Weston, Frédéric Ratle +1 more
It is shown how nonlinear embedding algorithms popular for use with shallow semi-supervised learning techniques such as kernel methods can be applied to deep multilayer architectures, either as a regularizer at the output layer, or on each layer of the architecture.
Game theory, on-line prediction and boosting
386 Citations1996Yoav Freund, Robert E. Schapire
An algorithm for learning to play repeated games based on the on-line prediction methods of Littlestone and Warmuth is described, which yields a simple proof of von Neumann’s famous minmax theorem, as well as a provable method of approximately solving a game.
2009 IEEE Conference on Computer Vision and Pattern RecognitionLearning invariant features through topographic filter maps
275 Citations2009Koray Kavukcuoglu, Marc’Aurelio Ranzato +2 more
This work proposes a method that automatically learns feature extractors in an unsupervised fashion by simultaneously learning the filters and the pooling units that combine multiple filter outputs together.
Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation
268 Citations2012Masashi Sugiyama, Motoaki Kawanabe
Semi-Supervised Learning in Gigantic Image Collections
225 Citations2009Rob Fergus, Yair Weiss +1 more
This paper uses the convergence of the eigenvectors of the normalized graph Laplacian to eigenfunctions of weighted Laplace-Beltrami operators to obtain highly efficient approximations for semi-supervised learning that are linear in the number of images.
Semi-supervised learning of compact document representations with deep networks
223 Citations2008Marc ' Aurelio Ranzato, Martin Szummer
An algorithm to learn text document representations based on semi-supervised autoencoders that are stacked to form a deep network that can be trained efficiently on partially labeled corpora, producing very compact representations of documents, while retaining as much class information and joint word statistics as possible.
Large-scale machine learning at twitter
218 Citations2012Jimmy Lin, Alek Kolcz
A case study of Twitter's integration of machine learning tools into its existing Hadoop-based, Pig-centric analytics platform to provide predictive analytics capabilities that incorporate machine learning, focused specifically on supervised classification.
Scalable stacking and learning for building deep architectures
213 Citations2012Li Deng, Dong Yu +1 more
The Deep Stacking Network (DSN) is presented, which overcomes the problem of parallelizing learning algorithms for deep architectures and provides a method of stacking simple processing modules in buiding deep architectures, with a convex learning problem in each module.
The MIT Press eBooksMachine Learning in Non-Stationary Environments
208 Citations2012Masashi Sugiyama, Motoaki Kawanabe
This book focuses on a specific non-stationary environment known as covariate shift, in which the distributions of inputs and outputs change but the conditional distribution of outputs is unchanged, and presents machine learning theory, algorithms, and applications to overcome this variety of non-stationarity.
IEEE Transactions on Neural NetworksAdaptive Probabilistic Neural Networks for Pattern Classification in Time-Varying Environment
201 Citations2004Leszek Rutkowski
A new class of probabilistic neural networks (PNNs) working in nonstationary environment is proposed and definitions of optimality of PNNs in time-varying environment are presented, for the first time in literature,.
Springer optimization and its applicationsMultimodal Deep Learning
190 Citations2024Amirreza Shaban, Safoora Yousefi
NatureComputer science: The learning machines
189 Citations2014N.B. Jones
Using massive amounts of data to recognize photos and speech, deep-learning computers are taking a big step towards true artificial intelligence.
IEEE Transactions on Neural NetworksA Convolutional Neural Network Approach for Objective Video Quality Assessment
185 Citations2006Patrick Le Callet, Christian Viard-Gaudin +1 more
The proposed approach solves some problems inherent to objective metrics that should predict subjective quality score obtained using the single stimulus continuous quality evaluation (SSCQE) method and relies on the use of a convolutional neural network that allows a continuous time scoring of the video.
Physical Review LettersExact Solution for On-Line Learning in Multilayer Neural Networks
174 Citations1995David Saad, Sara A. Solla
An analytic solution to the problem of on-line gradient-descent learning for two-layer neural networks with an arbitrary number of hidden units in both teacher and student networks is presented.
IEEE Transactions on Neural NetworksOn-line learning algorithms for locally recurrent neural networks
164 Citations1999P. Campolucci, Aurelio Uncini +2 more
A new gradient-based procedure called recursive backpropagation (RBP) is proposed whose on-line version, causal recursive back propagation (CRBP), presents some advantages with respect to the other on- line training methods.
IEEE Transactions on Pattern Analysis and Machine IntelligenceTensor Deep Stacking Networks
158 Citations2013Brian Hutchinson, Li Deng +1 more
A sufficient depth of the T-DSN, a symmetry in the two hidden layers structure in each T- DSN block, the model parameter learning algorithm, and a softmax layer on top of T-DsN are shown to have all contributed to the low error rates observed in the experiments for all three tasks.
IEEE Transactions on Neural NetworksJust-in-Time Adaptive Classifiers—Part I: Detecting Nonstationary Changes
153 Citations2008Cesare Alippi, Manuel Roveri
This paper addresses the change detection aspect leaving the design of just-in-time adaptive classification systems to a companion paper, and two completely automatic tests for detecting nonstationarity phenomena are suggested.
International Conference on Machine LearningUnsupervised and Transfer Learning Challenge: a Deep Learning Approach
147 Citations2011Grégoire Mesnil, Yann Dauphin +11 more
This paper describes different kinds of layers the authors trained for learning representations in the setting of the Unsupervised and Transfer Learning Challenge, and the particular one-layer learning algorithms feeding a simple linear classifier with a tiny number of labeled training samples.
Cambridge University Press eBooksLarge-Scale FPGA-Based Convolutional Networks
122 Citations2011Clément Farabet, Yann LeCun +5 more
The majority of the feature extraction systems have a common structure composed of a filter bank, a nonlinear operation (quantization, winner-take-all, sparsification, normalization, and/or pointwise saturation), and finally a pooling operation (max, average, or histogramming).
Modeling High-Dimensional Discrete Data with Multi-Layer Neural Networks
110 Citations1999Yoshua Bengio, Samy Bengio
A new architecture for modeling high-dimensional data that requires resources that grow only at most as the square of the number of variables is proposed, using a multi-layer neural network to represent the joint distribution of the variables as the product of conditional distributions.
IEEE Transactions on Neural NetworksJust-in-Time Adaptive Classifiers—Part II: Designing the Classifier
81 Citations2008Cesare Alippi, Manuel Roveri
It is shown that the classification system grants consistency once the change targets the process generating the data in a new stationary state, as it is the case in many real applications.
North-Holland mathematical libraryOn-line learning processes in artificial neural networks
78 Citations1993Tom Heskes, Bert Kappen
This work studies on-line learning processes in artificial neural networks from a general point of view, and applies the results on the transitions from “twists” in two-dimensional self-organizing maps to perfectly ordered configurations.
IEEE Transactions on Neural NetworksNeural Networks for Continuous Online Learning and Control
68 Citations2006Min Chee Choy, Dipti Srinivasan +1 more
Results show that the hybrid NN model provides significant improvement in traffic conditions when evaluated against an existing traffic signal control algorithm as well as a new, continuously updated simultaneous perturbation stochastic approximation-based neural network (SPSA-NN).
Journal of Physics A Mathematical and GeneralOn-line backpropagation in two-layered neural networks
66 Citations1995Peter Riegler, Michael Biehl
This work presents an exact analysis of learning a rule by on-line gradient descent in a two-layered neural network with adjustable hidden-to-output weights (backpropagation of error) with fixed weights in the second layer.
Incremental learning in nonstationary environments with controlled forgetting
59 Citations2009Ryan Elwell, Robi Polikar
This contribution expands the analysis of the algorithm to include various ensemble pruning methods to introduce controlled forgetting, and analyzes the tradeoff between these precautions and the desire to handle recurring contexts (cyclical data).
IEEE AccessLarge-Scale Deep Belief Nets With MapReduce
56 Citations2014Kunlei Zhang, Xuewen Chen
This paper presents a distributed learning paradigm for the RBMs and the backpropagation algorithm using MapReduce, a popular parallel programming model, and demonstrates that the distributedRBMs and DBNs are amenable to large-scale data with a good performance in terms of accuracy and efficiency.
Large Margin Semi-supervised Learning
55 Citations2007Junhui Wang, Xiaotong Shen
International Journal of Artificial Intelligence ToolsKNOWLEDGE TRANSFER IN DEEP CONVOLUTIONAL NEURAL NETS
45 Citations2008Steven Gutstein, Olac Fuentes +1 more
This paper demonstrates that components of a trained deep convolutional neural net can constructively transfer information to another such net, and shows a clear advantage in relying upon transferred knowledge to learn new tasks when given small training sets, if the new tasks are sufficiently similar to the previously mastered one.
IEEE Transactions on Systems Man and Cybernetics Part C (Applications and Reviews)Online pattern classification with multiple neural network systems: an experimental study
44 Citations2003Chee Peng Lim, Robert F. Harrison
The experiments demonstrate the potentials of the proposed multiple neural network systems in offering an alternative to handle online pattern classification tasks in possibly nonstationary environments.
IEEE Transactions on Neural Networks and Learning SystemsNonstationary Source Separation Using Sequential and Variational Bayesian Learning
42 Citations2013Jen‐Tzung Chien, Hsin-Lung Hsieh
This paper presents an online learning algorithm for the Gaussian process (GP) and establishes a separation procedure in the presence of nonstationary and temporally correlated mixing coefficients and source signals and captures the evolved statistics from sequential signals according to online Bayesian learning.
Semi-Supervised Learning Using Sparse Eigenfunction Bases.
41 Citations2009K. P. Sinha, Mikhail Belkin
Machine LearningOn-line prediction and conversion strategies
34 Citations1996Nicolò Cesa‐Bianchi, Yoav Freund +2 more
This work presents a deterministic algorithm using binomial weights that has a better worst case mistake bound than the best deterministic algorithm using exponential weights and shows that it is better to use sums of binomials as weights.
IEEE Transactions on Neural NetworksOn-line learning with minimal degradation in feedforward networks
33 Citations1995Vicente Ruiz de Angulo, Carme Torras
This work explains why overtraining affects forgetting and fault tolerance, which are seen as related problems, and proposes an algorithm for solving the problem, which is called learning with minimal degradation (LMD).
Computers & Operations ResearchMultilayer neural networks: an experimental evaluation of on-line training methods
30 Citations2003Rafael Martı́, Abdellah El Fallahi
This paper presents extensive computational experiments to compare 12 "on-line" training methods over a collection of 45 functions from the literature within a short-term horizon and proposes a new method based, on the tabu search methodology, which can compete in quality with the best previous approaches.
Physical review. E, Statistical physics, plasmas, fluids, and related interdisciplinary topicsOn-line learning with adaptive back-propagation in two-layer networks
19 Citations1997Ansgar H. L. West, David Saad
Analysis of adaptive back-propagation algorithms for on-line learning in two-layer neural networks with an arbitrary number of hidden units shows that it results generally in faster training by breaking the symmetry between hidden units more efficiently and by providing faster convergence to optimal generalization than gradient descent.
Journal of Physics A Mathematical and GeneralGlobally optimal on-line learning rules for multi-layer neural networks
15 Citations1997Magnus Rattray, David Saad
A method is presented for determining the globally optimal on-line learning rule for a soft committee machine under a statistical mechanics framework that maximizes the total reduction in generalization error over the whole learning process.
Cambridge University Press eBooksMapReduce and Its Application to Massively Parallel Learning of Decision Tree Ensembles
12 Citations2011Biswanath Panda, Joshua S. Herbach +2 more
IEEE Transactions on Neural NetworksThe Rosenblatt Bayesian Algorithm Learning in a Nonstationary Environment
6 Citations2007Evaldo Araújo de Oliveira
This letter study of online learning in neural networks obtained by approximating Bayesian learning with the Rosenblatt potential in a nonstationary environment finds a good performance in the particular case of learning linearly separable rules.
Voice Search
6 Citations2011Ye‐Yi Wang, Dong Yu +2 more
Lecture notes in computer scienceOptimal Online Prediction in Adversarial Environments
3 Citations2010Peter L. Bartlett
In many prediction problems, including those that arise in computer security and computational finance, the process generating the data is best modelled as an adversary with whom the predictor competes.
…