Domain adaptation for large-scale sentiment classification: A deep learning approach
Published 1 January 2011
Xavier Glorot, Antoine Bordes, Yoshua Bengio
Citations1,563
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A deep learning approach is proposed which learns to extract a meaningful representation for each review in an unsupervised fashion and clearly outperform state-of-the-art methods on a benchmark composed of reviews of 4 types of Amazon products.
Abstract
The exponential increase in the availability of online reviews and recommendations makes sentiment classification an interesting topic in academic and industrial research. Reviews can span so many different domains that it is difficult to gather annotated training data for all of them. Hence, this paper studies the problem of domain adaptation for sentiment classifiers, hereby a system is trained on labeled reviews from one source domain but is meant to be deployed on another. We propose a deep learning approach which learns to extract a meaningful representation for each review in an unsupervised fashion. Sentiment classifiers trained with this high-level feature representation clearly outperform state-of-the-art methods on a benchmark composed of reviews of 4 types of Amazon products. Furthermore, this method scales well and allowed us to successfully perform domain adaptation on a larger industrial-strength dataset of 22 domains. 1.
Keywords
Computer Science
ScienceReducing the Dimensionality of Data with Neural Networks
20,953 Citations2006Geoffrey E. Hinton, Ruslan Salakhutdinov
Neural ComputationA Fast Learning Algorithm for Deep Belief Nets
16,404 Citations2006Geoffrey E. Hinton, Simon Osindero +1 more
A fast, greedy algorithm is derived that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory.
International Conference on Machine LearningRectified Linear Units Improve Restricted Boltzmann Machines
13,244 Citations2010Vinod Nair, Geoffrey E. Hinton
Restricted Boltzmann machines were developed using binary stochastic hidden units that learn features that are better for object recognition on the NORB dataset and face verification on the Labeled Faces in the Wild dataset.
Extracting and composing robust features with denoising autoencoders
7,343 Citations2008Pascal Vincent, Hugo Larochelle +2 more
This work introduces and motivate a new training principle for unsupervised learning of a representation based on the idea of making the learned representations robust to partial corruption of the input pattern.
Thumbs up?
6,987 Citations2002Bo Pang, Lillian Lee +1 more
This work considers the problem of classifying documents not by topic, but by overall sentiment, e.g., determining whether a review is positive or negative, and concludes by examining factors that make the sentiment classification problem more challenging.
now publishers, Inc. eBooksOpinion Mining and Sentiment Analysis
6,745 Citations2008Bo Pang, Lillian Lee
Deep Sparse Rectifier Neural Networks
5,408 Citations2012Xavier Glorot, Antoine Bordes +1 more
This paper shows that rectifying neurons are an even better model of biological neurons and yield equal or better performance than hyperbolic tangent networks in spite of the hard non-linearity and non-dierentiabil ity.
A unified architecture for natural language processing
5,204 Citations2008Ronan Collobert, Jason Weston
This work describes a single convolutional neural network architecture that, given a sentence, outputs a host of language processing predictions: part-of-speech tags, chunks, named entity tags, semantic roles, semantically similar words and the likelihood that the sentence makes sense using a language model.
The MIT Press eBooksGreedy Layer-Wise Training of Deep Networks
4,704 Citations2007Yoshua Bengio, Pascal Lamblin +2 more
These experiments confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.
Biographies, Bollywood, Boom-boxes and Blenders: Domain Adaptation for Sentiment Classification
2,026 Citations2007John Blitzer, Mark Dredze +1 more
This work extends to sentiment classification the recently-proposed structural correspondence learning (SCL) algorithm, reducing the relative error due to adaptation between domains by an average of 30% over the original SCL algorithm and 46% over a supervised baseline.
The MIT Press eBooksAnalysis of Representations for Domain Adaptation
2,023 Citations2007Shai Ben-David, John Blitzer +2 more
The theory illustrates the tradeoffs inherent in designing a representation for domain adaptation and gives a new justification for a recently proposed model which explicitly minimizes the difference between the source and target domains, while at the same time maximizing the margin of the training set.
Domain adaptation with structural correspondence learning
1,562 Citations2006John Blitzer, Ryan McDonald +1 more
This work introduces structural correspondence learning to automatically induce correspondences among features from different domains in order to adapt existing models from a resource-rich source domain to aresource-poor target domain.
Self-taught learning
1,533 Citations2007Rajat Raina, Alexis Battle +3 more
An approach to self-taught learning that uses sparse coding to construct higher-level features using the unlabeled data to form a succinct input representation and significantly improve classification performance.
International Journal of Approximate ReasoningSemantic hashing
1,272 Citations2008Ruslan Salakhutdinov, Geoffrey E. Hinton
A deep graphical model of the word-count vectors obtained from a large set of documents is shown, which achieves higher accuracy than applying TF-IDF to the entire document set.
Neural ComputationA Connection Between Score Matching and Denoising Autoencoders
980 Citations2011Pascal Vincent
A proper probabilistic model for the denoising autoencoder technique is defined, which makes it in principle possible to sample from them or rank examples by their energy, and a different way to apply score matching that is related to learning to denoise and does not require computing second derivatives is suggested.
Journal of Artificial Intelligence ResearchDomain Adaptation for Statistical Classifiers
831 Citations2006Hal Daumé, Daniel Marcu
This work introduces a statistical formulation of this problem in terms of a simple mixture model and presents an instantiation of this framework to maximum entropy classifiers and their linear chain counterparts and leads to improved performance on three real world tasks on four different data sets from the natural language processing domain.
Cross-domain sentiment classification via spectral feature alignment
789 Citations2010Sinno Jialin Pan, Xiaochuan Ni +3 more
This work develops a general solution to sentiment classification when the authors do not have any labels in a target domain but have some labeled data in a different domain, regarded as source domain and proposes a spectral feature alignment (SFA) algorithm to align domain-specific words from different domains into unified clusters, with the help of domain-independent words as a bridge.
Singapore Management University Institutional Knowledge (InK) (Singapore Management University)Instance Weighting for Domain Adaptation in NLP
754 Citations2007Jing Jiang, ChengXiang Zhai
This paper formally analyze and characterize the domain adaptation problem from a distributional view, and shows that there are two distinct needs for adaptation, corresponding to the different distributions of instances and classification functions in the source and the target domains.
Get out the vote
497 Citations2006Matt Thomas, Bo Pang +1 more
It is found that the incorporation of sources of information regarding relationships between discourse segments, such as whether a given utterance indicates agreement with the opinion expressed by another, yields substantial improvements over classifying speeches in isolation.
Measuring Invariances in Deep Networks
344 Citations2009Ian Goodfellow, Honglak Lee +3 more
A number of empirical tests are proposed that directly measure the degree to which these learned features are invariant to different input transformations and find that stacked autoencoders learn modestly increasingly invariant features with depth when trained on natural images and convolutional deep belief networks learn substantially more invariant Features in each layer.
Multiple Aspect Ranking Using the Good Grief Algorithm
316 Citations2007Benjamin Snyder, Regina Barzilay
An algorithm is presented that jointly learns ranking models for individual aspects by modeling the dependencies between assigned ranks, and it is proved that the agreementbased joint model is more expressive than individual ranking models.
Transferring naive bayes classifiers for text classification
313 Citations2007Wenyuan Dai, Gui-Rong Xue +2 more
This paper proposes a novel transfer-learning algorithm for text classification based on an EM-based Naive Bayes classifiers and shows that the algorithm outperforms the traditional supervised and semi-supervised learning algorithms when the distributions of the training and test sets are increasingly different.
Large scale semi-supervised linear SVMs
233 Citations2006Vikas Sindhwani, S. Sathiya Keerthi
An implementation of Transductive SVM (TSVM) that is significantly more efficient and scalable than currently used dual techniques, for linear classification problems involving large, sparse datasets, and a variant of TSVM that involves multiple switching of labels.
Semi-supervised learning of compact document representations with deep networks
223 Citations2008Marc ' Aurelio Ranzato, Martin Szummer
An algorithm to learn text document representations based on semi-supervised autoencoders that are stacked to form a deep network that can be trained efficiently on partially labeled corpora, producing very compact representations of documents, while retaining as much class information and joint word statistics as possible.
Multi-domain adaptation for sentiment classification: Using multiple classifier combining methods
22 Citations2008Shoushan Li, Chengqing Zong
This paper proposes a new combining method, called Multi-label Consensus Training (MCT), to combine the base classifiers for selecting dasiaautomatically-labeledpsila samples from unlabeled data in the target domain and shows that multi-domain adaptation using this method improves adaptation performance.