Transformer to CNN: Label-scarce distillation for efficient text classification
arXiv (Cornell University)Published 8 September 2019Open access
Yew Ken Chia, Sam Witteveen, Martin Andrews
Citations21
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This workshop paper outlines how the proposed convolutional student architecture, having been trained by a distillation process from a large-scale model, can achieve 300x inference speedup and 39x reduction in parameter count.
Abstract
Significant advances have been made in Natural Language Processing (NLP) modelling since the beginning of 2018. The new approaches allow for accurate results, even when there is little labelled data, because these NLP models can benefit from training on both task-agnostic and task-specific unlabelled data. However, these advantages come with significant size and computational costs. This workshop paper outlines how our proposed convolutional student architecture, having been trained by a distillation process from a large-scale model, can achieve 300x inference speedup and 39x reduction in parameter count. In some cases, the student model performance surpasses its teacher on the studied tasks.
Keywords
Computer Science
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
arXiv (Cornell University)Distributed Representations of Words and Phrases and their Compositionality
18,086 Citations2013Tomáš Mikolov, Ilya Sutskever +3 more
This paper presents a simple method for finding phrases in text, and shows that learning good vector representations for millions of phrases is possible and describes a simple alternative to the hierarchical softmax called negative sampling.
arXiv (Cornell University)Distilling the Knowledge in a Neural Network
13,965 Citations2015Geoffrey E. Hinton, Oriol Vinyals +1 more
This work shows that it can significantly improve the acoustic model of a heavily used commercial system by distilling the knowledge in an ensemble of models into a single model and introduces a new type of ensemble composed of one or more full models and many specialist models which learn to distinguish fine-grained classes that the full models confuse.
Lecture notes in computer scienceText categorization with Support Vector Machines: Learning with many relevant features
7,925 Citations1998Thorsten Joachims
SVMs achieve substantial improvements over the currently best performing methods and behave robustly over a variety of di-erent learning tasks, eliminating the need for manual parameter tuning.
Hierarchical Attention Networks for Document Classification
4,829 Citations2016Zichao Yang, Diyi Yang +4 more
Experiments conducted on six large scale text classification tasks demonstrate that the proposed architecture outperform previous methods by a substantial margin.
Deep Contextualized Word Representations
1,785 Citations2018Matthew E. Peters, Mark E Neumann +5 more
A new type of deep contextualized word representation is introduced that models both complex characteristics of word use and how these uses vary across linguistic contexts, allowing downstream models to mix different types of semi-supervision signals.
arXiv (Cornell University)Convolutional Neural Networks for Sentence Classification
1,116 Citations2014Yoon Kim
Neural Machine Translation of Rare Words with Subword Units
446 Citations2016Rico Sennrich, Barry Haddow +1 more
This paper introduces a simpler and more effective approach, making the NMT model capable of open-vocabulary translation by encoding rare and unknown words as sequences of subword units, and empirically shows that subword models improve over a back-off dictionary baseline for the WMT 15 translation tasks English-German and English-Russian by 1.3 BLEU.
Joint Embedding of Words and Labels for Text Classification
429 Citations2018Guoyin Wang, Chunyuan Li +6 more
This work proposes to view text classification as a label-word joint embedding problem: each label is embedded in the same space with the word vectors, and introduces an attention framework that measures the compatibility of embeddings between text sequences and labels.
arXiv (Cornell University)Recurrent neural network for text classification with multi-task learning
177 Citations2016Pengfei Liu, Xipeng Qiu +1 more
This paper uses the multi-task learning framework to jointly learn across multiple related tasks based on recurrent neural network to propose three different mechanisms of sharing information to model text with task-specific and shared layers.
arXiv (Cornell University)Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction
53 Citations2018Maha Elbayad, Laurent Besacier +1 more
arXiv (Cornell University)Semi-Supervised Sequence Modeling with Cross-View Training
20 Citations2018Kevin B. Clark, Minh-Thang Luong +2 more