Don’t Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization
Published 1 January 2018Open access
Shashi Narayan, Shay B. Cohen, Mirella Lapata
Citations1,197
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A novel abstractive model is proposed which is conditioned on the article’s topics and based entirely on convolutional neural networks, outperforming an oracle extractive system and state-of-the-art abstractive approaches when evaluated automatically and by humans.
Abstract
We introduce extreme summarization, a new single-document summarization task which does not favor extractive strategies and calls for an abstractive modeling approach. The idea is to create a short, one-sentence news summary answering the question "What is the article about?". We collect a real-world, large scale dataset for this task by harvesting online articles from the British Broadcasting Corporation (BBC). We propose a novel abstractive model which is conditioned on the article's topics and based entirely on convolutional neural networks. We demonstrate experimentally that this architecture captures longrange dependencies in a document and recognizes pertinent content, outperforming an oracle extractive system and state-of-the-art abstractive approaches when evaluated automatically and by humans. 1
Keywords
Computer Science
Deep Residual Learning for Image Recognition
222,082 Citations2016Kaiming He, Xiangyu Zhang +2 more
This work presents a residual learning framework to ease the training of networks that are substantially deeper than those used previously, and provides comprehensive empirical evidence showing that these residual networks are easier to optimize, and can gain accuracy from considerably increased depth.
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
arXiv (Cornell University)Neural Machine Translation by Jointly Learning to Align and Translate
14,565 Citations2014Dzmitry Bahdanau
Adaptive Subgradient Methods for Online Learning and Stochastic Optimization
8,612 Citations2011John C. Duchi, Elad Hazan +1 more
Get To The Point: Summarization with Pointer-Generator Networks
3,813 Citations2017Abigail See, Peter J. Liu +1 more
A novel architecture that augments the standard sequence-to-sequence attentional model in two orthogonal ways, using a hybrid pointer-generator network that can copy words from the source text via pointing, which aids accurate reproduction of information, while retaining the ability to produce novel words through the generator.
arXiv (Cornell University)On the difficulty of training Recurrent Neural Networks
3,778 Citations2012Razvan Pascanu, Tomáš Mikolov +1 more
This paper proposes a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem and validates empirically the hypothesis and proposed solutions.
On the importance of initialization and momentum in deep learning
3,526 Citations2013Ilya Sutskever, James Martens +2 more
It is shown that when stochastic gradient descent with momentum uses a well-designed random initialization and a particular type of slowly increasing schedule for the momentum parameter, it can train both DNNs and RNNs to levels of performance that were previously achievable only with Hessian-Free optimization.
Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond
2,209 Citations2016Ramesh Nallapati, Bowen Zhou +3 more
This work proposes several novel models that address critical problems in summarization that are not adequately modeled by the basic architecture, such as modeling key-words, capturing the hierarchy of sentence-to-word structure, and emitting words that are rare or unseen at training time.
Oxford University Research Archive (ORA) (University of Oxford)Teaching Machines to Read and Comprehend
1,936 Citations2015Karl Moritz Hermann, Tomáš Kočiský +5 more
arXiv (Cornell University)Convolutional Sequence to Sequence Learning
1,897 Citations2017Jonas Gehring, Michael Auli +3 more
This work introduces an architecture based entirely on convolutional neural networks, which outperform the accuracy of the deep LSTM setup of Wu et al. (2016) on both WMT'14 English-German and WMT-French translation at an order of magnitude faster speed, both on GPU and CPU.
Automatic evaluation of summaries using N-gram co-occurrence statistics
1,588 Citations2003Chin-Yew Lin, Eduard Hovy
The results show that automatic evaluation using unigram co-occurrences between summary pairs correlates surprising well with human evaluations, based on various statistical metrics; while direct application of the BLEU evaluation procedure does not always give good results.
arXiv (Cornell University)Teaching Machines to Read and Comprehend
1,526 Citations2015Karl Moritz Hermann, Tomáš Kočiský +5 more
A new methodology is defined that resolves this bottleneck and provides large scale supervised reading comprehension data that allows a class of attention based deep neural networks that learn to read real documents and answer complex questions with minimal prior knowledge of language structure to be developed.
International Conference on Machine LearningConvolutional Sequence to Sequence Learning
1,340 Citations2017Jonas Gehring, Michael Auli +3 more
arXiv (Cornell University)A Deep Reinforced Model for Abstractive Summarization
1,272 Citations2017Romain Paulus, Caiming Xiong +1 more
arXiv (Cornell University)Language Modeling with Gated Convolutional Networks
1,129 Citations2016Yann Dauphin, Angela Fan +2 more
Hierarchical Neural Story Generation
1,124 Citations2018Angela Fan, Mike Lewis +1 more
This work collects a large dataset of 300K human-written stories paired with writing prompts from an online forum that enables hierarchical story generation, where the model first generates a premise, and then transforms it into a passage of text.
Proceedings of the AAAI Conference on Artificial IntelligenceSummaRuNNer: A Recurrent Neural Network Based Sequence Model for Extractive Summarization of Documents
958 Citations2017Ramesh Nallapati, Feifei Zhai +1 more
SummaRuNNer, a Recurrent Neural Network (RNN) based sequence model for extractive summarization of documents and show that it achieves performance better than or comparable to state-of-the-art.
International Conference on Machine LearningLanguage modeling with gated convolutional networks
884 Citations2017Yann Dauphin, Angela Fan +2 more
Natural language processingAutomatic Summarization
839 Citations2001Inderjeet Mani
The challenges that remain open, in particular the need for language generation and deeper semantic understanding of language that would be necessary for future advances in the field are discussed.
arXiv (Cornell University)SummaRuNNer: A Recurrent Neural Network based Sequence Model for Extractive Summarization of Documents
751 Citations2016Ramesh Nallapati, Feifei Zhai +1 more
Neural Summarization by Extracting Sentences and Words
747 Citations2016Jianpeng Cheng, Mirella Lapata
This work develops a general framework for single-document summarization composed of a hierarchical document encoder and an attention-based extractor that allows for different classes of summarization models which can extract sentences or words.
A Neural Attention Model for Abstractive Sentence Summarization
703 Citations2015Alexander M. Rush, Sumit Chopra +1 more
This work proposes a fully data-driven approach to abstractive sentence summarization by utilizing a local attention-based model that generates each word of the summary conditioned on the input sentence.
arXiv (Cornell University)End-To-End Memory Networks
689 Citations2015Sainbayar Sukhbaatar, Arthur Szlam +2 more
A neural network with a recurrent attention model over a possibly large external memory that is trained end-to-end, and hence requires significantly less supervision during training, making it more generally applicable in realistic settings.
Context dependent recurrent neural network language model
574 Citations2012Tomáš Mikolov, Geoffrey Zweig
This paper improves recurrent neural network language models performance by providing a contextual real-valued input vector in association with each word to convey contextual information about the sentence being modeled by performing Latent Dirichlet Allocation using a block of preceding text.
Best-Worst Scaling: Theory, Methods and Applications
450 Citations2015Jordan J. Louviere, Terry N. Flynn +1 more
A Convolutional Encoder Model for Neural Machine Translation
435 Citations2017Jonas Gehring, Michael Auli +2 more
A faster and simpler architecture based on a succession of convolutional layers that allows to encode the source sentence simultaneously compared to recurrent networks for which computation is constrained by temporal dependencies is presented.
Abstractive Document Summarization with a Graph-Based Attentional Neural Model
353 Citations2017Jiwei Tan, Xiaojun Wan +1 more
A novel graph-based attention mechanism in the sequence-to-sequence framework to address the saliency factor of summarization, which has been overlooked by prior works and is competitive with state-of-the-art extractive methods.
Controllable Abstractive Summarization
273 Citations2018Angela Fan, David Grangier +1 more
A neural summarization model with a simple but effective mechanism to enable users to specify high level attributes in order to control the shape of the final summaries to better suit their needs.
arXiv (Cornell University)Contextual LSTM (CLSTM) models for Large scale NLP tasks
190 Citations2016Shalini Ghosh, Oriol Vinyals +4 more
Results from experiments indicate that using both words and topics as features improves performance of the CLSTM models over baseline L STM models for these tasks, demonstrating the significant benefit of using context appropriately in natural language (NL) tasks.
Columbia Academic Commons (Columbia University)Automatic Text Summarization of Newswire: Lessons Learned from the Document Understanding Conference
156 Citations2005Ani Nenkova
An overview of the achieved results in the different types of summarization tasks, comparing both the broader classes of baselines, systems and humans, as well as individual pairs of summarizers (both human and automatic).
arXiv (Cornell University)Hierarchical Neural Story Generation
151 Citations2018Angela Fan, Mike Lewis +1 more
Learning-Based Single-Document Summarization with Compression and Anaphoricity Constraints
147 Citations2016Greg Durrett, Taylor Berg-Kirkpatrick +1 more
A discriminative model for single-document summarization that integrally combines compression and anaphoricity constraints that outperforms prior work on both ROUGE as well as on human judgments of linguistic quality.
arXiv (Cornell University)TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency
129 Citations2016Adji Bousso Dieng, Chong Wang +2 more
The limits of automatic summarisation according to ROUGE
98 Citations2017Natalie Schluter
This paper discusses some central caveats of summarisation, incurred in the use of the ROUGE metric for evaluation, with respect to optimal solutions, and shows empirically for three central benchmark datasets for the task, greedy algorithms empirically seem to perform optimally according to the metric.
Computational LinguisticsDiscourse Constraints for Document Compression
89 Citations2010James Clarke, Mirella Lapata
A discourse-informed model which is capable of producing document compressions that are coherent and informative is presented, inspired by theories of local coherence and formulated within the framework of integer linear programming.
International Joint Conference on Artificial IntelligenceDistraction-based neural networks for modeling documents
79 Citations2016Chen Qian, Xiaodan Zhu +3 more
This paper proposes neural models to train computers not just to pay attention to specific regions and content of input documents with attention models, but also distract them to traverse between different content of a document so as to better grasp the overall meaning for summarization.
arXiv (Cornell University)Neural Extractive Summarization with Side Information
68 Citations2017Shashi Narayan, Nikos Papasarantopoulos +2 more
It is shown that extractive summarization with side information consistently outperforms its counterpart that does not use any side information, in terms of both informativeness and fluency.
Document Modeling with External Attention for Sentence Extraction
32 Citations2018Shashi Narayan, Ronald Cardenas +5 more