Interactively-Propagative Attention Learning for Implicit Discourse Relation Recognition
Published 1 January 2020Open access
Huibin Ruan, Yu Hong, Yang Xu, Zhen Huang, Guodong Zhou, Min Zhang
Citations23
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A propagative attention learning model using a cross-coupled two-channel network that yields substantial improvements over the baselines (BiLSTM and BERT).
Abstract
We tackle implicit discourse relation recognition. Both self-attention and interactive-attention mechanisms have been applied for attention-aware representation learning, which improves the current discourse analysis models. To take advantages of the two attention mechanisms simultaneously, we develop a propagative attention learning model using a cross-coupled two-channel network. We experiment on Penn Discourse Treebank. The test results demonstrate that our model yields substantial improvements over the baselines (BiLSTM and BERT).
Keywords
Computer Science
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)MizAR 60 for Mizar 50
76,311 Citations2023Jakubův, Jan, Chvalovský, Karel +7 more
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at Scale
45,681 Citations2018Kefallinos, Dionysios, Alexandris, Georgios +6 more
Dropout: a simple way to prevent neural networks from overfitting
34,279 Citations2014Nitish Srivastava, Geoffrey E. Hinton +3 more
It is shown that dropout improves the performance of neural networks on supervised learning tasks in vision, speech recognition, document classification and computational biology, obtaining state-of-the-art results on many benchmark data sets.
Glove: Global Vectors for Word Representation
33,769 Citations2014Jeffrey Pennington, Richard Socher +1 more
A new global logbilinear regression model that combines the advantages of the two major model families in the literature: global matrix factorization and local context window methods and produces a vector space with meaningful substructure.
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
IEEE Transactions on Signal ProcessingBidirectional recurrent neural networks
10,022 Citations1997Mike Schuster, Kuldip K. Paliwal
It is shown how the proposed bidirectional structure can be easily modified to allow efficient estimation of the conditional posterior probability of complete symbol sequences without making any explicit assumption about the shape of the distribution.
ACM Transactions on Information SystemsCumulated gain-based evaluation of IR techniques
4,634 Citations2002Kalervo Järvelin, Jaana Kekäläinen
This article proposes several novel measures that compute the cumulative gain the user obtains by examining the retrieval result up to a given ranked position, and test results indicate that the proposed measures credit IR methods for their ability to retrieve highly relevant documents and allow testing of statistical significance of effectiveness differences.
arXiv (Cornell University)A Structured Self-attentive Sentence Embedding
1,458 Citations2017Zhouhan Lin, Minwei Feng +5 more
The Penn Discourse TreeBank 2.0.
1,130 Citations2008Rashmi Prasad, Nikhil Dinesh +5 more
The second version of the Penn Discourse Treebank, PDTB-2.0, is presented, describing its lexically-grounded annotations of discourse relations and their two abstract object arguments over the 1 million word Wall Street Journal corpus.
Journal of Wildlife ManagementThe Insignificance of Statistical Significance Testing
1,107 Citations1999Douglas H. Johnson
This paper describes how statistical hypothesis tests are often viewed, and then contrasts that interpretation with the correct one, and discusses the arbitrariness of P-values, conclusions that the null hypothesis is true, power analysis, and distinctions between statistical and biological significance.
Interactive Attention Networks for Aspect-Level Sentiment Classification
1,014 Citations2017Dehong Ma, Sujian Li +2 more
It is argued that both targets and contexts deserve special treatment and need to be learned their own representations via interactive learning and proposed the interactive attention networks (IAN) to interactively learn attentions in the contexts and targets.
An unsupervised approach to recognizing discourse relations
384 Citations2001Daniel Marcu, Abdessamad Echihabi
It is shown that discourse relation classifiers trained on examples that are automatically extracted from massive amounts of text can be used to distinguish between some of these relations with accuracies as high as 93%, even when the relations are not explicitly marked by cue phrases.
International Conference on Learning RepresentationsA Structured Self-Attentive Sentence Embedding.
364 Citations2017Zhouhan Lin, Minwei Feng +5 more
The Hitchhiker’s Guide to Testing Statistical Significance in Natural Language Processing
350 Citations2018Rotem Dror, Gili Baumer +2 more
This opinion/ theoretical paper proposes a simple practical protocol for statistical significance test selection in NLP setups and accompanies this protocol with a brief survey of the most relevant tests.
Automatic sense prediction for implicit discourse relations in text
285 Citations2009Emily Pitler, Annie Louis +1 more
A series of experiments on automatically identifying the sense of implicit discourse relations, i.e. relations that are not marked with a discourse connective such as "but" or "because", are presented.
Recognizing implicit discourse relations in the Penn Discourse Treebank
283 Citations2009Ziheng Lin, Min‐Yen Kan +1 more
An implicit discourse relation classifier is presented in the Penn Discourse Treebank that considers the context of the two arguments, word pair information, as well as the arguments' internal constituent and dependency parses.
Natural Language EngineeringA PDTB-styled end-to-end discourse parser
266 Citations2012Ziheng Lin, Hwee Tou Ng +1 more
This work designed and developed an end-to-end discourse parser-to-parse free texts in the PDTB style in a fully data-driven approach and proposes and presents a comprehensive evaluation from both component-wise and error-cascading perspectives.
Shallow Convolutional Neural Network for Implicit Discourse Relation Recognition
118 Citations2015Biao Zhang, Jinsong Su +4 more
A Shallow Convolutional Neural Network (SCNN) is proposed, which contains only one hidden layer but is effective in relation recognition, which alleviates the overfitting problem, while the convolution and nonlinear operations help preserve the recognition and generalization ability of the model.
Multi-task Attention-based Neural Networks for Implicit Discourse Relationship Representation and Identification
107 Citations2017Man Lan, Jianxiang Wang +3 more
A novel multi-task attention based neural network model to address implicit discourse relationship representation and identification through two types of representation learning and a multi- task framework for learning knowledge from annotated and unannotated corpora is presented.
Recognizing Implicit Discourse Relations via Repeated Reading: Neural Networks with Multi-Level Attention
92 Citations2016Yang Liu, Sujian Li
This work proposes the neural networks with multi-level attention (NNMA), combining the attention mechanism and external memories to gradually fix the attention on some specific words helpful to judging the discourse relations.
Proceedings of the AAAI Conference on Artificial IntelligenceImplicit Discourse Relation Classification via Multi-Task Neural Networks
89 Citations2016Yang Liu, Sujian Li +2 more
This work designs related discourse classification tasks specific to a corpus, and proposes a novel Convolutional Neural Network embedded multi-task learning system to synthesize these tasks by learning both unique and shared representations for each task.
Adversarial Connective-exploiting Networks for Implicit Discourse Relation Classification
88 Citations2017Lianhui Qin, Zhisong Zhang +3 more
This work develops an adversarial model to enable an adaptive imitation scheme through competition between the implicit network and a rival feature discriminator, and achieves state-of-the-art performance on the PDTB benchmark.
Implicit Discourse Relation Detection via a Deep Architecture with Gated Relevance Network
86 Citations2016Jifan Chen, Qi Zhang +3 more
The use of word embeddings to replace the original words is proposed to overcome the data sparsity problem, and a gated relevance network is adopted to capture the semantic interaction between word pairs.
arXiv (Cornell University)Implicit Discourse Relation Classification via Multi-Task Neural Networks
75 Citations2016Yang Liu, Sujian Li +2 more
arXiv (Cornell University)Deep Enhanced Representation for Implicit Discourse Relation Recognition
70 Citations2018Hongxiao Bai, Hai Zhao
Improving Implicit Discourse Relation Classification by Modeling Inter-dependencies of Discourse Units in a Paragraph
63 Citations2018Zeyu Dai, Ruihong Huang
A paragraph-level neural networks that model inter-dependencies between discourse units as well as discourse relation continuity and patterns, and predict a sequence of discourse relations in a paragraph are introduced.
International Conference on Computational LinguisticsDeep Enhanced Representation for Implicit Discourse Relation Recognition
55 Citations2018Hongxiao Bai, Hai Zhao
A model augmented with different grained text representations, including character, subword, word, sentence, and sentence pair levels is proposed, which achieves state-of-the-art accuracy with greater than 48% in 11-way and F1 score greater than 50% in 4-way classifications for the first time.
Proceedings of the AAAI Conference on Artificial IntelligenceLinguistic Properties Matter for Implicit Discourse Relation Recognition: Combining Semantic Interaction, Topic Continuity and Attribution
48 Citations2018Wenqiang Lei, Yuanxin Xiang +4 more
This model is built from first principles, analyzing the linguistic properties of the individual top-level Penn Discourse Treebank styled implicit discourse relations: Comparison, Contingency and Expansion to achieve a new state-of-the-art performance level.
Employing the Correspondence of Relations and Connectives to Identify Implicit Discourse Relations via Label Embeddings
35 Citations2019Linh The Nguyen, Linh Ngo Van +2 more
This work explores a multi-task learning framework for IDRR in which the relations and the connectives are simultaneously predicted, and the mapping is leveraged to transfer knowledge between the two prediction tasks via the embeddings of relations and connectives.
International Conference on Computational LinguisticsImplicit Discourse Relation Recognition using Neural Tensor Network with Interactive Attention and Sparse Learning
34 Citations2018Fengyu Guo, Ruifang He +4 more
A novel neural Tensor network framework with Interactive Attention and Sparse Learning (TIASL) for implicit discourse relation recognition, and mines the most correlated word pairs from two discourse arguments to model pair specific clues and integrates them as interactive attention into argument representations produced by the bidirectional long short-term memory network.
TransS-Driven Joint Learning Architecture for Implicit Discourse Relation Recognition
30 Citations2020Ruifang He, Jian Wang +2 more
A novel TransS-driven joint learning architecture to address implicit discourse relation recognition, based on the multi-level encoder that translates discourse relations in low-dimensional embedding space (called TransS), which could mine the latent geometric structure information of argument-relation instances.
arXiv (Cornell University)Interactive Attention for Neural Machine Translation
24 Citations2016Fandong Meng, Zhengdong Lu +2 more