Leveraging Non-Conversational Tasks for Low Resource Slot Filling: Does it help?
Published 1 January 2019Open access
Samuel Louvan, Bernardo Magnini
Citations8
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
It is shown that using auxiliary non-conversational tasks in a multi-task learning setup consistently improves low resource slot filling performance.
Abstract
Slot filling is a core operation for utterance understanding in task-oriented dialogue systems. Slots are typically domain-specific, and adding new domains to a dialogue system involves data and time-intensive processes. A popular technique to address the problem is transfer learning, where it is assumed the availability of a large slot filling dataset for the source domain, to be used to help slot filling on the target domain, with fewer data. In this work, instead, we propose to leverage source tasks based on semantically related non-conversational resources (e.g., semantic sequence tagging datasets), as they are both cheaper to obtain and reusable to several slot filling domains. We show that using auxiliary non-conversational tasks in a multi-task learning setup consistently improves low resource slot filling performance.
Keywords
Computer Science
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)AI-Assisted Pipeline for Dynamic Generation of Trustworthy Health Supplement Content at Scale
45,681 Citations2018Kefallinos, Dionysios, Alexandris, Georgios +6 more
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Machine LearningMultitask Learning
6,312 Citations1997Rich Caruana
Prior work on MTL is reviewed, new evidence that MTL in backprop nets discovers task relatedness without the need of supervisory signals is presented, and new results for MTL with k-nearest neighbor and kernel regression are presented.
Neural Architectures for Named Entity Recognition
4,428 Citations2016Guillaume Lample, Miguel Ballesteros +3 more
Comunicacio presentada a la 2016 Conference of the North American Chapter of the Association for Computational Linguistics, celebrada a San Diego (CA, EUA) els dies 12 a 17 of juny 2016.
End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF
2,589 Citations2016Xuezhe Ma, Eduard Hovy
A novel neutral network architecture is introduced that benefits from both word- and character-level representations automatically, by using combination of bidirectional LSTM, CNN and CRF, thus making it applicable to a wide range of sequence labeling tasks.
The Berkeley FrameNet Project
2,564 Citations1998Collin F. Baker, Charles J. Fillmore +1 more
This report will present the project's goals and workflow, and information about the computational tools that have been adapted or created in-house for this work.
Computational LinguisticsThe Proposition Bank: An Annotated Corpus of Semantic Roles
2,302 Citations2005Martha Palmer, Daniel Gildea +1 more
An automatic system for semantic role tagging trained on the corpus is described and the effect on its performance of various types of information is discussed, including a comparison of full syntactic parsing with a flat representation and the contribution of the empty trace categories of the treebank.
Lecture notes in computer scienceThe PASCAL Recognising Textual Entailment Challenge
1,626 Citations2006Ido Dagan, Oren Glickman +1 more
arXiv (Cornell University)Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
1,574 Citations2002Erik F. Tjong Kim Sang
Text, speech and language technologyText Chunking Using Transformation-Based Learning
1,257 Citations1999Lance Ramshaw, Mitchell P. Marcus
This work has shown that the transformation-based learning approach can be applied at a higher level of textual interpretation for locating chunks in the tagged text, including non-recursive “baseNP” chunks.
Abstract Meaning Representation for Sembanking
1,131 Citations2013Laura Banarescu, Claire Bonial +8 more
A sembank of simple, whole-sentence semantic structures will spur new work in statistical natural language understanding and generation, like the Penn Treebank encouraged work on statistical parsing.
Institutional Research Information System (Università degli Studi di Trento)CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes
669 Citations2012Sameer Pradhan, Alessandro Moschitti +3 more
The OntoNotes annotation (coreference and other layers) is described and the parameters of the shared task including the format, pre-processing information, evaluation criteria, and presents and discusses the results achieved by the participating systems.
Slot-Gated Modeling for Joint Slot Filling and Intent Prediction
502 Citations2018Chih-Wen Goo, Guang Gao +5 more
A slot gate that focuses on learning the relationship between intent and slot attention vectors in order to obtain better semantic frame results by the global optimization is proposed.
arXiv (Cornell University)Text Chunking using Transformation-Based Learning
471 Citations1995Lance Ramshaw, Mitchell P. Marcus
IEEE/ACM Transactions on Audio Speech and Language ProcessingUsing Recurrent Neural Networks for Slot Filling in Spoken Language Understanding
465 Citations2014Grégoire Mesnil, Yann Dauphin +9 more
This paper implemented and compared several important RNN architectures, including Elman, Jordan, and hybrid variants, and implemented these networks with the publicly available Theano neural network toolkit and completed experiments on the well-known airline travel information system (ATIS) benchmark.
Deep multi-task learning with low level tasks supervised at lower layers
459 Citations2016Anders Søgaard, Yoav Goldberg
It is consistently better to have POS supervision at the innermost rather than the outermost layer, and it is argued that “lowlevel” tasks are better kept at the lower layers, enabling the higher- level tasks to make use of the shared representation of the lower-level tasks.
Investigation of recurrent-neural-network architectures and learning methods for spoken language understanding
428 Citations2013Grégoire Mesnil, Xiaodong He +2 more
The results show that on this task, both types of recurrent networks outperform the CRF baseline substantially, and a bi-directional Jordantype network that takes into account both past and future dependencies among slots works best, outperforming a CRFbased baseline by 14% in relative error reduction.
Evaluation of spoken language systems
397 Citations1990P. Price
This paper will address the emerging standards for evaluation of spoken language systems with quantifiable measures essential for comparing results and assessing claims.
Institutional Research Information System (Università degli Studi di Trento)Towards Robust Linguistic Analysis using OntoNotes
362 Citations2013Sameer Pradhan, Alessandro Moschitti +6 more
An analysis of the performance of publicly available, state-of-the-art tools on all layers and languages in the OntoNotes v5.0 corpus should set the benchmark for future development of various NLP components in syntax and semantics, and possibly encourage research towards an integrated system that makes use of the various layers jointly to improve overall performance.
The third PASCAL recognizing textual entailment challenge
360 Citations2007Danilo Giampiccolo, Bernardo Magnini +2 more
How Transferable are Neural Networks in NLP Applications?
269 Citations2016Lili Mou, Meng Zhao +5 more
In this paper, systematic case studies are conducted and an illuminating picture is provided on the transferability of neural networks in NLP.
arXiv (Cornell University)Transfer Learning for Sequence Tagging with Hierarchical Recurrent\n Networks
232 Citations2017Zhilin Yang, Ruslan Salakhutdinov +1 more
A joint model of intent determination and slot filling for spoken language understanding
228 Citations2016Xiaodong Zhang, Houfeng Wang
A joint model is proposed based on the idea that the intent and semantic slots of a sentence are correlative, and it outperforms the state-of-the-art approaches on both tasks.
arXiv (Cornell University)Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks
218 Citations2017Zhilin Yang, Ruslan Salakhutdinov +1 more
The effects of transfer learning for deep hierarchical recurrent networks across domains, applications, and languages are examined, and it is shown that significant improvement can often be obtained.
Language Resources and EvaluationA massively parallel corpus: the Bible in 100 languages
206 Citations2014Christos Christodouloupoulos, Mark Steedman
The creation of a massively parallel corpus based on 100 translations of the Bible and a statistical analysis of the corpora collected and a detailed comparison between the English translation and other English corpora are presented.
A Bi-Model Based RNN Semantic Frame Parsing Model for Intent Detection and Slot Filling
205 Citations2018Yu Wang, Yilin Shen +1 more
New Bi-model based RNN semantic frame parsing network structures are designed to perform the intent detection and slot filling tasks jointly, by considering their cross-impact to each other using two correlated bidirectional LSTMs (BLSTM).
A Hierarchical Multi-Task Approach for Learning Embeddings from Semantic Tasks
203 Citations2019Victor Sanh, Thomas Wolf +1 more
Identifying beneficial task relations for multi-task learning in deep neural networks
200 Citations2017Joachim Bingel, Anders Søgaard
Light is shed on the specific task relations that can lead to gains from MTL models over single-task setups in deep neural networks for NLP.
The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representations
123 Citations2017Lasha Abzianidze, Johannes Bjerva +6 more
The approach is based on cross-lingual projection: automatically produced (and manually corrected) semantic annotations for English sentences are mapped onto their word-aligned translations, assuming that the translations are meaning-preserving.
Asgard: A portable architecture for multilingual dialogue systems
100 Citations2013Jun Liu, Panupong Pasupat +2 more
This work investigates the portability issue from the language understanding perspective and presents the Asgard architecture, a CRF-based (Conditional Random Fields) and crowd-sourcing-centered framework, which supports expert-free development of multilingual dialogue systems and seamless deployment to mobile platforms.
Joint Online Spoken Language Understanding and Language Modeling With Recurrent Neural Networks
94 Citations2016Bing Liu, Ian Lane
A recurrent neural network model that jointly performs intent detection, slot filling, and language modeling is described that outperforms the independent task training model on SLU tasks and shows advantageous performance in the realistic ASR settings with noisy speech input.
University of Groningen research database (University of Groningen / Centre for Information Technology)When is multitask learning effective? Semantic sequence prediction under varying data conditions
82 Citations2017Hėctor Martínez Alonso, Barbara Plank
Domain Attention with an Ensemble of Experts
67 Citations2017Young‐Bum Kim, Karl Stratos +1 more
A weighted combination of the K domain experts’ feedback along with its own opinion to make predictions on the new domain is described, which significantly outperforms baselines that do not use domain adaptation and also performs better than the full retraining approach.
Syntax or semantics? knowledge-guided joint semantic frame parsing
52 Citations2016Yun-Nung Chen, Dilek Hakanni-Tur +4 more
This paper proposes to apply knowledge-guided structural attention networks (K-SAN), which additionally incorporate non-flat network topologies guided by prior knowledge, to a language understanding task, and shows that the proposed K-SAN models with syntax or semantics outperform the state-of-the-art neural network based results.
SRL4ORL: Improving Opinion Role Labeling Using Multi-Task Learning with Semantic Role Labeling
38 Citations2018Ana Marasović, Anette Frank
It is found that the vanilla MTL model, which makes predictions using only shared ORL and SRL features, performs the best, and two MTL models improve significantly over the single-task model for labeling of both holders and targets, on the development and the test sets.
Proceedings of the AAAI Conference on Artificial IntelligenceDeep Cascade Multi-Task Learning for Slot Filling in Online Shopping Assistant
36 Citations2019Yu Gong, Xusheng Luo +7 more
This work proposes a novel multi-task model with cascade and residual connections, which jointly learns segment tagging, named entity tagging and slot filling and has a 14.6% advantage in F1 score over the strong baseline methods on a new Chinese E-commerce shopping assistant dataset.
arXiv (Cornell University)Why Comparing Single Performance Scores Does Not Allow to Draw Conclusions About Machine Learning Approaches
32 Citations2018Nils Reimers, Iryna Gurevych
There is a high risk that a statistical significance in this type of evaluation is not due to a superior learning approach, and there is ahigh risk that the difference is due to chance.
arXiv (Cornell University)Towards Universal Semantic Tagging
26 Citations2017Lasha Abzianidze, Johan Bos
Bag of Experts Architectures for Model Reuse in Conversational Language Understanding
23 Citations2018Rahul Jha, Alex Marin +2 more
This work describes Bag of Experts (BoE) architectures for model reuse for both LSTM and CRF based models for slot tagging and shows that these models outperform the baseline models with a statistically significant average margin of 5.06% in absolute F1-score.
TUbilio (Technical University of Darmstadt)Why Comparing Single Performance Scores Does Not Allow to Draw\n Conclusions About Machine Learning Approaches
22 Citations2018Nils Reimers, Iryna Gurevych
International Conference on Computational LinguisticsDomainless Adaptation by Constrained Decoding on a Schema Lattice.
14 Citations2016Young‐Bum Kim, Karl Stratos +1 more
A framework that allows us to have a single model that can handle all domains: including unknown domains that may be created in the future as long as they are covered in the master schema is introduced.
arXiv (Cornell University)SRL4ORL: Improving Opinion Role Labelling using Multi-task Learning with Semantic Role Labeling
2 Citations2017Ana Marasović, Anette Frank
Two MTL models improve significantly over the single-task model for labeling of both holders and targets, on the development and the test sets and it is found that the vanilla MTL model, which makes predictions using only shared ORL and SRL features, performs the best.
Accademia University Press eBooksFrom General to Specific: Leveraging Named Entity Recognition for Slot Filling in Conversational Language Understanding
1 Citations2018Samuel Louvan, Bernardo Magnini
Multi-task learning is used to leverage general knowledge of a task to improve slot filling performance on a semantically similar domain-specific task and show that, for some datasets, transfer learning from NER can achieve competitive performance compared with the state-of-the-art.