A Unified Generative Framework for Various NER Subtasks
Published 1 January 2021Open access
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, Xipeng Qiu
Citations257
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work proposes to formulate the NER subtasks as an entity span sequence generation task, which can be solved by a unified sequence-to-sequence (Seq2Seq) framework, and can leverage the pre-trained Seq1Seq model to solve all three kinds of N ER subtasks without the special design of the tagging schema or ways to enumerate spans.
Abstract
Hang Yan, Tao Gui, Junqi Dai, Qipeng Guo, Zheng Zhang, Xipeng Qiu. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). 2021.
Keywords
Computer Science
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)MizAR 60 for Mizar 50
76,311 Citations2023Jakubův, Jan, Chvalovský, Karel +7 more
32,525 Citations2019Jacob Devlin, Ming‐Wei Chang +2 more
A new language representation model, BERT, designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers, which can be fine-tuned with just one additional output layer to create state-of-the-art models for a wide range of tasks.
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation
24,447 Citations2014Kyunghyun Cho, Bart van Merriënboer +5 more
Qualitatively, the proposed RNN Encoder‐Decoder model learns a semantically and syntactically meaningful representation of linguistic phrases.
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
12,978 Citations2001John Lafferty, Andrew McCallum +1 more
This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
Effective Approaches to Attention-based Neural Machine Translation
8,582 Citations2015Thang Luong, Hieu Pham +1 more
A global approach which always attends to all source words and a local one that only looks at a subset of source words at a time are examined, demonstrating the effectiveness of both approaches on the WMT translation tasks between English and German in both directions.
arXiv (Cornell University)Exploring the Limits of Transfer Learning with a Unified Text-to-Text\n Transformer
8,313 Citations2019Colin Raffel, Noam Shazeer +7 more
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Neural Architectures for Named Entity Recognition
4,428 Citations2016Guillaume Lample, Miguel Ballesteros +3 more
Comunicacio presentada a la 2016 Conference of the North American Chapter of the Association for Computational Linguistics, celebrada a San Diego (CA, EUA) els dies 12 a 17 of juny 2016.
Natural Language Processing (almost) from Scratch
3,987 Citations2011Ronan Collobert, Jason Weston +4 more
Leibniz-Zentrum für Informatik (Schloss Dagstuhl)NER in Archival Finding Aids
3,270 Citations2021Zhiheng Huang, Wei Xu +1 more
This work has created a set of datasets to train Machine Learning algorithms to find named entities and geographical locations and tested them in several datasets and registered their precision and accuracy.
arXiv (Cornell University)Introduction to the CoNLL-2002 Shared Task: Language-Independent Named Entity Recognition
1,574 Citations2002Erik F. Tjong Kim Sang
Design challenges and misconceptions in named entity recognition
1,476 Citations2009Lev Ratinov, Dan Roth
Some of the fundamental design challenges and misconceptions that underlie the development of an efficient and robust NER system are analyzed, and several solutions to these challenges are developed.
Science China Technological SciencesPre-trained models for natural language processing: A survey
1,429 Citations2020Xipeng Qiu, Tianxiang Sun +4 more
This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.
Message Understanding Conference-6
1,365 Citations1996Ralph Grishman, Beth Sundheim
MUC-6 introduced several innovations over prior MUCs, most notably in the range of different tasks for which evaluations were conducted and the motivations for the new format.
Neural Information Processing SystemsPointer networks
1,332 Citations2015Oriol Vinyals, Meire Fortunato +1 more
A new neural architecture to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence using a recently proposed mechanism of neural attention, called Ptr-Nets, which improves over sequence-to-sequence with input attention, but also allows it to generalize to variable size output dictionaries.
BioinformaticsGENIA corpus—a semantically annotated corpus for bio-textmining
1,243 Citations2003JD Kim, Tomoko Ohta +2 more
Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons
1,161 Citations2003Andrew McCallum, Wei Li
This work has shown that conditionally-trained models, such as conditional maximum entropy models, handle inter-dependent features of greedy sequence modeling in NLP well.
Institutional Research Information System (Università degli Studi di Trento)CoNLL-2012 Shared Task: Modeling Multilingual Unrestricted Coreference in OntoNotes
669 Citations2012Sameer Pradhan, Alessandro Moschitti +3 more
The OntoNotes annotation (coreference and other layers) is described and the parameters of the shared task including the format, pre-processing information, evaluation criteria, and presents and discusses the results achieved by the participating systems.
A Unified MRC Framework for Named Entity Recognition
607 Citations2020Xiaoya Li, Jingrong Feng +4 more
This paper proposes to formulate the task of NER as a machine reading comprehension (MRC) task, and naturally tackles the entity overlapping issue in nested NER: the extraction of two overlapping entities with different categories requires answering two independent questions.
arXiv (Cornell University)MASS: Masked Sequence to Sequence Pre-training for Language Generation
580 Citations2019Kaitao Song, Xu Tan +3 more
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention
551 Citations2020Ikuya Yamada, Akari Asai +3 more
New pretrained contextualized representations of words and entities based on the bidirectional transformer, and an entity-aware self-attention mechanism that considers the types of tokens (words or entities) when computing attention scores are proposed.
arXiv (Cornell University)MASS: Masked Sequence to Sequence Pre-training for Language Generation
478 Citations2019Kaitao Song, Xu Tan +3 more
This work proposes MAsked Sequence to Sequence pre-training (MASS) for the encoder-decoder based language generation tasks, which achieves the state-of-the-art accuracy on the unsupervised English-French translation, even beating the early attention-based supervised model.
arXiv (Cornell University)Non-Autoregressive Neural Machine Translation
449 Citations2017Jiatao Gu, James Bradbury +3 more
Named Entity Recognition as Dependency Parsing
425 Citations2020Juntao Yu, Bernd Bohnet +1 more
Ideas from graph-based dependency parsing are used to provide the model a global view on the input via a biaffine model and show that the model works well for both nested and flat NER, through evaluation on 8 corpora and achieving SoTA performance on all of them.
FLAT: Chinese NER Using Flat-Lattice Transformer
419 Citations2020Xiaonan Li, Hang Yan +2 more
Experiments show FLAT outperforms other lexicon-based models in performance and efficiency and can fully leverage the lattice information and has an excellent parallel ability.
Semi-Supervised Sequence Modeling with Cross-View Training
393 Citations2018Kevin B. Clark, Minh-Thang Luong +2 more
Cross-View Training (CVT), a semi-supervised learning algorithm that improves the representations of a Bi-LSTM sentence encoder using a mix of labeled and unlabeled data, is proposed and evaluated, achieving state-of-the-art results.
Institutional Research Information System (Università degli Studi di Trento)Towards Robust Linguistic Analysis using OntoNotes
362 Citations2013Sameer Pradhan, Alessandro Moschitti +6 more
An analysis of the performance of publicly available, state-of-the-art tools on all layers and languages in the OntoNotes v5.0 corpus should set the benchmark for future development of various NLP components in syntax and semantics, and possibly encourage research towards an integrated system that makes use of the various layers jointly to improve overall performance.
Nested named entity recognition
322 Citations2009Jenny Rose Finkel, Christopher D. Manning
This paper presents a new technique for recognizing nested named entities, by using a discriminative constituency parser, which outperforms a standard semi-CRF on the more traditional top-level entities.
A Neural Layered Model for Nested Named Entity Recognition
315 Citations2018Meizhi Ju, Makoto Miwa +1 more
A novel neural model to identify nested entities by dynamically stacking flat NER layers that outperforms state-of-the-art feature-based systems on nested NER, achieving 74.7% and 72.2% on GENIA and ACE2005 datasets, respectively, in terms of F-score.
A general framework for information extraction using dynamic span graphs
314 Citations2019Yi Luan, Dave Wadden +4 more
This framework significantly outperforms state-of-the-art on multiple information extraction tasks across multiple datasets reflecting different domains and is good at detecting nested span entities, with significant F1 score improvement on the ACE dataset.
Pooled Contextualized Embeddings for Named Entity Recognition
307 Citations2019Alan Akbik, Tanja Bergmann +1 more
This work proposes a method in which it dynamically aggregate contextualized embeddings of each unique string that the authors encounter and uses a pooling operation to distill a ”global” word representation from all contextualized instances.
Neural Architectures for Nested NER through Linearization
298 Citations2019Jana Straková, Milan Straka +1 more
Two neural network architectures for nested named entity recognition (NER), a setting in which named entities may overlap and also be labeled with more than one label, are proposed and outperform the nested NER state of the art on four corpora.
Journal of Biomedical InformaticsCadec: A corpus of adverse drug event annotations
277 Citations2015Sarvnaz Karimi, Alejandro Metke‐Jimenez +2 more
A new rich annotated corpus of medical forum posts on patient-reported Adverse Drug Events (ADEs), which contains text that is largely written in colloquial language and often deviates from formal English grammar and punctuation rules.
Nested Named Entity Recognition Revisited
252 Citations2018Arzoo Katiyar, Claire Cardie
A novel recurrent neural network-based approach to simultaneously handle nested named entity recognition and nested entity mention detection that significantly outperforms existing state-of-the-art methods, which are feature-based.
arXiv (Cornell University)TENER: Adapting Transformer Encoder for Named Entity Recognition
245 Citations2019Hang Yan, Bocao Deng +2 more
TENER, a NER architecture adopting adapted Transformer Encoder to model the character- level features and word-level features is proposed, proving the Transformer-like encoder is just as effective for NER as other NLP tasks.
Joint Mention Extraction and Classification with Mention Hypergraphs
230 Citations2015Wei Lu, Dan Roth
A novel model is presented for the task of joint mention extraction and classification that is able to effectively capture overlapping mentions with unbounded lengths and can be extended to additionally capture mention heads explicitly in a joint manner under the same time complexity.
Multilingual Language Processing From Bytes
220 Citations2016Dan Gillick, Cliff Brunk +2 more
An LSTM-based model that reads text as bytes and outputs span annotations of the form [start, length, label] where start positions, lengths, and labels are separate entries in the authors' vocabulary is described.
Pyramid: A Layered Model for Nested Named Entity Recognition
169 Citations2020Jue Wang, Lidan Shou +2 more
This paper presents Pyramid, a novel layered model for Nested Named Entity Recognition (nested NER) that achieves state-of-the-art F1 scores in nested NER on ACE-2004, ACE-2005, GENIA, and NNE, and a preliminary experiment confirms the effectiveness of the method in overlapping NER.
Neural Segmental Hypergraphs for Overlapping Mention Recognition
160 Citations2018Bailin Wang, Wei Lu
This work proposes a novel segmental hypergraph representation to model overlapping entity mentions that are prevalent in many practical datasets and shows that the model built on top of such a new representation is able to capture features and interactions that cannot be captured by previous models while maintaining a low time complexity for inference.
Recognising nested named entities in biomedical text
159 Citations2007Beatrice Alex, Barry Haddow +1 more
This paper introduces and compares three techniques for modelling and recognising nested entities by means of a conventional sequence tagger and yields an improvement over the baseline tagger that is only trained on flat annotation.
Does syntax matter? A strong baseline for Aspect-based Sentiment Analysis with RoBERTa
155 Citations2021Junqi Dai, Hang Yan +3 more
This paper compares the induced trees from PTMs and the dependency parsing trees on several popular models for the ABSA task, showing that the induced tree from fine-tuned RoBERTa (FT-RoBERTa) outperforms the parser-provided tree and reveals that the FT-RoberTa Induced Tree is more sentiment-word-oriented and could benefit theABSA task.
Transactions of the Association for Computational LinguisticsNested Named Entity Recognition via Second-best Sequence Learning and Decoding
149 Citations2020Takashi Shibuya, Eduard Hovy
A new method to recognize not only outermost named entities but also inner nested ones, and an objective function for training a neural model that treats the tag sequence for nested entities as the second best path within the span of their parent entity is designed.
A Boundary-aware Neural Model for Nested Named Entity Recognition
148 Citations2019Changmeng Zheng, Yi Cai +3 more
This work proposes a boundary-aware neural model for nested NER which leverages entity boundaries to predict entity categorical labels, which can decrease computation cost and relieve error propagation problem in layered sequence labeling model.
A Local Detection Approach for Named Entity Recognition and Mention Detection
143 Citations2017Mingbin Xu, Hui Jiang +1 more
A new local detection approach for named entity recognition (NER) and mention detection (MD) in natural language processing is proposed, which relies on the recent fixed-size ordinally forgetting encoding (FOFE) method to fully encode each sentence fragment and its left/right contexts into a fixed- size representation.
Sequence-to-Nuggets: Nested Entity Mention Detection via Anchor-Region Networks
116 Citations2019Hongyu Lin, Yaojie Lu +2 more
This paper proposes Anchor-Region Networks (ARNs), a sequence-to-nuggets architecture for nested mention detection which first identifies anchor words of all mentions, and then recognizes the mention boundaries for each anchor word by exploiting regular phrase structures.
Research Portal (King's College London)Task 2: ShARe/CLEF eHealth Evaluation Lab 2014
80 Citations2013Danielle L. Mowery, Sumithra Velupillai +9 more
Bipartite Flat-Graph Network for Nested Named Entity Recognition
79 Citations2020Ying Luo, Hai Zhao
A novel bipartite flat-graph network for nested named entity recognition (NER), which contains two subgraph modules: a flat NER module for outermost entities and a graph module for all the entities located in inner layers, which outperforms previous state-of-the-art models.
An Effective Transition-based Model for Discontinuous NER
77 Citations2020Xiang Dai, Sarvnaz Karimi +2 more
This work proposes a simple, effective transition-based model with generic neural encoding for discontinuous NER that can effectively recognize discontinuous mentions without sacrificing the accuracy on continuous mentions.
Wireless Communications and Mobile ComputingRecognizing Continuous and Discontinuous Adverse Drug Reaction Mentions from Social Media Using LSTM‐CRF
57 Citations2018Buzhou Tang, Jianglu Hu +2 more
A deep neural network combining long short-term memory (LSTM) neural networks (a type of recurrent neural networks) and conditional random fields (CRFs) to recognize ADR mentions from social media in medicine is proposed and experiments show that LSTM-CRF achieves better -score than CRF.
Task 1: ShARe/CLEF eHealth evaluation lab 2013
42 Citations2013Sameer Pradhan, Noémie Elhadad +7 more
Combining Spans into Entities: A Neural Two-Stage Approach for Recognizing Discontiguous Entities
41 Citations2019Bailin Wang, Wei Lu
A neural two-stage approach to recognizing discontiguous and overlapping entities by decomposing this problem into two subtasks, which achieves the state-of-the-art performance in a standard dataset, even in the absence of external features that previous methods used.
HIT: Nested Named Entity Recognition via Head-Tail Pair and Token Interaction
35 Citations2020Yu Wang, Yun Li +2 more
This work designs a novel nested NER model named HIT that leverages two key properties pertaining to the (nested) named entity, including (1) explicit boundary tokens and (2) tight internal connection between tokens within the boundary.
Learning Tag Dependencies for Sequence Tagging
26 Citations2018Yuan Zhang, Hongshen Chen +3 more
A novel multi-channel model is proposed, which handles different ranges of token-tag dependencies and their interactions simultaneously, and outperforms the BiLSTM-CRF baseline by effectively incorporating the tag dependency feature.
Institutional Research Information System (Università degli Studi di Trento)Learning to Progressively Recognize New Named Entities with Sequence to Sequence Models
10 Citations2018Lingzhen Chen, Alessandro Moschitti
A new NER approach based on sequence to sequence (Seq2Seq) models, which can intuitively work better in the authors' progressive setting and it is more robust in the progressive setting.
Lecture notes in computer scienceSC-NER: A Sequence-to-Sequence Model with Sentence Classification for Named Entity Recognition
7 Citations2019Yu Wang, Yun Li +3 more
This work proposes a novel seq2seq model, named SC-NER, for NER task, where the classifier’s input is the last hidden state of the encoder and decoder and achieves better performance than other baseline methods.