Multilingual Language Processing From Bytes
Published 1 January 2016Open access
Dan Gillick, Cliff Brunk, Oriol Vinyals, Amarnag Subramanya
Citations220
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
An LSTM-based model that reads text as bytes and outputs span annotations of the form [start, length, label] where start positions, lengths, and labels are separate entries in the authors' vocabulary is described.
Abstract
Dan Gillick, Cliff Brunk, Oriol Vinyals, Amarnag Subramanya. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.
Keywords
Computer Science
Neural ComputationLong Short-Term Memory
98,079 Citations1997Sepp Hochreiter, Jürgen Schmidhuber
A novel, efficient, gradient based method called long short-term memory (LSTM) is introduced, which can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units.
arXiv (Cornell University)Neural Machine Translation by Jointly Learning to Align and Translate
14,565 Citations2014Dzmitry Bahdanau
arXiv (Cornell University)Sequence to Sequence Learning with Neural Networks
13,362 Citations2014Ilya Sutskever, Oriol Vinyals +1 more
ScholarlyCommons (University of Pennsylvania)Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data
12,978 Citations2001John Lafferty, Andrew McCallum +1 more
This work presents iterative parameter estimation algorithms for conditional random fields and compares the performance of the resulting models to HMMs and MEMMs on synthetic and natural-language data.
arXiv (Cornell University)Improving neural networks by preventing co-adaptation of feature detectors
6,631 Citations2012Geoffrey E. Hinton, Nitish Srivastava +3 more
arXiv (Cornell University)Natural Language Processing (almost) from Scratch
5,172 Citations2011Ronan Collobert, Jason Weston +4 more
Natural Language Processing (almost) from Scratch
3,987 Citations2011Ronan Collobert, Jason Weston +4 more
Leibniz-Zentrum für Informatik (Schloss Dagstuhl)NER in Archival Finding Aids
3,270 Citations2021Zhiheng Huang, Wei Xu +1 more
This work has created a set of datasets to train Machine Learning algorithms to find named entities and geographical locations and tested them in several datasets and registered their precision and accuracy.
arXiv (Cornell University)Character-level Convolutional Networks for Text Classification
3,266 Citations2015Xiang Zhang, Junbo Zhao +1 more
This article constructed several large-scale datasets to show that character-level convolutional networks could achieve state-of-the-art or competitive results in text classification.
arXiv (Cornell University)Recurrent Neural Network Regularization
2,274 Citations2014Wojciech Zaremba, Ilya Sutskever +1 more
This paper shows how to correctly apply dropout to LSTMs, and shows that it substantially reduces overfitting on a variety of tasks.
Journal of Machine Learning ResearchA Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data
1,372 Citations2005Rie Kubota Ando, Tong Zhang
This paper presents a general framework in which the structural learning problem can be formulated and analyzed theoretically, and relate it to learning with unlabeled data, and algorithms for structural learning will be proposed, and computational issues will be investigated.
Proceedings of the AAAI Conference on Artificial IntelligenceCharacter-Aware Neural Language Models
1,046 Citations2016Yoon Kim, Yacine Jernite +2 more
A simple neural language model that relies only on character-level inputs that is able to encode, from characters only, both semantic and orthographic information and suggests that on many languages, character inputs are sufficient for language modeling.
arXiv (Cornell University)Character-Aware Neural Language Models
1,022 Citations2015Yoon Kim, Yacine Jernite +2 more
arXiv (Cornell University)A Universal Part-of-Speech Tagset
731 Citations2011Slav Petrov, Dipanjan Das +1 more
This work proposes a tagset that consists of twelve universal part-of-speech categories and develops a mapping from 25 different treebank tagsets to this universal set, which when combined with the original treebank data produces a dataset consisting of common parts- of-speech for 22 different languages.
Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
568 Citations2015Ling Wang, Chris Dyer +6 more
A model for constructing vector representations of words by composing characters using bidirectional LSTMs that requires only a single vector per character type and a fixed set of parameters for the compositional model, which yields state- of-the-art results in language modeling and part-of-speech tagging.
arXiv (Cornell University)Semi-supervised Sequence Learning
490 Citations2015Andrew M. Dai, Quoc V. Le
Two approaches to use unlabeled data to improve Sequence Learning with recurrent networks are presented and it is found that long short term memory recurrent networks after pretrained with the two approaches become more stable to train and generalize better.
Lecture notes in computer sciencePart-of-Speech Tagging from 97% to 100%: Is It Time for Some Linguistics?
408 Citations2011Christopher D. Manning
It is suggested and demonstrated that the largest opportunity for further progress comes from improving the taxonomic basis of the linguistic resources from which taggers are trained, that is, from improved descriptive linguistics.
Named entity recognition through classifier combination
403 Citations2003Radu Florian, Abe Ittycheriah +2 more
A classifier-combination experimental framework for named entity recognition in which four diverse classifiers (robust linear classifier, maximum entropy, transformation-based learning, and hidden Markov model) are combined under different conditions is presented.
Low Resource Dependency Parsing: Cross-lingual Parameter Sharing in a Neural Network Parser
378 Citations2015Long Duong, Trevor Cohn +2 more
This work proposes a learning method that needs less data, based on the observation that there are underlying shared structures across languages, and exploits cues from a different source language in order to guide the learning process.
Artificial IntelligenceLearning multilingual named entity recognition from Wikipedia
366 Citations2012Joel Nothman, Nicky Ringland +3 more
The approach outperforms other approaches to automatic ne annotation; competes with gold-standard training when tested on an evaluation corpus from a different source; and performs 10% better than newswire-trained models on manually-annotated Wikipedia text.
International Conference on Artificial Intelligence and StatisticsThe Difficulty of Training Deep Architectures and the Effect of Unsupervised Pre-Training
323 Citations2009Dumitru Erhan, Pierre-Antoine Manzagol +3 more
The experiments confirm and clarify the advantage of unsupervised pre- training, and empirically show the influence of pre-training with respect to architecture depth, model capacity, and number of training examples.
arXiv (Cornell University)Listen, Attend and Spell
312 Citations2015William Chan, Navdeep Jaitly +2 more
Lexicon Infused Phrase Embeddings for Named Entity Resolution
266 Citations2015Alexandre Passos, Vineet Kumar +1 more
Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs
245 Citations2015Miguel Ballesteros, Chris Dyer +1 more
Extensions to a continuousstate dependency parsing method that makes it applicable to morphologically rich languages replace lookup-based word representations with representations constructed from the orthographic representations of the words, also using LSTMs.
Named entity recognition with character-level models
195 Citations2003Dan Klein, Joseph Smarr +2 more
Two named-entity recognition models which use characters and character n-grams either exclusively or as an important part of their data representation are discussed, both of which are a character-level HMM with minimal context information and a maximum-entropy conditional markov model with substantially richer context features.
Char_align
191 Citations1993Kenneth Church
This paper describes a new program, char_align, that aligns texts at the character level rather than at the sentence/paragraph level, based on the cognate approach proposed by Simard et al.
Named Entity Extraction using AdaBoost
185 Citations2002Xavier Carreras, Lluı́s Màrquez +1 more
This paper presents a Named Entity Extraction (NEE) system for the CoNLL 2002 competition, which makes use of binary AdaBoost classifiers.
Language independent authorship attribution using character level language models
155 Citations2003Fuchun Peng, Dale Schuurmans +2 more
Boosting Named Entity Recognition with Neural Character Embeddings
155 Citations2015Cícero Nogueira dos Santos, Victor Guimarães
This work proposes a language-independent NER system that uses automatically learned features only and demonstrates that the same neural network which has been successfully applied to POS tagging can also achieve state-of-the-art results for language-independet NER, using the same hyperparameters, and without any handcrafted features.
arXiv (Cornell University)Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
131 Citations2015Ling Wang, Tiago Luís +6 more
Effective identification of source code authors using byte-level information
104 Citations2006Georgia Frantzeskou, Efstathios Stamatatos +2 more
The SCAP approach is language independent and that n-gram author profiles are better able to capture the idiosyncrasies of the source code authors, and is able to deal surprisingly well with cases where only a limited amount of very short programs per programmer is available for training.
Lexicon Infused Phrase Embeddings for Named Entity Resolution
68 Citations2014Alexandre Passos, Vineet Kumar +1 more
A new form of learning word embeddings that can leverage information from relevant lexicons to improve the representations, and the first system to use neural word embedDings to achieve state-of-the-art results on named-entity recognition in both CoNLL and Ontonotes NER are presented.
Chinese and Japanese word segmentation using word-level and character-level information
57 Citations2004Tetsuji Nakagawa
A hybrid method for Chinese and Japanese word segmentation that utilizes word-level and character-level information in order to effectively handle known and unknown words.
From speech to letters - using a novel neural network architecture for grapheme based ASR
45 Citations2009Florian Eyben, Martin Wöllmer +2 more
This work investigates a novel ASR approach using Bidirectional Long Short-Term Memory Recurrent Neural Networks and Connectionist Temporal Classification, which is capable of transcribing graphemes directly and yields results highly competitive with phoneme transcription.
Variable-Length Word Encodings for Neural Translation Models
35 Citations2015Rohan Chitnis, John DeNero
This work proposes and compares three variable-length encoding schemes that represent a large vocabulary corpus using a much smaller vocabulary with no loss in information and improves WMT English-French translation performance by up to 1.7 BLEU.