Exact Hard Monotonic Attention for Character-Level Transduction
Published 1 January 2019Open access
Shijie Wu, Ryan Cotterell
Citations54
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work develops a hard attention sequence-to-sequence model that enforces strict monotonicity and learns alignment jointly and achieves state-of-the-art performance on morphological inflection.
Abstract
Many common character-level, string-to-string transduction tasks, e.g., grapheme-to-phoneme conversion and morphological inflection, consist almost exclusively of monotonic transduction. Neural sequence-to-sequence models with soft attention, non-monotonic models, outperform popular monotonic models. In this work, we ask the following question: Is monotonicity really a helpful inductive bias in these tasks? We develop a hard attention sequence-to-sequence model that enforces strict monotonicity and learns alignment jointly. With the help of dynamic programming, we are able to compute the exact marginalization over all alignments. Our models achieve state-of-the-art performance on morphological inflection. Furthermore, we find strong performance on two other character-level transduction tasks. Code is available at https://github.com/shijie-wu/neural-transducer.
Keywords
Computer Science
UvA-DARE (University of Amsterdam)Adam: A Method for Stochastic Optimization
84,783 Citations2014Diederik P. Kingma, Jimmy Ba
DROPS (Schloss Dagstuhl – Leibniz Center for Informatics)
50,318 Citations2021Mandi, Jayanta, Canoy, Rocsildes +2 more
A simple numeric simulation of DNA-co-polymerized hydrogel shape change and a genetic algorithm that generates and selects large batches of material designs that compete with one another to evolve and converge on optimal objective-matching designs are constructed.
Proceedings of the IEEEA tutorial on hidden Markov models and selected applications in speech recognition
22,785 Citations1989L. R. Rabiner
arXiv (Cornell University)Neural Machine Translation by Jointly Learning to Align and Translate
14,565 Citations2014Dzmitry Bahdanau
Effective Approaches to Attention-based Neural Machine Translation
8,582 Citations2015Thang Luong, Hieu Pham +1 more
A global approach which always attends to all source words and a local one that only looks at a subset of source words at a time are examined, demonstrating the effectiveness of both approaches on the WMT translation tasks between English and German in both directions.
arXiv (Cornell University)Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
7,525 Citations2015Kelvin Xu, Jimmy Ba +6 more
An attention based model that automatically learns to describe the content of images is introduced that can be trained in a deterministic manner using standard backpropagation techniques and stochastically by maximizing a variational lower bound.
The mathematics of statistical machine translation: parameter estimation
4,125 Citations1993Peter F. Brown, Vincent J. Della Pietra +2 more
It is reasonable to argue that word-by-word alignments are inherent in any sufficiently large bilingual corpus, given a set of pairs of sentences that are translations of one another.
Parallel Networks that Learn to Pronounce English Text
1,556 Citations1989Terrence J. Sejnowski
arXiv (Cornell University)Sequence Transduction with Recurrent Neural Networks
1,292 Citations2012Alex Graves
This paper introduces an end-to-end, probabilistic sequence transduction system, based entirely on RNNs, that is in principle able to transform any input sequence into any finite, discrete output sequence.
The SIGMORPHON 2016 Shared Task—Morphological Reinflection
255 Citations2016Ryan Cotterell, Christo Kirov +4 more
The 2016 SIGMORPHON Shared Task was devoted to the problem of morphological reinflection, and introduced morphological datasets for 10 languages with diverse ty-pological characteristics, showing a strong state of the art.
arXiv (Cornell University)Online and Linear-Time Attention by Enforcing Monotonic Alignments
198 Citations2017Colin Raffel, Minh-Thang Luong +3 more
Morphological Inflection Generation with Hard Monotonic Attention
119 Citations2017Roee Aharoni, Yoav Goldberg
A neural model for morphological inflection generation which employs a hard attention mechanism, inspired by the nearly-monotonic alignment commonly found between the characters in a word and the charactersIn its inflection, is presented.
Online Segment to Segment Neural Transduction
76 Citations2016Lei Yu, Jan Buys +1 more
An online neural sequence to sequence model that learns to alternate between encoding and decoding segments of the input as it is read that tackles the bottleneck of vanilla encoder-decoders that have to read and memorize the entire input sequence in their fixed-length hidden states.
Data Augmentation for Morphological Reinflection
75 Citations2017Miikka Silfverberg, Adam Wiemerslage +2 more
Training Data Augmentation for Low-Resource Morphological Inflection
62 Citations2017Toms Bergmanis, Katharina Kann +2 more
It is found that autoencoding random strings works surprisingly well, outperformed only slightly by autoencoding words from an unlabelled corpus, and the random string method also works well in the 10,000-example setting despite not being tuned for it.
Weighting Finite-State Transductions With Neural Context
60 Citations2016Pushpendre Rastogi, Ryan Cotterell +1 more
This work proposes to keep the traditional architecture, which uses a finite-state transducer to score all possible output strings , but to augment the scoring function with the help of recurrent networks, and combines these learned features with the transducers to define a probability distribution over aligned output strings, in the form of a weighted flnites-state automaton.
arXiv (Cornell University)Sequence-to-sequence neural network models for transliteration
57 Citations2016Mihaela Rosca, Thomas M. Breuel
It is demonstrated that neural sequence-to-sequence models obtain state of the art or close to state-of-the- art results on existing datasets.
Hard Non-Monotonic Attention for Character-Level Transduction
50 Citations2018Shijie Wu, Pamela Shapiro +1 more
An exact, polynomial-time algorithm for marginalizing over the exponential number of non-monotonic alignments between two strings is introduced, showing that hard attention models can be viewed as neural reparameterizations of the classical IBM Model 1.
Multi-space Variational Encoder-Decoders for Semi-supervised Labeled Sequence Transduction
49 Citations2017Chunting Zhou, Graham Neubig
This paper proposes multi-space variational encoder-decoders, a new model for labeled sequence transduction with semi-supervised learning that outperforms single-model state-of-art results by a large margin for the majority of languages.
Imitation Learning for Neural Morphological String Transduction
40 Citations2018Peter Makarov, Simon Clematide
In imitation learning is employed to train a neural transition-based string transducer for morphological tasks such as inflection generation and lemmatization, eliminating the need for a character aligner or warm start and leading to strong and state-of-the-art performance on several benchmarks.
Neural Hidden Markov Model for Machine Translation
28 Citations2018Weiyue Wang, Derui Zhu +3 more
It is shown that the attention component can be effectively replaced by the neural network alignment model and the neural HMM approach is able to provide comparable performance with the state-of-the-art attention-based models on the WMT 2017 German↔English and Chinese→English translation tasks.
Noise-Aware Character Alignment for Bootstrapping Statistical Machine Transliteration from Bilingual Corpora
14 Citations2013Katsuhito Sudoh, Shinsuke Mori +1 more
A novel noise-aware character alignment method for bootstrapping statistical machine transliteration from automatically extracted phrase pairs is proposed, an extension of a Bayesian many-to-many aligned method for distinguishing nontransliteration (noise) parts in phrase pairs.