Statistical Machine Translation of Texts with Misspelled Words
Published 2 June 2010
Nicola Bertoldi, Mauro Cettolo, Marcello Federico
Citations22
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Experimental results are presented on two translation tasks of increasing complexity which show how misspellings of different types do affect performance of a statistical machine translation decoder and to what extent the enhanced system is able to recover from such errors.
Abstract
This paper investigates the impact of misspelled words in statistical machine translation and proposes an extension of the translation engine for handling misspellings. The enhanced system decodes a word-based confusion network representing spelling variations of the input text. We present extensive experimental results on two translation tasks of increasing complexity which show how misspellings of different types do affect performance of a statistical machine translation decoder and to what extent our enhanced system is able to recover from such errors. 1
Keywords
Computer Science
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
Mining the peanut gallery
1,908 Citations2003Kushal Dave, Steve Lawrence +1 more
This work develops a method for automatically distinguishing between positive and negative reviews and draws on information retrieval techniques for feature extraction and scoring, and the results for various metrics and heuristics vary depending on the testing situation.
Computer Speech & LanguageFinding consensus in speech recognition: word error minimization and other applications of confusion networks
641 Citations2000Lidia Mangu, Eric Brill +1 more
A new framework for distilling information from word lattices is described to improve the accuracy of the speech recognition output and obtain a more perspicuous representation of a set of alternative hypotheses.
National Conference on Artificial IntelligenceEffects of Age and Gender on Blogging
558 Citations2005Jonathan Schler, Moshe Koppel +2 more
Analysis of a corpus of tens of thousands of blogs – incorporating close to 300 million words – indicates significant differences in writing style and content between male and female bloggers as well as among authors of different ages.
An improved error model for noisy channel spelling correction
537 Citations2000Eric Brill, Robert C. Moore
A new channel model for spelling correction, based on generic string to string edits, is described, which gives significant performance improvements compared to previously proposed models.
Pronunciation modeling for improved spelling correction
190 Citations2001Kristina Toutanova, Robert C. Moore
The proposed method builds an explicit error model for word pronunciations by modeling pronunciation similarities between words and achieves a substantial performance improvement over the previous best performing models for spelling correction.
Natural Language EngineeringCorrecting real-word spelling errors by restoring lexical cohesion
180 Citations2005Graeme Hirst, Alexander Budanitsky
A method for detecting and correcting many spelling errors by identifying tokens that are semantically unrelated to their context and are spelling variations of words that would be related to the context is presented.
Statistics and ComputingProbability scoring for spelling correction
180 Citations1991Kenneth Church, William A. Gale
This paper describes a new program, CORRECT, which takes words rejected by the Unix® SPELL program, proposes a list of candidate corrections, and sorts them by probability score, and finds that human judges were extremely reluctant to cast a vote given only the information available to the program.
A survey of types of text noise and techniques to handle noisy text
82 Citations2009L. Venkata Subramaniam, Shourya Roy +2 more
A survey of the existing measures for noise in text is presented and application areas that ingest this noisy text for various tasks like Information Retrieval and Information Extraction are covered.
IEEE Signal Processing MagazineRecent efforts in spoken language translation
52 Citations2008Francisco Casacuberta, Marcello Federico +2 more
A unifying framework for this ASR-MT interface is provided by applying the Bayes decision rule to the speech translation tasks as whole rather than to each task individually, which helps to pass on the ASR ambiguities to the MT process.
Communications of the ACMSpelling correction for the telecommunications network for the deaf
45 Citations1992Karen Kukich
Studying the effects of noisy text on text mining applications
39 Citations2009Lipika Dey, Shamsul Haque
Analysis is presented on how noise introduced due to incorrect English affects the performance of some of the NLP tools and thereafter the text mining applications.
I saw TREE trees in the park: How to Correct Real-Word Spelling Mistakes
30 Citations2008Davide Fossati, Barbara Di Eugenio
A context sensitive spell checking system that uses mixed trigram models, and a new empirically grounded method for building confusion sets is presented, which shows that the method is effective.
IEEE Transactions on Audio Speech and Language ProcessingEfficient Speech Translation Through Confusion Network Decoding
22 Citations2008Nicola Bertoldi, Richard Zens +2 more
A decoding algorithm for confusion networks is presented which results as an extension of a state-of-the-art phrase-based text translation decoder which significantly improves both in efficiency and performance over previous work along this direction, and outperforms the background text translation system.
Text alignment in the real world
18 Citations1995Mark M. Davis, Ted Dunning +1 more
A method based on a dynamic programming framework, but using a decision criterion derived from a combination of byte-length ratio measures, hard matching of numbers, string comparisons and n-gram co-occurrence matching substantially improves the performance of the alignment process.
Multi-Level Feature Extraction for Spelling Correction
17 Citations2008Johannes Schaback
This work develops a multi-level featurebased framework that allows to correct non-word errors as well as real- word errors simultaneously using the same feature extraction methods, and it closes the gap separating isolated error correction techniques from context-sensitive methods.
Using noisy bilingual data for statistical machine translation
15 Citations2003Stephan Vogel
This paper investigates possibilities to use word-to-word and phrase- to-phrase translations extracted not only from clean parallel corpora but also from noisy comparable corpora to train the translation model.