Arabic diacritization using weighted finite-state transducers
Published 1 January 2005Open access
Rani Nelken, Stuart M. Shieber
Citations91
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A novel algorithm for restoring symbols of Arabic without short vowels and additional diacritics is presented, using a cascade of probabilistic finite-state transducers trained on the Arabic treebank, integrating a word-based language model, a letter-basedlanguage model, and an extremely simple morphological model.
Abstract
Arabic is usually written without short vowels and additional diacritics, which are nevertheless important for several applications. We present a novel algorithm for restoring these symbols, using a cascade of probabilistic finite- state transducers trained on the Arabic treebank, integrating a word-based language model, a letter-based language model, and an extremely simple morphological model. This combination of probabilistic methods and simple linguistic information yields high levels of accuracy.
Keywords
Computer Science
IEEE Transactions on Acoustics Speech and Signal ProcessingEstimation of probabilities from sparse data for the language model component of a speech recognizer
1,645 Citations1987Slava M. Katz
The model offers, via a nonlinear recursive procedure, a computation and space efficient solution to the problem of estimating probabilities from sparse data, and compares favorably to other proposed methods.
Automatic tagging of Arabic text
311 Citations2004Mona Diab, Kadri Hacıoğlu +1 more
A Support Vector Machine (SVM) based approach to automatically tokenize, tag and annotate base phrases (BPs) in Arabic text and adapt highly accurate tools that have been developed for English text and apply them to Arabic text.
Theoretical Computer ScienceThe design principles of a weighted finite-state transducer library
187 Citations2000Mehryar Mohri, Fernando Pereira +1 more
By taking advantage of the theory of rational power series, the design was able to achieve high degrees of generality, modularity and irredundancy, while attaining competitive efficiency in demanding speech processing applications involving weighted automata of more than 10 7 states and transitions.
Machine transliteration of names in Arabic text
160 Citations2002Yaser Al-Onaizan, Kevin Knight
A new spelling-based model is introduced that is much more accurate than state-of-the-art phonetic-based models and can be trained on easier-to-obtain training data.
The MIT Press eBooksSpeech Recognition by Composition of Weighted Finite Automata
152 Citations1997Fernando C. N. Pereira, Michael Riley
Generalized algorithms for constructing statistical language models
126 Citations2003Cyril Allauzen, Mehryar Mohri +1 more
An algorithm for computing efficiently the expected counts of any sequence in a word lattice output by a speech recognizer or any arbitrary weighted automaton is given and a new technique for creating exact representations of n-gram language models by weighted automata is described.
2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03).Novel approaches to Arabic speech recognition: report from the 2002 Johns-Hopkins Summer Workshop
101 Citations2003Katrin Kirchhoff, Jeffrey A. Bilmes +11 more
Novel approaches to automatic vowel restoration, morphology-based language modeling and the integration of out-of-corpus language model data are presented, and significant word error rate improvements on the LDC Arabic CallHome task are reported.
Automatic diacritization of Arabic for acoustic modeling in speech recognition
95 Citations2004Dimitra Vergyri, Katrin Kirchhoff
Various procedures that enable us to use acoustic information in combination with different levels of morphological and contextual constraints by automatically inserting the missing diacritics into the transcription are investigated.
Journal of Psycholinguistic ResearchThe Effect of Arabic Vowels on the Reading Comprehension of Second- and Sixth-Grade Native Arab Children
87 Citations1999Salim Abu‐Rabia
The results revealed that vowels were a significant facilitator of reading comprehension in both age groups, and reading in Arabic orthography is not an autonomous word recognition process.
Issues in Arabic orthography and morphology analysis
82 Citations2004Tim Buckwalter
This paper discusses several issues in Arabic orthography that were encountered in the process of performing morphology analysis and POS tagging of 542,543 Arabic words in three newswire corpora at the LDC during 2002--2004, by means of the Buckwalter Arabic Morphological Analyzer.
An HMM approach to vowel restoration in Arabic and Hebrew
79 Citations2002Kobi Gal
It is shown that Hidden Markov Models are a useful tool for the task of vowel restoration in Semitic languages and does not require any language specific knowledge to be embedded in the model and generalizes well to both Hebrew and Arabic.
Resources for Processing Hebrew
9 Citations2003Shuly Wintner
The design of the resources guarantees their reusability, such that the output of one system can naturally be the input to another, and the methodological issue of well-defined standards for the resources to be developed is emphasized.