Farasa: A Fast and Furious Segmenter for Arabic
Published 1 January 2016Open access
Ahmed Abdelalí, Kareem Darwish, Nadir Durrani, Hamdy Mubarak
Citations372
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Farasa outperforms or is at par with the state-of-the-art Arabic segmenters (Stanford and MADAMIRA), while being more than one order of magnitude faster in two NLP tasks, namely Machine Translation and Information Retrieval.
Abstract
In this paper, we present Farasa, a fast and accurate Arabic segmenter. Our approach is based on SVM-rank using linear kernels. We measure the performance of the segmenter in terms of accuracy and efficiency, in two NLP tasks, namely Machine Translation (MT) and Information Retrieval (IR). Farasa outperforms or is at par with the stateof-the-art Arabic segmenters (Stanford and MADAMIRA), while being more than one order of magnitude faster.
Keywords
Computer ScienceArts and Humanities
Moses
4,868 Citations2007Philipp Koehn, Richard Zens +12 more
An open-source toolkit for statistical machine translation whose novel contributions are support for linguistically motivated factors, confusion network decoding, and efficient data formats for translation models and language models.
Training linear SVMs in linear time
1,947 Citations2006Thorsten Joachims
A Cutting Plane Algorithm for training linear SVMs that provably has training time 0(s,n) for classification problems and o(sn log (n)) for ordinal regression problems and several orders of magnitude faster than decomposition methods like svm light for large datasets.
RWTH Publications (RWTH Aachen)Moses: Open Source Toolkit for Statistical Machine Translation
1,451 Citations2007Philipp Koehn, Hieu Hoang +12 more
KenLM: Faster and Smaller Language Model Queries
1,142 Citations2011Kenneth Heafield
KenLM is a library that implements two data structures for efficient language model queries, reducing both time and memory costs and is integrated into the Moses, cdec, and Joshua translation systems.
Research Showcase @ Carnegie Mellon University (Carnegie Mellon University)A Simple, Fast, and Effective Reparameterization of IBM Model 2
839 Citations2018Chris Dyer, Victor Chahuneau +1 more
A simple log-linear reparameterization of IBM Model 2 that overcomes problems arising from Model 1’'s strong assumptions and Model 2’s overparameterization is presented.
NPARCBatch Tuning Strategies for Statistical Machine Translation
338 Citations2012Colin Cherry, George Foster
It is found that a simple and efficient batch version of MIRA performs at least as well as training online, and consistently outperforms other options.
Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)Minimum Bayes-risk decoding for statistical machine translation
330 Citations2004Shankar Kumar, Bill Byrne
The results show that MBR decoding can be used to tune statistical MT performance for specific loss functions, and a hierarchy of loss functions that incorporate different levels of linguistic information from word strings, word-to-word alignments from an MT system, and syntactic structure from parse-trees of source and target language sentences.
A simple and effective hierarchical phrase reordering model
297 Citations2008Michel Galley, Christopher D. Manning
A novel hierarchical phrase reordering model aimed at improving non-local reorderings, which seamlessly integrates with a standard phrase-based system with little loss of computational efficiency is presented.
Forest Rescoring: Faster Decoding with Integrated Language Models
290 Citations2007Liang Huang, David Chiang
This work develops faster approaches for efficient decoding based on k-best parsing algorithms and demonstrates their effectiveness on both phrase-based and syntax-based MT systems.
MultiUN: A Multilingual Corpus from United Nation Documents
214 Citations2010Andreas Eisele, Yu Chen
The interpretation is that the 'fetal origins' hypothesis' is valid for middle-age subjects who grow up in a welfare society and was associated with increased SBP and DBP.
Building a shallow Arabic Morphological Analyzer in one day
173 Citations2002Kareem Darwish
The paper presents a rapid method of developing a shallow Arabic morphological analyzer based on automatically derived rules and statistics that will only be concerned with generating the possible roots of any given Arabic word.
On arabic search
172 Citations2002Mohammed Aljlayl, Ophir Frieder
This work proposes a novel light-stemming algorithm for Arabic texts that significantly outperforms the root-based algorithm and shows that a significant improvement in retrieval precision can be achieved with light inflectional analysis of Arabic words.
Arabic finite-state morphological analysis and generation
162 Citations1996Kenneth R. Beesley
A large-scale system that performs morphological analysis and generation of on-line Arabic words represented in the standard orthography, whether fully voweled, partially voweled or unvoweled, using Xerox Finite-State Morphology tools.
A Joint Sequence Translation Model with Integrated Reordering
108 Citations2011Nadir Durrani, Helmut Schmid +1 more
A novel machine translation model which models translation by a linear sequence of operations which includes not only translation but also reordering operations, and a joint sequence model for the translation and reordering probabilities which is more flexible than standard phrase-based MT.
Language processing for arabic microblog retrieval
105 Citations2012Kareem Darwish, Walid Magdy +1 more
Some of the required processing for effective retrieval of Arabic microblog retrieval such as improved letter normalization, elongated word handling, stopword removal, and stemming are presented.
Integrating an Unsupervised Transliteration Model into Statistical Machine Translation
89 Citations2014Nadir Durrani, Hassan Sajjad +2 more
This work induces a transliteration model from parallel data and uses it to translate OOV words and shows that the mined transliterations provide better rule coverage and translation quality compared to the gold standard transliterated corpora.
Word Segmentation of Informal Arabic with Domain Adaptation
76 Citations2014Will Monroe, Spence Green +1 more
This work extends an existing MSA segmenter with a simple domain adaptation technique and new features in order to segment informal and dialectal Arabic text, and shows that the system outperforms existing systems on newswire, broadcast news and Egyptian dialect.
Urdu Word Segmentation
68 Citations2010Nadir Durrani, Sarmad Hussain
This paper discusses how orthographic and linguistic features in Urdu trigger these two problems and employs a hybrid solution that performs an n-gram ranking on top of rule based maximum matching heuristic.
A Class-Based Agreement Model for Generating Accurately Inflected Translations
48 Citations2012Spence Green, John DeNero
A target-side, class-based agreement model for automatically translating from a weakly inflected source language like English to a target language with richer grammatical features such as gender and dual number that yields an average improvement over a state-of-the-art baseline.
Edinburgh’s Phrase-based Machine Translation Systems for WMT-14
44 Citations2014Nadir Durrani, Barry Haddow +2 more
UEDIN’s phrase-based submissions to the translation and medical translation shared tasks of the 2014 Workshop on Statistical Machine Translation (WMT) are described.
Computational LinguisticsThe Operation Sequence Model—Combining N-Gram-Based and Phrase-Based Statistical Machine Translation
40 Citations2015Nadir Durrani, Helmut Schmid +3 more
A novel machine translation model, the Operation Sequence Model (OSM), which combines the benefits of phrase-based and N-gram-based statistical machine translation (SMT) and remedies their drawbacks and outperforms lexicalized reordering on all translation tasks.
International Conference on Computational LinguisticsInvestigating the Usefulness of Generalized Word Representations in SMT
37 Citations2014Nadir Durrani, Philipp Koehn +2 more
This work investigates the use of generalized representations (POS, morphological analysis and word clusters) in phrase-based models and the N-gram-based Operation Sequence Model and finds that this integration enables these models to learn richer lexical and reordering patterns, consider wider contextual information and generalize better in sparse data conditions.
Finite-State Methods and Natural Language ProcessingAn Open-Source Finite State Morphological Transducer for Modern Standard Arabic
29 Citations2011Mohammed Attia, Pavel Pecina +3 more
An open-source large-scale finitestate morphological processing toolkit (AraComLex) for Modern Standard Arabic (MSA) distributed under the GPLv3 license and evaluated and compared to LDC's SAMA (Standard Arabic Morphological Analyser).
QCRI at IWSLT 2013: Experiments in Arabic-English and English-Arabic Spoken Language Translation
21 Citations2013Hassan Sajjad, Francisco Guzmán +5 more
A specialized normalization scheme for evaluating Arabic output is described, which was adopted for the IWSLT’2013 evaluation campaign and experimented with three domain adaptation methods, and with various Arabic word segmentation schemes.
Text, speech and language technologyAdapting Morphology for Arabic Information Retrieval*
18 Citations2007Kareem Darwish, Douglas W. Oard
An adaptation of existing techniques in Arabic morphology by leveraging corpus statistics to make them suitable for Information Retrieval (IR) resulted in the development of Sebawai, an shallow Arabic morphological analyzer, and Al-Stem, an Arabic light stemmer.
Edinburgh SLT and MT System Description for the IWSLT 2013 Evaluation
15 Citations2013Alexandra Birch, Matthias Huck +1 more
The University of Edinburgh’s spoken language translation and machine translation systems for the IWSLT 2014 evaluation campaign are described and using unsupervised transliteration for languages which have a different script than English is explored.