Character contiguity in N-gram-based word matching: the case for Arabic text searching

Information Processing & ManagementPublished 29 March 2004

Suleiman H. Mustafa

DOI10.1016/j.ipm.2004.02.003

Source URL

Citations28

SJR quartileQ1

SJR score2.06

SNIP2.91

TL;DR

The results of the hybrid approach showed significant performance improvement over the conventional contiguous approach, especially in the cases where stemming was used.

Abstract

This work assesses the performance of two N-gram matching techniques for Arabic root-driven string searching: contiguous N-grams and hybrid N-grams, combining contiguous and non-contiguous. The two techniques were tested using three experiments involving different levels of textual word stemming, a textual corpus containing about 25 thousand words (with a total size of about 160KB), and a set of 100 query textual words. The results of the hybrid approach showed significant performance improvement over the conventional contiguous approach, especially in the cases where stemming was used. The present results and the inconsistent findings of previous studies raise some questions regarding the efficiency of pure conventional N-gram matching and the ways in which it should be used in languages other than English.

Keywords

Computer Science

N-gram-based text categorization

1,503 Citations1994William B. Cavnar, John M. Trenkle

An N-gram-based approach to text categorization that is tolerant of textual errors is described, which worked very well for language classification and worked reasonably well for classifying articles from a number of different computer-oriented newsgroups according to subject.

Improving stemming for Arabic information retrieval

360 Citations2002Leah S. Larkey, Lisa Ballesteros +1 more

Several light stemmers based on heuristics and a statistical stemmer based on co-occurrence for Arabic retrieval and the retrieval effectiveness of these stemmers and of a morphological analyzer on the TREC-2001 data were compared.

2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP '03).Novel approaches to Arabic speech recognition: report from the 2002 Johns-Hopkins Summer Workshop

101 Citations2003Katrin Kirchhoff, Jeffrey A. Bilmes +11 more

Novel approaches to automatic vowel restoration, morphology-based language modeling and the integration of out-of-corpus language model data are presented, and significant word error rate improvements on the LDC Arabic CallHome task are reported.

Empirical studies in strategies for Arabic retrieval

90 Citations2002Jinxi Xu, Alexander Fraser +1 more

Evaluated search strategies for Arabic monolingual and cross-lingual retrieval, using the TREC Arabic corpus as the test-bed, found that spelling normalization and stemming have little impact and a novel thesaurus-based technique is proposed.

A morphologically sensitive clustering algorithm for identifying Arabic roots

66 Citations2000Anne N. De Roeck, Waleed Al-Fares

A clustering algorithm for Arabic words sharing the same root can substitute dictionaries in indexing for IR and shows a successful treatment of infixes and accurate clustering to up to 94.06% for unedited Arabic text samples, without the use of dictionaries.

View PDF

Term selection for searching printed Arabic

58 Citations2002Kareem Darwish, Douglas W. Oard

Alternative choices of indexing terms are explored using both an existing electronic text collection and a newly developed collection built from images of actual printed Arabic documents, and character n-grams or lightly stemmed words were found to typically yield near-optimal retrieval effectiveness.

DOAJ (DOAJ: Directory of Open Access Journals)Targeted s-gram matching: a novel n-gram matching technique for cross- and monolingual word form variants

58 Citations2002Ari Pirkola, Heikki Keskustalo +3 more

View PDF

Journal of the American Society for Information Science and TechnologyUsing N‐grams for Arabic text searching

54 Citations2004Suleiman H. Mustafa, Qasem A. Al‐Radaideh

The results indicate that the digram method offers a better performance than trigram with respect to conflation precision and conflation recall ratios, and does not appear to provide an efficient conflation approach due to the peculiarities imposed by the Arabic infix structure.

Probabilistic methods for searching ocr-degraded arabic text

26 Citations2003Kareem M. Darwish, Bruce Jacob +1 more

New probabilistic structured query methods that leverage replacement probabilities were developed in this research to improve retrieval effectiveness in OCR degraded text retrieval and their generality has been shown in cross-language information retrieval.

JHU/APL at TREC 2002: Experiments in Filtering and Arabic Retrieval

16 Citations2002Paul McNamee, Christine Piatko +1 more

A linguistically motivated probabilistic model is described that does not explicitly incorporate inverse document frequency, but does favor documents that contain more of the rare query terms.