Character N-Grams Translation in Cross-Language Information Retrieval

Lecture notes in computer sciencePublished 14 August 2007

Jesús Vilares, Michael Oakes, Jesús Vilares

DOI10.1007/978-3-540-73351-5_19

Source URL

Citations7

SJR quartileQ2

SJR score0.35

SNIP0.55

TL;DR

This paper describes a new technique for the direct translation of character n-grams for use in Cross-Language Information Retrieval systems that avoids the need for word normalization during indexing or translation, and it can also deal with out-of-vocabulary words.

Abstract

This paper describes a new technique for the direct translation of character n-grams for use in Cross-Language Information Retrieval systems. This solution avoids the need for word normalization during indexing or translation, and it can also deal with out-of-vocabulary words. This knowledge-light approach does not rely on language-specific processing, and it can be used with languages of very different natures even when linguistic information and resources are scarce or unavailable. Our proposal also tries to achieve a higher speed during the n-gram alignment process with respect to previous approaches.

Keywords

Computer Science

Foundations of statistical natural language processing

9,996 Citations1999Christopher D. Manning, Hinrich Schütze

Program electronic library and information systemsAn algorithm for suffix stripping

8,136 Citations1980Martin Porter

An algorithm for suffix stripping is described, which has been implemented as a short, fast program in BCPL, and performs slightly better than a much more elaborate system with which it has been compared.

Computational LinguisticsA Systematic Comparison of Various Statistical Alignment Models

3,928 Citations2003Franz Josef Och, Hermann Ney

An important result is that refined alignment models with a first-order dependence and a fertility model yield significantly better results than simple heuristic models.

View PDF

Statistical phrase-based translation

3,268 Citations2003Philipp Koehn, Franz Josef Och +1 more

The empirical results suggest that the highest levels of performance can be obtained through relatively simple means: heuristic learning of phrase translations from word-based alignments and lexical weighting of phrase translation.

View PDF

Europarl: A Parallel Corpus for Statistical Machine Translation

3,106 Citations2005Philipp Koehn

A corpus of parallel text in 11 languages from the proceedings of the European Parliament is collected and its acquisition and application as training data for statistical machine translation (SMT) is focused on.

View PDF

ACM Transactions on Information SystemsProbabilistic models of information retrieval based on measuring the divergence from randomness

883 Citations2002Gianni Amati, Cornelis J. van Rijsbergen

A framework for deriving probabilistic models of Information Retrieval using term-weighting models obtained in the language model approach by measuring the divergence of the actual term distribution from that obtained under a random process is introduced.

Information RetrievalCharacter N-Gram Tokenization for European Language Text Retrieval

329 Citations2003Paul McNamee, James Mayfield

It is demonstrated empirically how overlapping character n-gram tokenization can provide retrieval accuracy that rivals the best current language-specific approaches for European languages and is a good choice for those languages, and the increased storage and time requirements of the technique.

View PDF

Information Processing & ManagementCross-language information retrieval: experiments based on CLEF 2000 corpora

44 Citations2002Jacques Savoy

Lecture notes in computer scienceJHU/APL Experiments in Tokenization and Non-word Translation

25 Citations2004Paul McNamee, James Mayfield

New findings reaffirming that the judicious use of n-grams can lead to performance surpassing that of word-based tokenization are reported, and that a combination of indexing methods is best of all.

CoLesIR at CLEF 2006: Rapid Prototyping of an N-gram-Based CLIR System

4 Citations2006Jesús Vilares, Michael Oakes +1 more

An n-gram model inspired by the previous work of the Johns Hopkins University Applied Physics Lab is developed, which avoids the need for word normalization during indexing or translation, and it can also deal with out-of-vocabulary words.