Comparing methods for language identification

Published 1 September 2004

Lluís Padró, Lluís Padró

Source URL

Citations28

SJR quartileQ1

SJR score0.57

SNIP1.61

TL;DR

Three different statistical language identification methods are compared, and a detailed study of the influence on those systems of some basic parameters is performed.

Abstract

Resumen: En este artículo se comparan tres sistemas estadísticos de identificación de idioma. Se presenta también un estudio detallado de la influencia de algunos factores importantes sobre la precisión de los sistemas. Estos factores son: la medida del corpus de entrenamiento, la cantidad de texto que se quiere clasificar y las lenguas entre las cuales el sistema es capaz de distinguir (se estudiará tanto el número de lenguas cómo cuáles son esas lenguas). Palabras clave: identificación de idioma, sistemas estadísticos, multilinguismo, modelos de Markov visibles, vectores de frecuencias de trigramas, categorización de textos basada en-gramas Abstract: In this work three different statistical language identification methods are compared, and a detailed study of the influence on those systems of some basic parameters is performed. The analyzed parameters are the size of the train set, the amount of text that we want to classify and the languages the system is able to distinguish (it will be studied not only the influence of the number of languages but also the influence of wich are the considered languages).

Keywords

Computer Science

N-gram-based text categorization

1,503 Citations1994William B. Cavnar, John M. Trenkle

An N-gram-based approach to text categorization that is tolerant of textual errors is described, which worked very well for language classification and worked reasonably well for classifying articles from a number of different computer-oriented newsgroups according to subject.

ScienceGauging Similarity with <i>n</i> -Grams: Language-Independent Categorization of Text

535 Citations1995Marc Damashek

A language-independent means of gauging topical similarity in unrestricted text by combining information derived from n-grams with a simple vector-space technique that makes sorting, categorization, and retrieval feasible in a large multilingual collection of documents.

Automatic language identification of telephone speech messages using phoneme recognition and N-gram modeling

121 Citations2002M.A. Zissman, Elliot Singer

The paper compares the performance of four approaches to automatic language identification of telephone speech messages: Gaussian mixture model classification (GMM), language-independent phoneme recognition followed by language-dependent language modeling (PRLM), parallel PRLM, PRLM-P, and language- dependent parallel phoneme Recognition (PPR).

Language identification using phone-based acoustic likelihoods

59 Citations2002Lori Lamel, J.-L. Gauvain

Applies the technique of phone-based acoustic likelihoods to the problem of language identification to process the unknown speech signal by language-specific phone model sets in parallel, and hypothesize the language associated with the model set having the highest likelihood.

Pattern RecognitionLanguage identification in web documents using discrete HMMs

39 Citations2003A. Xafopoulos, Constantine Kotropoulos +2 more

The proposed system is built on hidden Markov models that enable the modeling of character sequences that provide the means for language tracking, that is, language identification across the segments of a multilingual document.