The Web as a Parallel Corpus
Computational LinguisticsPublished 1 September 2003Open access
Philip Resnik, Noah A. Smith
Citations525
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The use of supervised learning based on structural features of documents to improve classification performance, a new content-based measure of translational equivalence, and adaptation of the system to take advantage of the Internet Archive for mining parallel text from the Web on a large scale are presented.
Abstract
Parallel corpora have become an essential resource for work in multilingual natural language processing. In this article, we report on our work using the STRAND system for mining parallel text on the World Wide Web, first reviewing the original algorithm and results and then presenting a set of significant enhancements. These enhancements include the use of supervised learning based on structural features of documents to improve classification performance, a new content-based measure of translational equivalence, and adaptation of the system to take advantage of the Internet Archive for mining parallel text from the Web on a large scale. Finally, the value of these techniques is demonstrated in the construction of a significant parallel corpus for a low-density language pair.
Keywords
Computer Science
Bell System Technical JournalA Mathematical Theory of Communication
80,675 Citations1948Claude E. Shannon
Bell System Technical JournalA Mathematical Theory of Communication
9,723 Citations1948Claude E. Shannon
Journal of the Operational Research SocietyNetwork Flows: Theory, Algorithms, and Applications.
8,138 Citations1994David K. Smith, Ravindra K. Ahuja +2 more
In-depth, self-contained treatments of shortest path, maximum flow, and minimum cost flow problems, including descriptions of polynomial-time algorithms for these core models are presented.
Combining labeled and unlabeled data with co-training
5,604 Citations1998Avrim Blum, Tom M. Mitchell
An Information-Theoretic Definition of Similarity
3,662 Citations1998Dekang Lin
This work presents an informationtheoretic definition of similarity that is applicable as long as there is a probabilistic model and demonstrates how this definition can be used to measure the similarity in a number of different domains.
Unsupervised word sense disambiguation rivaling supervised methods
2,408 Citations1995David Yarowsky
An unsupervised learning algorithm for sense disambiguation that, when trained on unannotated English text, rivals the performance of supervised techniques that require time-consuming hand annotations.
A statistical approach to machine translation
1,699 Citations1990Peter F. Brown, John Cocke +6 more
The application of the statistical approach to translation from French to English and preliminary results are described and the results are given.
N-gram-based text categorization
1,503 Citations1994William B. Cavnar, John M. Trenkle
An N-gram-based approach to text categorization that is tolerant of textual errors is described, which worked very well for language classification and worked reasonably well for classifying articles from a number of different computer-oriented newsgroups according to subject.
Computer Networks and ISDN SystemsSyntactic clustering of the Web
1,347 Citations1997Andrei Broder, S. Glassman +2 more
An efficient way to determine the syntactic similarity of files is developed and applied to every document on the World Wide Web, and a clustering of all the documents that are syntactically similar is built.
Computational linguistics in 1990
1,247 Citations1990Hans Karlgren
Computational linguistics is what the authors make it, and the concept also evolves with each of us in the vague, successive way in which other words in human language so fruitfully keep changing.
Discriminative training and maximum entropy models for statistical machine translation
1,044 Citations2001Franz Josef Och, Hermann Ney
A framework for statistical machine translation of natural languages based on direct maximum entropy models, which contains the widely used source-channel approach as a special case and shows that a baseline statistical machinetranslation system is significantly improved using this approach.
Wiley Encyclopedia of Electrical and Electronics EngineeringComputational Linguistics
834 Citations1999Fred Popowich
Inducing multilingual text analysis tools via robust projection across aligned corpora
497 Citations2001David Yarowsky, Grace Ngai +1 more
Noise-robust tagger, bracketer and lemmatizer training procedures capable of accurate system bootstrapping from noisy and incomplete initial projections are presented, which significantly exceeds that obtained by direct annotation projection.
Identifying word correspondence in parallel texts
297 Citations1991William A. Gale, Kenneth Church
Researchers in both machine translation and bilingual lexicography have recently become interested in studying parallel texts, bodies of text which are available in multiple languages and who outline a self-organizing method for using these parallel texts to build a machine translation system.
Natural Language EngineeringA hierarchical Dirichlet language model
276 Citations1995David Mackay, Linda C. Bauman Peto
A hierarchical probabilistic model whose predictions are similar to those of the popular language modelling procedure known as ‘smoothing’ is discussed, and the methods prove to be about equally accurate, with the hierarchical model using fewer computational resources.
Computational LinguisticsModels of Translational Equivalence among Words
269 Citations2000I. Dan Melamed
This article presents methods for biasing statistical translation models to reflect bitext properties, and shows how a statistical translation model can take advantage of preexisting knowledge that might be available about particular language pairs.
Inducing multilingual POS taggers and NP bracketers via robust projection across aligned corpora
247 Citations2001David Yarowsky, Grace Ngai
This paper investigates the potential for projecting linguistic annotations including part-of-speech tags and base noun phrase bracketings from one language to another via automatically word-aligned parallel corpora.
Mining the Web for bilingual text
238 Citations1999Philip Resnik
The preliminary STRAND results are extended by adding automatic language identification, scaling up by orders of magnitude, and formally evaluating performance.
An unsupervised method for word sense tagging using parallel corpora
232 Citations2001Mona Diab, Philip Resnik
An unsupervised method for word sense disambiguation that exploits translation correspondences in parallel corpora is presented, using pseudo-translations, created by machine translation systems, in order to make possible the evaluation of the approach against a standard test set.
Building a shallow Arabic Morphological Analyzer in one day
173 Citations2002Kareem Darwish
The paper presents a rapid method of developing a shallow Arabic morphological analyzer based on automatically derived rules and statistics that will only be concerned with generating the possible roots of any given Arabic word.
Bootstrapping
148 Citations2001Steven Abney
This chapter develops a new conception of the bootstrapping objection, according to which the problem with the idea that intentions provide reasons is that the problem with the idea that intentions provide reasons is that the authors find ourselves with a normative theory that issues incoherent advice.
Computers and the HumanitiesThe Bible as a Parallel Corpus: Annotating the ‘Book of 2000 Tongues’
127 Citations1999Philip Resnik, Mari Broman Olsen +1 more
A project to annotate biblical texts in order to create an aligned multilingual Bible corpus for linguistic research, particularly computational linguistics, including automatically creating and evaluating translation lexicons and semantically tagged texts is reported on.
Statistical Machine Translation
120 Citations2000Franz Josef Och, Hermann Ney
Evaluating translational correspondence using annotation projection
114 Citations2001Rebecca Hwa, Philip Resnik +2 more
An empirical study that quantifies the degree to which syntactic dependencies are preserved when parses are projected directly from English to Chinese and shows that although the direct correspondence assumption is often too restrictive, a small set of principled, elementary linguistic transformations can boost the quality of the projected Chinese parses by 76% relative to the unimproved baseline.
Lecture notes in computer scienceParallel Strands: A Preliminary Investigation into Mining the Web for Bilingual Text
99 Citations1998Philip Resnik
A method for automatically finding parallel translated documents on the Web is presented, conceptually simple, fully language independent, and scalable, and preliminary evaluation results indicate that the method may be accurate enough to apply without human intervention.
BITS: a method for bilingual text search over the Web
94 Citations1999Xiaoyi Ma, Mark Liberman
This paper presents BITS (Bilingual Internet Text Search), a system which harvests multilingual texts over the World Wide Web with virtually no human intervention, and proves that the method is very successful.
A best-first alignment algorithm for automatic extraction of transfer mappings from bilingual corpora
82 Citations2001Arul Menezes, Stephen D. Richardson
An algorithm is described that uses a best-first strategy and a small alignment grammar to significantly improve the quality of the mappings extracted in translation systems that automatically extract transfer mappings from bilingual corpora.
Text REtrieval ConferenceA TREC Evaluation of Query Translation Methods For Multi-Lingual Text Retrieval.
78 Citations1995Mark M. Davis, Ted Dunning
Five methods for query translation that rely on lexical-transfer and corpus-based methods for creating multi-lingual queries were investigated and the resulting queries produced were used in a competitive information-retrieval environment and the results evaluated by the TREC evaluation group.
Parallel web text mining for cross-language IR
72 Citations2000Chen Jiang, Jian‐Yun Nie
This paper describes a parallel text mining system called PTMiner (Parallel Text Miner) for the Web environment as well as its implementation using a distributed model and database technology.
arXiv (Cornell University)Automatic Discovery of Non-Compositional Compounds in Parallel Data
67 Citations1997I. Dan Melamed
Improved cross-language retrieval using backoff translation
56 Citations2001Philip Resnik, Douglas W. Oard +1 more
Two techniques for combining evidence from dictionary-based and corpus-based translation lexicons are presented, and it is shown that backoff translation outperforms a technique based on merging lexicons.
Inducing information extraction systems for new languages via cross-language projection
46 Citations2002Ellen Riloff, Charles Schafer +1 more
This paper presents a novel method for rapidly creating IE systems for new languages by exploiting existing IE systems via cross-language projection, and explores several ways of realizing both the transfer and learning processes using off-the-shelf machine translation systems, induced word alignment, attribute projection and transformation-based learning.
Semi-automatic acquisition of domain-specific translation lexicons
43 Citations1997Philip Resnik, I. Dan Melamed
The utility of an algorithm for translation lexicon acquisition (SABLE), used previously on a very large corpus to acquire general translation lexicons, when that algorithm is applied to a much smaller corpus to produce candidates for domain-specifictranslation lexicons is investigated.
Automatic Construction of Weighted String Similarity Measures
37 Citations1999Jörg Tiedemann
Three approaches to the automatic generation of language dependent string matching functions are presented and it is shown that two of them are good candidates for bilingual text-processing.
Filtering noisy parallel corpora of web pages
22 Citations2002Jian‐Yun Nie, Jian Cai
From words to corpora
18 Citations2002Noah A. Smith
This paper presents a technique for discovering translationally equivalent texts comprised of the application of a matching algorithm at two different levels of analysis and a well-founded similarity score that is adaptable to varying levels of multilingual resource availability.
Spanish Language Processing at University of Maryland: Building Infrastructure for Multilingual Applications
10 Citations2001Clara I. Cabezas, Bonnie J. Dorr +1 more
This work describes the construction of lexical resources, tool creation, building of an aligned parallel corpus, and an approach to automatic treebank creation, which has been developing using Spanish data, based on projection of English syntactic dependency information across a parallel corpus.
University Libraries (University of Maryland)Detection of Translational Equivalence
7 Citations2001Noah A. Smith
Experimental results which support the hypothesis that translational equivalence is empirically observable and suggest profitable directions for improving performance on this recognition task are shown.