Comparing corpora using frequency profiling
Published 1 January 2000Open access
Paul Rayson, Roger Garside
Citations587
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The method can be used to discover key words in the corpora which differentiate one corpus from another and is shown to have applications in the study of social differentiation in the use of English vocabulary, profiling of learner English and document analysis in the software engineering process.
Abstract
This paper describes a method of comparing corpora which uses frequency profiling. The method can be used to discover key words in the corpora which differentiate one corpus from another. Using annotated corpora, it can be applied to discover key grammatical or word-sense categories. This can be used as a quick way in to find the differences between the corpora and is shown to have applications in the study of social differentiation in the use of English vocabulary, profiling of learner English and document analysis in the software engineering process.
Keywords
PsychologyComputer ScienceArts and Humanities
Accurate methods for the statistics of surprise and coincidence
2,688 Citations1993Ted Dunning
Journal of the Royal Statistical Society Series B (Statistical Methodology)Multinomial Goodness-Of-Fit Tests
1,352 Citations1984Noel Cressie, Timothy R. C. Read
Literary and Linguistic ComputingRepresentativeness in Corpus Design
1,204 Citations1993Douglas Biber
The paper distinguishes among various ways that linguistic features can be distributed within and across texts; it analyzes the distributions of several particular features, and it discusses the implications of these distributions for corpus design.
Springer series in statisticsGoodness-of-Fit Statistics for Discrete Multivariate Data
746 Citations1988Timothy R. C. Read, Noel Cressie
Choice Reviews OnlineGoodness-of-fit statistics for discrete multivariate data
744 Citations1989
This goodness of fit statistics for discrete multivariate data helps people to read a good book with a cup of tea in the afternoon, instead they are facing with some infectious virus inside their laptop.
Edinburgh University Press eBooksStatistics for Corpus Linguistics
534 Citations2019Michael Oakes
This book in the Edinburgh Textbooks in Empirical Linguistics series is a comprehensive introduction to the statistics currently used in corpus linguistics.
The BNC Handbook: Exploring the British National Corpus with SARA
481 Citations1998Guy Aston, Lou Burnard
This textbook is designed to provide a detailed understanding of the principles and practices underlying the use of large language corpora in exploratory learning and English language teaching and research and on the search tool SARA (SGML Aware Retrieval Application).
Ethnographically-informed systems design for air traffic control
415 Citations1992Richard Bentley, John A. Hughes +5 more
The ethnographic work undertaken studying air traffic control as a cooperative activity is discussed, some of the difficulties in collaboration between software developers and sociologists are described and how ethnographic studies have influenced the systems design process is described.
BioScienceA Comparative Review On Aggression Konrad Lorenz The Territorial Imperative Robert Ardrey
401 Citations1968Richard D. Alexander, Donald W. Tinkle
Learner English on Computer
347 Citations2014Sylviane Granger
Learner corpus data in the foreign language classroom: form-focused instruction and data-driven learning, Sylviane Granger and Chris Tribble Software mentioned in the book Bibliography Subject Index.
Representativeness in Corpus Design
283 Citations1994Douglas Biber
Corpus Annotation: Linguistic Information from Computer Text Corpora
280 Citations1997Roger Garside, Geoffrey Leech +1 more
International Journal of Corpus LinguisticsSocial Differentiation in the Use of English Vocabulary
239 Citations1997Paul Rayson, Geoffrey Leech +1 more
International Statistical ReviewPearson's X 2 and the Loglikelihood Ratio Statistic G 2 : A Comparative Review
123 Citations1989Noel Cressie, Timothy R. C. Read
Automatic profiling of learner texts
78 Citations2014Sylviane Granger, Paul Rayson
Though limited to one specific type of interlanguage, the approach presented here is applicable to any learner variety and demonstrates a potential of automatic profiling for revealing the stylistic characteristics of EFL texts.
Using Word Frequency Lists to Measure Corpus Homogeneity and Similarity between Corpora
65 Citations1997Adam Kilgarriff
Goldsmiths (University of London)Measures for Corpus Similarity and Homogeneity
59 Citations1998Adam Kilgarriff, Tony Rose
An operational denition of corpus similarity is presented which addresses or circumvents the problems, using purposebuilt sets of iknown-similarity corporai, and three variants of the information theoretic measure ‘perplexity’ are evaluated.
Lancaster EPrints (Lancaster University)Assisting requirements engineering with semantic document analysis
19 Citations2000Paul Rayson, Roger Garside +1 more
This paper shows that tools exploiting natural language processing techniques, in particular semantic analysis, are able to assist in retrieval from large documents.
Lancaster EPrints (Lancaster University)Social differentiation in the use of English vocabulary: some analyses of the conversational component of the British National Corpus
10 Citations1997Rayson, Paul, Leech, Geoffrey +1 more