Towards a Theory of Semantic Space

A theoretical framework for semantic space models is developed by synthesizing theoretical analyses from vector space information re- trieval and categorical data analysis with new basic re- search.

Abstract

Towards a Theory of Semantic Space Will Lowe ([email protected]) Center for Cognitive Studies Tufts University; MA 21015 USA Abstract This paper adds some theory to the growing literature of semantic space models. We motivate semantic space models from the perspective of distributional linguistics and show how an explicit mathematical formulation can provide a better understanding of existing models and suggest changes and improvements. In addition to pro- viding a theoretical framework for current models, we consider the implications of statistical aspects of language data that have not been addressed in the psychological modeling literature. Statistical approaches to language must deal principally with count data, and this data will typically have a highly skewed frequency distribution due to Zipf’s law. We consider the consequences of these facts for the construction of semantic space models, and present methods for removing frequency biases from se- mantic space models. Introduction There is a growing literature on the empirical adequacy of semantic space models across a wide range of sub- ject domains (Burgess et al., 1998; Landauer et al., 1998; Foltz et al., 1998; McDonald and Lowe, 1998; Lowe and McDonald, 2000). However, semantic space mod- els are typically structured and parameterized differently by each researcher. Levy and Bullinaria (2000) have ex- plored the implications of parameter changes empirically by running multiple simulations, but there has up until now been no work that places semantic space models in an overarching theoretical framework; consequently there there are few statements of how semantic spaces ought to be structured in the light of their intended pur- pose. In this paper we attempt to develop a theoretical framework for semantic space models by synthesizing theoretical analyses from vector space information re- trieval and categorical data analysis with new basic re- search. The structure of the paper is as follows. The next sec- tion brie¤y motivates semantic space models using ideas from distributional linguistics. We then review Zipf’s law and its consequences the distributional character of linguistic data. The £nal section presents a formal de£- nition of semantic space models and considers what ef- fects different choices of component have on the result- ing models. Motivating Semantic Space Firth (1968) observed that “you shall know a word by the company it keeps”. If we interpret company as lex- ical company, the words that occur near to it in text or speech, then two related claims are possible. The £rst is unexceptional: we come to know about the syntactic character of a word by examining the other words that may and may not occur around it in text. Syntactic theory then postulates latent variables e.g. parts of speech and branching structure, that control the distributional prop- erties of words and restrictions on their contexts of occur- rence. The second claim is that we come to know about the semantic character of a word by examining the other words that may and may not occur around it in text. The intuition for this distributional characterization of semantics is that whatever makes words similar or dis- similar in meaning, it must show up distributionally, in the lexical company of the word. Otherwise the suppos- edly semantic difference is not available to hearers and it is not easy to see how it may be learned. If words are similar to the extent that they occur in the similar contexts then we may de£ne a statistical re- placement test (Finch, 1993) which tests the meaning- fulness of the result of switching one word for another in a sentence. When a corpus of meaningful sentences is available the test may be reversed (Lowe, 2000a), and un- der a suitable representation of lexical context, we may hold each word constant and estimate its typical sur- rounding context. A semantic space model is a way of representing similarity of typical context in a Euclidean space with axes determined by local word co-occurrence counts. Counting the co-occurrence of a target word with a £xed set of D other words makes it possible to position the target in a space of dimension D. A target’s position with respect to other words then expresses similarity of lexical context. Since the basic notion from distributional linguistics is ‘intersubstitutability in context’, a semantic space model is effective to the extent it realizes this idea accurately. Zipf’s Law The frequency of a word is (approximately) proportional to the reciprocal of its rank in a frequency list (Zipf, 1949; Mandelbrot, 1954). This is Zipf’s Law. Zipf’s law ensures dramatically skewed distributions for almost