Weight functions impact on LSA performance

Published 1 January 2001

Preslav Nakov, Antonia Popova, Plamen Mateev, James Bourchier

Citations105

TL;DR

Experimental results of usage of LSA for analysis of English literature texts and preliminary transformations of the frequency text-document matrix with different weight functions are tested on the basis of control subsets.

Abstract

This paper presents experimental results of usage of LSA for analysis of English literature texts. Several preliminary transformations of the frequency text-document matrix with different weight functions are tested on the basis of control subsets. Additional clustering based on correlation matrix is applied in order to reveal the latent structure. The algorithm creates a shaded form matrix via singular values and vectors. The results are interpreted as a quality of the transformations and compared to the control set tests. 1.

Keywords

Computer Science

Journal of the American Society for Information ScienceIndexing by latent semantic analysis

12,677 Citations1990Scott Deerwester, Susan Dumais +3 more

Discourse ProcessesAn introduction to latent semantic analysis

4,817 Citations1998Thomas K. Landauer, Peter W. Foltz +1 more

The adequacy of LSA's reflection of human knowledge has been established in a variety of ways, for example, its scores overlap those of humans on standard vocabulary and subject matter tests; it mimics human word sorting and category judgments; it simulates word‐word and passage‐word lexical priming data.

Behavior Research Methods, Instruments, & ComputersImproving the retrieval of information from external sources

576 Citations1991Susan Dumais

A statistical method is described called latent semantic indexing, which models the implicit higher order structure in the association of words and objects and improves retrieval performance by up to 30%.

View PDF

ACM Journal of Computer DocumentationThe measurement of readability

505 Citations2000George R. Klare

Journal of the American Society for Information ScienceHow effective is suffixing?

388 Citations1991Donna Harman

Three measures were selected which evaluate performance at given rank cutoff points, such as those cor- responding to a screenful of document titles,such as those responding to the lists of the top ranked documents.

Recognizing text genres with simple metrics using discriminant analysis

293 Citations1994Jussi Karlgren, Douglass R. Cutting

A simple method for categorizing texts into pre-determined text genre categories using the statistical standard technique of discriminant analysis is demonstrated with application to the Brown corpus.

View PDF

LSI meets TREC: a status report

103 Citations1992Susan Dumais

Describes the Latent Semantic Indexing approach, an extension of the vector retrieval method and the use of singular-value decomposition applied to the TREC collection.

SVDPACKC (Version 1.0) User''s Guide

90 Citations1993Michael W. Berry, Theresa Do +3 more

This software package implements Lanczos and subspace iteration-based methods for determining several of the largest singular triplets (singular values and corresponding left- and right-singular vectors) for large sparse matrices.

The Computer JournalDowndating the Latent Semantic Indexing Model for Conceptual Information Retrieval

24 Citations1998D. I. Witter

Implementing the DRM method within LSI++ not only provides downdating functionality, but is less time consuming than recomputing the SVD when removing a term, document or both.

Information Processing & ManagementText windows and phrases differing by discipline, location in document, and syntactic structure

16 Citations1996Robert M. Losee

Different syntactic structures in sublanguages are examined, and their use is considered for discriminating between specific academic disciplines and, more generally, between theory vs practice or knowledge vs applications-oriented documents.