The effectiveness of query-specific hierarchic clustering in information retrieval
Information Processing & ManagementPublished 1 July 2002
Anastasios Tombros, Robert Villa, C. J. van Rijsbergen
Citations180
SJR quartileQ1
SJR score2.06
SNIP2.91
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
If hierarchic clustering is applied to search results (query-specific clustering), then it has the potential to increase the retrieval effectiveness compared both to that of static clustering and of conventional IFS.
Abstract
Hierarchic document clustering has been widely applied to information retrieval (IR) on the grounds of its potential improved effectiveness over inverted file search (IFS). However, previous research has been inconclusive as to whether clustering does bring improvements. In this paper we take the view that if hierarchic clustering is applied to search results (query-specific clustering), then it has the potential to increase the retrieval effectiveness compared both to that of static clustering and of conventional IFS. We conducted a number of experiments using five document collections and four hierarchic clustering methods. Our results show that the effectiveness of query-specific clustering is indeed higher, and suggest that there is scope for its application to IR.
Keywords
Computer Science
Journal of the American Statistical AssociationHierarchical Grouping to Optimize an Objective Function
19,236 Citations1963Joe H. Ward
Journal of the American Statistical AssociationHierarchical Grouping to Optimize an Objective Function
3,231 Citations1963Joe H. Ward
Information Retrieval: Data Structures and Algorithms
2,428 Citations1992William B. Frakes, Ricardo Baeza‐Yates
For programmers and students interested in parsing text, automated indexing, its the first collection in book form of the basic data structures and algorithms that are critical to the storage and retrieval of documents.
Web document clustering
1,055 Citations1998Oren Zamir, Oren Etzioni
To satisfy the stringent requirements of the Web domain, an incremental, linear time algorithm called Suffix Tree Clustering (STC) is introduced which creates clusters based on phrases shared between documents, showing that STC is faster than standard clustering methods in this domain.
ACM SIGIR ForumPivoted Document Length Normalization
863 Citations2017Amit Singhal, Chris Buckley +1 more
Pivoted normalization is presented, a technique that can be used to modify any normalization function thereby reducing the gap between the relevance and the retrieval probabilities, and two new normalization functions--pivoted unique normalization and piuotert byte size normalization are presented.
Reexamining the cluster hypothesis
828 Citations1996Marti A. Hearst, Jan Pedersen
This work systematically evaluates Scatter/Gather in this context and finds significant improvements over similarity search ranking alone and provides evidence validating the cluster hypothesis which states that relevant documents tend to be more similar to each other than to non-relevant documents.
Information Processing & ManagementRecent trends in hierarchic document clustering: A critical review
707 Citations1988Peter Willett
Algorithms that can be used to allow the implementation of hierarchic agglomerative clustering methods for document retrieval, and experimental evidence suggests that nearest neighbor clusters provide a reasonably efficient and effective means of including interdocument similarity information in document retrieval systems.
Information Storage and RetrievalThe use of hierarchic clustering in information retrieval
522 Citations1971N. Jardine, C. J. van Rijsbergen
It is shown that cluster-based retrieval strategies can be devised which are as effective as linear associative retrieval strategies and much more efficient.
Advantages of query biased summaries in information retrieval
438 Citations1998Anastasios Tombros, Mark Sanderson
Investigation into the utility of document summarisation in the context of information retrieval and the application of so called query biased (or user directed) summaries indicate that the use of query biased summaries significantly improves both the accuracy and speed of user relevance judgements.
The Journal of Experimental MedicineInformation Retrieval Research
196 Citations1982Robert N Oddy
The electrocardiographic records taken during vagus stimulation by pressure from children suffering with chronic heart disease have shown that the stimulation of the vagi in these cases is strikingly effectual, leading to the belief that an important feature in the pathological physiology ofThese cases is hypertonus of the cardiac accelerator nerves.
The Computer JournalComparison of Hierarchic Agglomerative Clustering Methods for Document Retrieval
144 Citations1989Abdelmoula El-Hamdouchi
Information SystemsA model of cluster searching based on classification
143 Citations1980W. Bruce Croft
A probabilistic model of cluster searching based on query classification is described and it is tested with retrieval experiments which indicate that it can be more effective than heuristic cluster searches and cluster searches based on other models.
Information Processing & ManagementUsing clustering and SuperConcepts within SMART: TREC 6
131 Citations2000Chris Buckley, Mandar Mitra +2 more
The major focus for TREC 6 is on trying to maintain the balance of the query — attempting to ensure the various aspects of the original query are appropriately addressed, especially while adding expansion terms.
eCommons (Cornell University)The Effectiveness and Efficiency of Agglomerative Hierarchic Clustering in Document Retrieval
122 Citations1985Ellen M. Voorhees
The main goal of this thesis is to compare clustered file searches and inverted file searches in order to determine under what circumstances one search is to be preferred over the other.
Journal of DocumentationHIERARCHIC AGGLOMERATIVE CLUSTERING METHODS FOR AUTOMATIC DOCUMENT CLASSIFICATION
117 Citations1984Alan Griffiths, Lesley A. Robinson +1 more
The results would suggest that the single linkage method, which has been used extensively in previous work on document clustering, is not the most effective procedure of those tested, although it should be emphasized that the experiments have used only small document test collections.
Journal of the American Society for Information ScienceUsing interdocument similarity information in document retrieval systems
109 Citations1986Alan Griffiths, H. Claire Luckhurst +1 more
A comparison of the use of clusters consisting of a document together with that document with which it is most similar with conventional best match searches suggests that the two types of search are of comparable effectiveness, but they retrieve noticeably different sets of relevant documents.
Information Processing & ManagementDocument clustering: An evaluation of some experiments with the cranfield 1400 collection
88 Citations1975C. J. van Rijsbergen, W. Bruce Croft
A variety of retrieval strategies applied to this hierarchy are evaluated in terms of effectiveness and efficiency and comparisons are made between these results and those of similar experiments in document clustering on the Smart project.
An interface for navigating clustered document sets returned by queries
71 Citations1993Robert B. Allen, Pascal Obry +1 more
HJI%KLI MONQPSR(TVUWNYX T[Z \ NQNQI KL]_^+`LNQ]-nQI MON$abR c P NWd+^+‘Lc POKLI+efMgX+N ZOMgPOh UiMOh+PgNjckRla c+UWhC] NQI M-ZON.
Almost-constant-time clustering of arbitrary corpus subsets4
54 Citations1997Craig Silverstein, Jan Pedersen
An algorithm for alrnostconstant-time clustering of arbitrary corpus subsets of large corpora is given, which is never slower than clustering the document set from scratch, and for medium-sised and large sets it is significantly faster.
Journal of Information ScienceTechniques for the measurement of clustering tendency in document retrieval systems
44 Citations1987Abdelmoula El-Hamdouchi, Peter Willett
It is shown that the three tests are not in complete agreement with each other in their evaluation of the degree of clustering tendency present in seven document test collections, and it is suggested that the density test gives the most useful results.
Journal of the American Society for Information ScienceThe retrieval effectiveness of five clustering algorithms as a function of indexing exhaustivity
40 Citations1995Robert Burgin
Evaluations of retrieval effectiveness, based on three measures of optimal retrieval performance, confirm earlier findings that the performance of a retrieval system based on single‐link clustering varies as a function of indexing exhaustively but fail to find similar patterns for other clustering methods.
Information Storage and RetrievalFurther experiments with hierarchic clustering in document retrieval
38 Citations1974C. J. van Rijsbergen
A framework for the evaluation of cluster-based retrieval strategies is constructed and these strategies are shown to be dependent on the method of cluster representation (cluster profile) adopted.
Lecture notes in computer scienceEvaluating a Visual Navigation System for a Digital Library
33 Citations1998A. Leouski, James Allan
A method for measuring the amount of structure in the organization and how this knowledge can be used to refine the system is presented and it is shown that increasing the dimensionality of the organization generally improves its quality.
Information Processing & ManagementStructure of hierarchic clusterings: implications for information retrieval and for multivariate data analysis
25 Citations1984Fionn Murtagh
In this study, three different coefficients are defined, each of which quantify the symmetry-asymmetry (balancedness-unbalancedness) of hierarchic clusterings on a scale from 0 to 1.
Information Processing & ManagementDeciphering cluster representations
20 Citations2001Yasemin Kural, Stephen Robertson +1 more
The difficulties involved in representing clusters for users’ evaluation in a concise but easily interpretable form are discussed, based on findings and user feedback from a user study investigating the effectiveness of search output clustering.
Journal of the American Society for Information ScienceSubject and citation indexing. Part II: The optimal, cluster-based retrieval performance of composite representations
20 Citations1991W. M. Shaw
Measures of cluster‐based retrieval effectiveness are computed for five composite representations in the cystic fibrosis (CF) Document Collection and the effectiveness of the exhaustive representation composed of references and citations is materially superior to the effective of exhaustive composite representations that include subject descriptions.
Program electronic library and information systemsIdentification of duplicate and near‐duplicate full‐text records in database search‐outputs using hierarchic cluster analysis
11 Citations1995john kirriemuir, Peter Willett
It is demonstrated that clustering is capable of grouping documents that are identical to, or closely‐related to, other documents in the search‐output on the basis of their term similarities.