A review of performance evaluation measures for hierarchical classifiers
Kent Academic Repository (University of Kent)Published 1 July 2007
Eduardo Paulino da Costa, Ana Carolina Lorena, André C. P. L. F. de Carvalho, Alex A. Freitas
Citations95
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper reviews the main evaluation metrics proposed in the literature to evaluate hierarchical classification models and proposes several performance measures for binary and flat classification problems.
Abstract
Criteria for evaluating the performance of a classifier are an important part in its design. They allow to estimate the behavior of the generated classifier on unseen data and can be also used to compare its performance against the performance of classifiers generated by other classification algorithms. There are currently several performance measures for binary and flat classification problems. For hierarchical classification problems, where there are multiple classes which are hierarchically related, the evaluation step is more complex. This paper reviews the main evaluation metrics proposed in the literature to evaluate hierarchical classification models.
Keywords
Computer Science
Pattern RecognitionThe use of the area under the ROC curve in the evaluation of machine learning algorithms
7,233 Citations1997Andrew P. Bradley
AUC exhibits a number of desirable properties when compared to overall accuracy: increased sensitivity in Analysis of Variance (ANOVA) tests; a standard error that decreased as both AUC and the number of test samples increased; decision threshold independent; and it is invariant to a priori class probabilities.
Lecture notes in computer scienceBeyond Accuracy, F-Score and ROC: A Family of Discriminant Measures for Performance Evaluation
1,794 Citations2006Marina Sokolova, Nathalie Japkowicz +1 more
These measures – Youden's index, likelihood, Discriminant power – are used in medical diagnosis and it is shown that they are interrelated, and they are applied to a case study from the field of electronic negotiations.
BioinformaticsInvestigating semantic similarity measuresacross the Gene Ontology: the relationship betweensequence and annotation
847 Citations2003Phillip Lord, Robert Stevens +2 more
This paper investigates the use of ontological annotation to measure the similarities in knowledge content or 'semantic similarity' between entries in a data resource and shows a simple extension that enables a semantic search of the knowledge held within sequence databases.
Hierarchical document categorization with support vector machines
379 Citations2004Lijuan Cai, Thomas Hofmann
A novel hierarchical classification method that generalizes Support Vector Machine learning and that is based on discriminant functions that are structured in a way that mirrors the class hierarchy is proposed.
Probe, count, and classify
154 Citations2001Panagiotis G. Ipeirotis, Luis Gravano +1 more
This paper introduces a method for automating this classification process by using a small number of query probes to classify a database, which exploits the number of matches that each query probe generates at the database in question.
Hierarchical classification
143 Citations2006Nicolò Cesa‐Bianchi, Claudio Gentile +1 more
A refined evaluation scheme is introduced which turns the hierarchical SVM classifier into an approximator of the Bayes optimal classifier with respect to a simple stochastic model for the labels.
Nucleic Acids ResearchPrediction of functional modules based on comparative genome analysis and Gene Ontology application
140 Citations2005Hongsheng Wu
This work presents a computational method for the prediction of functional modules encoded in microbial genomes and develops a formal measure to quantify the degree of consistency between the predicted and the known modules.
Pattern Recognition LettersMulti-class ROC analysis from a multi-objective optimisation perspective
123 Citations2005Richard Everson, Jonathan E. Fieldsend
An evolutionary algorithm to locate the Pareto front-the optimal trade-off surface between misclassifications of different types is presented and a straightforward multi-class analogue of the Gini coefficient is presented.
Building Hierarchical Classifiers Using Class Proximity
118 Citations1999Ke Wang, Senqiang Zhou +1 more
The notion of class proximity is proposed and cast the hierarchical classi cation as a at classiCation with the class proximity modeling the closeness of classes.
Improving Protein Function Prediction using the Hierarchical Structure of the Gene Ontology
104 Citations2005Roman Eisner, Brett Poulin +3 more
This paper presents a method of creating hierarchy-aware training sets for machine-learned classifiers and shows that, in the case of GO molecular function, it is the most accurate method compared to not considering the hierarchy during training.
Lirias (KU Leuven)Hierarchical multi-classification
72 Citations2002Hendrik Blockeel, Maurice Bruynooghe +3 more
Journal of the American Society for Information Science and TechnologyPerformance measurement framework for hierarchical text classification
71 Citations2003Aixin Sun, Ee‐Peng Lim +1 more
New performance measures for hierarchical classification consist of category similarity measures and distance-based measures that consider the contributions of misclassified documents.
Protein ScienceA categorization approach to automated ontological function annotation
70 Citations2006Karin Verspoor, Judith D. Cohn +2 more
This work presents an application for function prediction from protein sequences using the POSet Ontology Categorizer to produce new annotations by analyzing collections of GO nodes derived from annotations of protein BLAST neighborhoods, and shows that the method provides substantially improved hierarchical precision when applied to the nearest BLAST neighbors of target proteins.
IGI Global eBooksA Tutorial on Hierarchical Classification with Applications in Bioinformatics
64 Citations2008Alex A. Freitas, André C. P. L. F. de Carvalho
Hierarchical Classification of Real Life Documents
56 Citations2001Ke Wang, Senqiang Zhou +1 more
This paper proposes a new technique for automatic classification of documents to address real life requirements of on-line information, and uses Yahoo! for explanation.
National Research Council Canada (Government of Canada)Hierarchical Text Categorization as a Tool of Associating Genes with Gene Ontology Codes
47 Citations2004Svetlana Kiritchenko, Stan Matwin +1 more
The hierarchical text categorization framework is designed, which helps visualize the results, address the scalability issue, improve classiflcation accuracy and trade ofi between precision and recall.
Kent Academic Repository (University of Kent)HIERARCHICAL CLASSIFICATION OF G-PROTEIN-COUPLED RECEPTORS WITH A PSO/ACO ALGORITHM
41 Citations2006Nicholas Holden, Alex A. Freitas
An extensive comparison of four versions of swarm intelligence algorithms based on the proposed algorithm and two versions based on Discrete PSO for discovering classification rules proposed i n the literature is reported.
uO Research (University of Ottawa)Hierarchical text categorization and its application to bioinformatics
26 Citations2006Svetlana Kiritchenko
This dissertation introduces the notion of consistent hierarchical classification that makes classification results more comprehensible and easily interpretable for end-users and uses a novel hierarchical evaluation measure that is superior to the existing hierarchical and non-hierarchical evaluation techniques.
Hierarchical Text Classification Methods and Their Specification
14 Citations2003Aixin Sun, Ee‐Peng Lim +1 more
This chapter proposes a specification language known as HCL (Hierarchical Classification Language), designed to describe a hierarchical classification method including the definition of a category tree and training of classifiers associated with the categories.