The Scientific Evaluation of Music Information Retrieval Systems: Foundations and Future
Computer Music JournalPublished 1 June 2004
J. Stephen Downie
Citations82
SJR quartileQ1
SJR score0.28
SNIP0.56
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This article provides an overview of the current scientific problem facing MIR research and reports upon the findings of the Music Information Retrieval (MIR)/ Music Digital Library (MDL) Evaluation Frameworks Project.
Abstract
June 01 2004 The Scientific Evaluation of Music Information Retrieval Systems: Foundations and Future J. Stephen Downie J. Stephen Downie Graduate School of Library and Information Science University of Illinois at Urbana-Champaign 501 East Daniel Street Champaign, Illinois 61820 USA [email protected] Search for other works by this author on: This Site Google Scholar Author and Article Information J. Stephen Downie Graduate School of Library and Information Science University of Illinois at Urbana-Champaign 501 East Daniel Street Champaign, Illinois 61820 USA [email protected] Online ISSN: 1531-5169 Print ISSN: 0148-9267 © 2004 Massachusetts Institute of Technology2004 Computer Music Journal (2004) 28 (2): 12–23. https://doi.org/10.1162/014892604323112211 Cite Icon Cite Permissions Share Icon Share Facebook Twitter LinkedIn Email Views Icon Views Article contents Figures & tables Video Audio Supplementary Data Peer Review Search Site Citation J. Stephen Downie; The Scientific Evaluation of Music Information Retrieval Systems: Foundations and Future. Computer Music Journal 2004; 28 (2): 12–23. doi: https://doi.org/10.1162/014892604323112211 Download citation file: Ris (Zotero) Reference Manager EasyBib Bookends Mendeley Papers EndNote RefWorks BibTex toolbar search Search Dropdown Menu toolbar search search input Search input auto suggest filter your search All ContentAll JournalsComputer Music Journal Search Advanced Search © 2004 Massachusetts Institute of Technology2004 Article PDF first page preview Close Modal You do not currently have access to this content.
Keywords
Computer ScienceNeuroscience
A Highly Robust Audio Fingerprinting System.
603 Citations2002Jaap Haitsma, Ton Kalker
An audio fingerprinting system that uses the fingerprint of an unknown audio clip as a query on a fingerprint database, which contains the fingerprints of a large library of songs, the audio clip can be identified.
Rwc Music Database: Popular, Classical And Jazz Music Databases.
553 Citations2002Masataka Goto, Hiroki Hashiguchi +2 more
The design policy and specifications of the RWC Music Database are described, a music database (DB) that is available to researchers for common use and research purposes, which contains four original DBs: the Popular Music Database (100 pieces), Royalty-Free Music Database(15 pieces), Classical Music Database ($50 pieces), and Jazz Music Database (£50 pieces).
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Content-based retrieval of music and audio</title>
394 Citations1997Jonathan Foote
A system to retrieve audio documents y acoustic similarity based on statistics derived from a supervised vector quantizer, rather than matching simple pitch or spectral characteristics, which may be applicable to image retrieval as well.
Relevance and Information Behavior.
315 Citations1994Linda Schamber
The author views relevance as a manifestation of human information behavior and excludes work that views relevance only as a matching or computational functions of IR systems.
Journal of Education for Library and Information ScienceReference and Information Services: An Introduction
161 Citations1992J. A. Benson, Richard E. Bopp +1 more
This text is designed to provide the beginning student of library and information science with an overview of the most important tools for general reference work and the concepts and theory behind today's reference services.
Manipulation of music for melody matching
141 Citations1998Alexandra L. Uitdenbogerd, Justin Zobel
This paper analyzes the propeties of music, music perception, and musicdatabase users, and uses the analysis to propose alternative techniques for extracting monophonic melodies from polyphonic music; it is believed that such melodies can subsequently be used formatching of queries to data.
Information Processing & ManagementProblems of music information retrieval in the real world
135 Citations2002Donald Byrd, Tim Crawford
This paper considers some of the most fundamental problems in music information retrieval, challenging the common assumption that searching on pitch (or pitch-contour) alone is likely to be satisfactory for all purposes.
Zenodo (CERN European Organization for Nuclear Research)Content-Based Playlist Generation: Exploratory Experiments.
109 Citations2002Beth Logan
Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)Pats: Realization And User Evaluation Of An Automatic Playlist Generator.
92 Citations2002Steffen Pauws, Berry Eggen
Results showed that PATS playlists contained increasingly more preferred music (increasingly higher precision ), covered more preferred music in the collection (higher coverage ), and were rated higher than randomly assembled playlists.
Lecture notes in computer scienceEvaluation in Information Retrieval
83 Citations2000Stephen Robertson
The components of a traditional laboratory-style evaluation experiment in information retrieval (as exemplified by TREC) are summarized, and some of the issues around this form of experiment are discussed.
International Symposium/Conference on Music Information RetrievalMusart: Music Retrieval Via Aural Queries.
80 Citations2001William P. Birmingham
The MME subsystem performs theme extraction, and two other analysis systems are described that discover structure in audio representations of music that promise to improve search quality and support better browsing and “audio thumbnailing.”
International Symposium/Conference on Music Information RetrievalInterdisciplinary Communities And Research Issues In Music Information Retrieval.
80 Citations2002Joe Futrelle, J. Stephen Downie
In order for MIR to succeed, researchers need to work with real user communities and develop research resources such as reference music collections, so that the wide variety of techniques being developed in MIR can be meaningfully compared with one another.
Research Commons (University of Waikato)Toward A Theory Of Music Information Retrieval Queries: System Design Implications.
75 Citations2002J. Stephen Downie, Sally Jo Cunningham
This paper analyzes a set of 161 music-related information requests posted to the rec.music.country.old-time newsgroup to suggest that similar studies of 'native' music information requests can be used to inform the design of effective, usable music information retrieval interfaces.
Zenodo (CERN European Organization for Nuclear Research)Categories Of Music Description And Search Terms And Phrases Used By Non-Music Experts.
64 Citations2002Jayoung Kim, Nicholas J. Belkin
The results suggest that music IR research should be extended to consider needs other than finding known items, or items identified by formal characteristics, and that understanding music information needs of users should be prioritized to design more sophisticated music IR systems.
A Comparison Of Language Modeling And Probabilistic Text Information Retrieval Approaches To Monophonic Music Retrieval.
51 Citations2000Jeremy Pickens
The goal of this paper is to explore some of the capabilities and limitations of current text information retrieval systems as applied to the task of music retrieval.
Zenodo (CERN European Organization for Nuclear Research)An Audio Front End For Query-By-Humming Systems.
48 Citations2001Emanuele Pollastri
A front end dedicated to the symbolic translation of voice into a sequence of pitch and duration pairs is developed, crucial for the effectiveness of searching for music by melodic similarity.
The Library QuarterlyAsking "Why" Questions in the Reference Interview: A Theoretical Justification
47 Citations1997Patricia Dewdney, Gillian Michell
Columbia Academic Commons (Columbia University)Toward Evaluation Techniques for Music Similarity
32 Citations2003Beth Logan, Daniel P. W. Ellis +1 more
A database, methodology and ground truth for the evaluation of automatic techniques for music similarity and a technique of sharing acoustic features rather than raw audio to avoid copyright problems are described.
Zenodo (CERN European Organization for Nuclear Research)Musical Works As Information Retrieval Entities: Epistemological Perspectives.
29 Citations2001Richard P. Smiraglia
Variability over time, demonstrated empirically, is an innate aspect of the set of all instantiations of a musical work, leading to complexity in the information retrieval domain.
Zenodo (CERN European Organization for Nuclear Research)Figured Bass And Tonality Recognition.
25 Citations2001Jérôme Barthélemy
This paper presents a method for automated harmonic analysis of a music score based on the extraction of a figured bass by means of a template-matching algorithm, where templates for chords can be entirely and easily redefined by the end-user.
International Symposium/Conference on Music Information RetrievalThe Jring System For Computer-Assisted Musicological Analysis.
19 Citations2001Andreas Kornstädt
The requirement analysis that led to JRing’s inception, its IR tools and graphical user interface plus the kind of musical material it works on and the Humdrum-based technical realization of IR operations are discussed.
International Symposium/Conference on Music Information RetrievalA Comparative And Fault-Tolerance Study Of The Use Of N-Grams With Polyphonic Music.
17 Citations2002Shyamala Doraisamy, Stefan Rüger
The experiments show that different ngramming strategies and encoding precision differ widely in their effectiveness, and the retrieval performance of monophonic queries made on a polyphonic music database using the n-gram approach for full-music indexing is investigated.
A QUERY BY EXAMPLE MUSIC RETRIEVAL ALGORITHM
17 Citations2003Hadi Harb, L. CHEN
A set of similarity measures aiming at expressing aspects of music similarity is proposed, based on the distance between statistical distributions of the audio spectrum and it can be applied to the raw audio data with no format restriction.
User Studies: A First Step in Designing an MIR Testbed
15 Citations2002Sally Jo Cunningham
The authors must have a clear idea of what an MIR system’s potential users are looking for, how they describe the object of their search, and how they approach the task of fulfilling their information needs.
Zenodo (CERN European Organization for Nuclear Research)Subject Search For Music: Quantitative Analysis Of Access Point Selection.
15 Citations2000M. Itoh
Query by Humming: How good can it get?
13 Citations2003Bryan Pardo, William P. Birmingham
A simple example experiment comparing three QBH systems to three human listeners indicates existing string matching and Markov model performance does not currently achieve humanlevel performance.
Comparing Aural Music-Information Retrieval Systems
6 Citations2002Bryan Pardo, Colin Meek +1 more
A test-bed needs to represent each data-processing stage, from initial transcription of an audio query, to the relevancy ranking of each piece in the database compared to a particular query, in order to let each research group concentrate on the portion of the task most interesting to that group.
Setting Up an Audio Database for Music Information Retrieval Benchmarking
6 Citations2002Perfecto Herrera
This white paper summarizes some general requirements and issues to be clarified in order to set up a usable database for audio-based MIR benchmarking and concludes with a proposal for calling up a specific task-force that is capable of setting up the needed database in the next twelve months.
Evaluating a Music Information Retrieval System - TREC Style
5 Citations2002Thomas Sødring, Alan F. Smeaton
The methodology and results of a TREC style evaluation of a music information retrieval system called Ceolaire are reported.
Indiana university digital music library project
5 Citations2001Jon Dunn, Eric J. Isaacson
The Indiana University Digital Music Library project plans to create a digital library testbed system containing music in a variety of formats, designed to support research and education in the field of music and to serve as a platform for digital library research.
A Task-Oriented Approach for the Development of a Test Collection for Music Information Retrieval
4 Citations2002Massimo Melucci, Nicola Orio
This paper addresses the design of a test collection for CBMR, by applying a task-oriented approach to provide assessors with a number of topics and to build musical queries starting from music documents, or documents excerpts, being particularly relevant to the topic.
Open Position Multilingual Orchestra Conductor. Lifetime Opportunity.
4 Citations2003Eloi Batlle, Enric Guaus +1 more
ACM SIGIR ForumReport on the panels and workshops of the music information retrieval (MIR) and music digital library (MDL) evaluation frameworks project
3 Citations2003J. Stephen Downie
Music Information Retrieval and Music Digital Library research brings together computer scientists, audio engineers, librarians, musicologists, educators and business executives in a common effort to provide robust mechanisms for organizing, storing and accessing the world’s everincreasing volume of music.