Content-Based Representation and Retrieval of Visual Media: A State-of-the-Art Review
Published 19 August 2007
Philippe Aigrain, Hao Zhang, Dragutin Petković
Citations382
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
This paper reviews a number of recently available techniques in content analysis of visual media and their application to the indexing, retrieval, abstracting, relevance assessment, interactive perception, annotation and re-use of visual documents.
Keywords
Computer Science
ComputerQuery by image and video content: the QBIC system
3,058 Citations1995Myron Flickner, Harpreet Sawhney +10 more
The QBIC system is described and its query capabilities are demonstrated, which allows queries on large image and video databases based on example images, user-constructed sketches and drawings, selected color and texture patterns, camera and object motion, and other graphical information.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIESimilarity of color images
1,744 Citations1995Markus Stricker, Markus Orengo
Two new color indexing techniques are described, one of which is a more robust version of the commonly used color histogram indexing and the other which is an example of a new approach tocolor indexing that contains only their dominant features.
International Journal of Computer VisionPhotobook: Content-based manipulation of image databases
1,564 Citations1996Alex Pentland, Rosalind W. Picard +1 more
The Photobook system is described, which is a set of interactive tools for browsing and searching images and image sequences that make direct use of the image content rather than relying on text annotations to provide a sophisticated browsing and search capability.
Multimedia SystemsAutomatic partitioning of full-motion video
1,275 Citations1993HongJiang Zhang, Atreyi Kankanhalli +1 more
A twin-comparison approach has been developed to solve the problem of detecting transitions implemented by special effects, and a motion analysis algorithm is applied to determine whether an actual transition has occurred.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Photobook: tools for content-based manipulation of image databases</title>
883 Citations1994Alexander P. Pentland, Rosalind W. Picard +1 more
The Photobook system is described, which is a set of interactive tools for browsing and searching images and image sequences that differ from those used in standard image databases in that they make direct use of the image content rather than relying on annotations.
Journal of Electronic ImagingComparison of video shot boundary detection techniques
567 Citations1996John Boreczky
IEEE Transactions on Pattern Analysis and Machine IntelligenceModal matching for correspondence and recognition
456 Citations1995Stan Sclaroff, Alex Pentland
Improved formulation of modal matching utilizes a new type of finite element formulation that allows for an object's eigenmodes to be computed directly from available image information, and is applicable to data of any dimensionality.
IEEE Transactions on Pattern Analysis and Machine IntelligenceAn investigation of methods for determining depth from focus
330 Citations1993J. Ens, Peter Lawrence
A general, matrix-based method using regularization is presented, which eliminates some fundamental problems with inverse filtering: inaccuracies in finding the frequency domain representation, windowing effects, and border effects.
Automatic parsing of TV soccer programs
320 Citations2002Yihong Gong, Lim Teck Sin +3 more
The proposed system can classify a sequence of soccer frames into various play categories, such as shot at left goal, top left corner kick play in right penalty area, in midfield, etc, based on a priori model comprising four major components: a soccer court, a ball, the players and the motion vectors.
ACM Transactions on Information SystemsDesign and evaluation of algorithms for image retrieval by spatial similarity
304 Citations1995Venkat N. Gudivada, Vijay V. Raghavan
An algorithm for computing the spatial similarity between two symbolic images that is robust in the sense that it can deal with translation, scale, and rotational variances in images, and has quadratic time complexity in terms of the total number of objects in both the database and query images.
Video parsing, retrieval and browsing
302 Citations1995Haochuan Zhang, Cheng-Yaw Low +2 more
This paper presents an integrated solution for computer assisted video parsing and content-based video retrieval and browsing that uses video content information provided by a parsing process driven by visual feature analysis.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Scene change detection in an MPEG-compressed video sequence</title>
299 Citations1995Jianhao Meng, Yujen Juan +1 more
An algorithm is proposed for the detection of abrupt scene change and special editing effects such as dissolve in a compressed MPEG/MPEG-2 bitstream with minimal decoding of the bitstream to improve the processing speed for searching a video database of compressed image sequences.
Mosaic based representations of video sequences and their applications
296 Citations2002M. Irani, P. Anandan +1 more
This paper systematically investigates how to go beyond thinking of the mosaic simply as a visualization device, but rather as a basis for efficient representation of video sequences to provide representations at multiple spatial and temporal resolutions and to handle 3D scene information.
IEEE Transactions on Software EngineeringQuery-by-Pictorial-Example
283 Citations1980Ning-San Chang, King‐Sun Fu
Query-by-Pictorial-Example is a relational query language introduced for manipulating queries regarding pictorial relations as well as conventional relations.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Database architecture for content-based image retrieval</title>
272 Citations1992Toshikazu Kato
This paper adopts both an image model and a user model to interpret and operate the contents of image data from the user''s viewpoint, and develops algorithms developed on the TRADEMARK and the ART MUSEUM.
Multimedia SystemsVision texture for annotation
253 Citations1995Rosalind W. Picard, Thomas P. Minka
This paper demonstrates a new application of computer vision to digital libraries — the use of texture forotation, the description of content, and is equipped with the ability to choose the one that “best explains” the regions selected by the user for annotating.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Scene change detection and content-based sampling of video sequences</title>
245 Citations1995Behzad Shahraray
Signal Processing Image CommunicationEfficient representations of video sequences and their applications
219 Citations1996Michal Irani, P. Anandan +3 more
This paper systematically investigates how to go beyond thinking of the mosaic simply as a visualization device, but rather as a basis for an efficient and complete representation of video sequences.
Multimedia SystemsAutomatic parsing and indexing of news video
212 Citations1995HongJiang Zhang, Shuang Yeo Tan +2 more
This paper's work on using domain knowledge to parse news video programs and to index them on the basis of their visual content is presented and experimental results are discussed in detail.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Comparison of video shot boundary detection techniques</title>
212 Citations1996John Boreczky, Lawrence A. Rowe
This paper presents a comparison of several shot boundary detection and classification techniques and their variations including histograms, discrete cosine transform, motion vector, and block matching methods.
Multimedia Tools and ApplicationsProduction model based digital video segmentation
205 Citations1995Arun Hampapur, Ramesh Jain +1 more
The approach proposed here is inspired and influenced by well established video production processes and is used to classify the transition effects used in video and to design automatic edit effect detection algorithms.
Proceedings - International Conference on Image ProcessingEfficient matching and clustering of video shots
203 Citations2002Minerva M. Yeung, Bede Liu
A scheme to match video shots and to cluster them by taking into account the temporal variations within individual shots is proposed, using a much reduced representation for a video shot.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIEStatistical approach to scene change detection
178 Citations1995Ishwar K. Sethi, Nilesh Patel
The statistical approach permits accurate detection of scene changes induced through straight as well as optical cuts and avoids an unnecessary decompression-compression cycle since it is common to store and transmit digital video in compressed form.
Salient video stills
174 Citations1993Laura Teodosio, Walter Bender
A new class of images called salient stills is demonstrated and a software development platform for their creation is discussed and a by-product of the salient still process is a structured representation of moving image data.
A magnifier tool for video data
169 Citations1992Michael Mills, Jonathan Cohen +1 more
An interface prototype, the Hierarchical Video Magnifier, is described, which allows users to work with a video source at fine-levels of detail while maintaining an awareness of temporal context.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Clustering methods for video browsing and annotation</title>
168 Citations1996Di Zhong, Hao Zhang +1 more
A generalized top-down hierarchial clustering process, which adopts partition clustering recursively at each level of the hierarchy, is studied and used to build hierarchical views of video shots.
Communications of the ACMVirtual video editing in interactive multimedia applications
157 Citations1989Wendy E. Mackay, Glorianna Davenport
Drawing examples from four interrelated sets of multimedia tools and applications under development at MIT, the authors examine the role of digitized video in the areas of entertainment, learning, research, and communication.
IMPACT
157 Citations1991Hirotada Ueda, Takafumi Miyatake +1 more
A new approach to achieving a natural-motion-picture dedicated multi-media authoring system is proposed, which has several image-processing functions that include object extraction of the picture, semi-automatic visualization of motion pictures structure, and certain descriptions of the scene.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Video browsing using clustering and scene transitions on compressed sequences</title>
156 Citations1995Minerva M. Yeung, Boon-Lock Yeo +2 more
A new technique for extracting a hierarchical decomposition of a complex video selection for browsing purposes that combines visual and temporal information to capture the important relations within a scene and between scenes in a video, thus allowing the analysis of the underlying story structure with no a priori knowledge of the content.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Video indexing using motion vectors</title>
146 Citations1992Akihito Akutsu, Yoshinobu Tonomura +2 more
A video indexing method that uses motion vectors to 'identify' video sequences and corresponding icons is presented, based on the identification of discrete cut points and camera operations made possible by analyzing motion vectors.
IEEE SoftwareA construction set for multimedia applications
120 Citations1989Matthew E. Hodges, Russ Sasnett +1 more
The authors describe an interface system called Athena Muse, an experiment kit for the construction of multimedia learning environments developed with Muse, which offers a diverse set of complementary interaction techniques, styles, and devices.
Lecture notes in computer scienceThe stratification system a design environment for random access video
107 Citations1993Thomas G. Aguierre Smith, Glorianna Davenport
Annotation of the video stream allows the movie maker to make decisions based on specific content of video and in the best case enables a machine to help in that process.
VideoMAP and VideoSpaceIcon
104 Citations1993Yoshinobu Tonomura, Akihito Akutsu +2 more
The basic concept of VideoMAP and VideoSpaceIcon, which allow the user's creativity to directly interact with the essential features of each video by offering spatial and temporal clues, are introduced.
Global zoom/pan estimation and compensation for video compression
103 Citations1991Yi-tong Tse, R. L. Baker
Simulations suggest the algorithm is robust and accurate, and can significantly reduce both the energy of the motion compensated residual image as well as the zeroth-order entropy of the local displacement vector field.
Communications of the ACMInformedia Digital Video Library
98 Citations1995Mike Christel, Takeo Kanade +5 more
The Informedia Digital Video Library Project is developing new technologies for creating full-content search and retrieval digital video libraries and is creating a testbed that will enable K-12 students to access, explore, and retrieve science and mathematics materials from the digital video library.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIEScheme for visual feature-based image indexing
97 Citations1995HongJiang Zhang, Di Zhong
This paper presents the initial work in developing an efficient indexing scheme using artificial neural network, which focuses on eliminating unlikely candidates rather than pin-pointing the targets directly in order to achieve content-based image indexing and retrieval.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Comparison of automatic video segmentation algorithms</title>
95 Citations1996Apostolos Dailianas, Robert B. Allen +1 more
This work examines several segmentation techniques across different types of videos that can identify over 90% of the real shot transitions but have a high percentage of false positives.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Video parsing using compressed data</title>
95 Citations1994Hao Zhang, Chien Yong Low +2 more
Algorithms to automate the video parsing task, including video partitioning and video clip classification according to camera operations using compressed video data, based on correlation of motion vectors are presented.
Proceedings - International Conference on Image ProcessingOn the extraction of DC sequence from MPEG compressed video
93 Citations2002Boon-Lock Yeo, B. Liu
This paper examines the direct reconstruction of reduced images formed from the collection of DC coefficients in intra-coded DCT compressed video from motion-compensated P-frames and B-frames of MPEG compressed video.
Computers & GraphicsThe automatic real-time analysis of film editing and transition effects and its applications
90 Citations1994Philippe Aigrain, Philippe Joly
An adaptive method for the automatic analysis of film editing that achieves detection of shot change even when done through smooth transition effects, and identifies classes of transition special effects that can be applied in real-time on a flow of digital or analog motion picture data of any resolution.
Video tomography
89 Citations1994Akihito Akutsu, Yoshihide Tonomura
This method is targeted towards video handling applications that attribute extracted motion data into a video index, and will allow the visualization of scenes spatiotemporally so that video may be accessed intuitively and spatially in relation to its temporal location.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Developing power tools for video indexing and retrieval</title>
88 Citations1994HongJiang Zhang, Stephen W. Smoliar
The Video Classification Project is presented, an effort toward automating content-based video indexing and retrieval, at the Institute of Systems Science of the National University of Singapore.
Medium knowledge-based macro-segmentation of video into sequences
88 Citations1997Philippe Aigrain, Philippe Joly +1 more
This paper presents a method for obtaining automatically a macro-segmentation in sequences based on the application of rules expressing the local (in time) clues which are given by the medium contents to enable identiication of more macroscopical changes.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Feature management for large video databases</title>
83 Citations1993Farshid Arman, Arding Hsu +1 more
This paper presents a frame selection procedure which takes advantage of the fact that the incoming video is encoded using one of several existing DCT- based standards and is performed in the frequency domain prior to video decoding.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Automatic generation of pictorial transcripts of video programs</title>
68 Citations1995Behzad Shahraray, David Gibbon
An automatic authoring system for the generation of pictorial transcripts of video programs which are accompanied by closed caption information is presented, which generates a compact multimedia presentation of the contents of the video program which lends itself to efficient storage and transmission.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Content-based video browsing tools</title>
60 Citations1995HongJiang Zhang, Stephen W. Smoliar +1 more
A set of video browsing tools which utilize video content information resulting from a parsing process and a detail description of both sequential and time-space browsing tools are presented.
DSpace@MIT (Massachusetts Institute of Technology)Structure out of sound
60 Citations1993Michael Jerome Hawley
Several implementations of sound sensors are presented, including a polyphonic pitch extractor, a music detector, and a talker recognizer for indexing conversations that filter sound to produce event-list scripts that describe the contents in some sense.
Journal of the Optical Society of America AEntropy-based depth from focus
52 Citations1993V. Michael Bove
Video and Image Semantics: Advanced Tools for Telecommunications
38 Citations1994Alex Pentland, Rosalind W. Picard +2 more
A demonstration system which gathers and presents video over standard ISDN telephone lines and develops tools for building and browsing multimedia databases and using these databases to automatically create multimedia presentations.
Signal Processing Image CommunicationEfficient automatic analysis of camera work and microsegmentation of video using spatiotemporal images
36 Citations1996Philippe Joly, Hae-Kwang Kim
An efficient method for the automatic detection of camera work changes using spatiotemporal images called X-ray images is proposed and various steps in the spatiotsemporal image analysis process are introduced which significantly improves its robustness and decreases its computational complexity.
Journal of the American Society for Information ScienceA method for automatically abstracting visual documents
25 Citations1993Mark E. Rorvig
A method for automatically selecting key frames from visual documents that may be used to represent the total image sequence of visual documents in visual libraries, hypermedia systems, and training guides is described.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Representation-based user interfaces for the audiovisual library of the year 2000</title>
20 Citations1995Philippe Aigrain, Philippe Joly +2 more
This communication addresses the user interface issues which will arise from the new situation of computerized access to digitized documents, and describes possible visual interfaces for various temporal media and proposes methods for the economically feasible large scale processing of documents.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIESpatial similarity measures for multimedia applications
15 Citations1995Venkat N. Gudivada
The notion of spatial similarity for 2D symbolic images is formalized and a framework for characterizing the robustness of spatial similarities algorithms with respect to their ability to deal with translation, scale, rotation (both perfect and multiple) variants as well as the variants obtained by an arbitrary composition of translation, Scale, and rotation is provided.
Proceedings of SPIE, the International Society for Optical Engineering/Proceedings of SPIE<title>Two-stage strategy for indexing and presenting video</title>
12 Citations1994Mourad Cherfaoui, Christian Bertin
The goal of this work is to offer ways of accessing videos which are similar to those of structured texts, which will make it possible to quickly become acquainted with the video content, quickly access the part of the video of particular interest and have entry points in the video.
European Conference on Artificial IntelligenceFRAMER: a persistent portable representation library
9 Citations1994Kenneth B. Haase
Information Processing & ManagementEvaluation of navigational links between images
6 Citations1992Philippe Aigrain, Véronique Longueville
A general framework for the evaluation of sets of links designed for user navigation between images in pictorial data banks is described and an experiment conducted for the observation of user navigation behaviour in this set of links.
ACM SIGOIS BulletinDevelopment of A Video Database System
6 Citations1994Hong Jiang Zhang, Stephen W. Smoliar +2 more
Building an effective video library will require a set of power tools whose functionality will suppor t the tasks of parsing, indexing, retrieval, an d browsing for video source material, which are developed in this paper.
IFLA JournalSoftware Research for Video Libraries and Archives
5 Citations1995Philippe Aigrain
Another face of digital video; the ability to use digital processing of video documents for the indexing, retrieval and access to video documents is emphasized.
HAL (Le Centre pour la Communication Scientifique Directe)Actes des Journées d'informatique musicale 2017
5 Citations2017Pierre Couprie, Cécile Davy-Rigaux +5 more
Signal Processing Image CommunicationFilmic space-time diagrams for video structure representation
5 Citations1996Sean Butler, Alan P. Parkes
A novel approach to computer representation for interactive film editing is proposed which involves the isolation of several principles of filmicity and the combination of these with an event structure and film grammar to produce a system which can intelligently edit a film.