Evaluating system utility and conceptual fit using CASSM
International Journal of Human-Computer StudiesPublished 26 November 2007
Ann Blandford, Thomas R. G. Green, Dominic Furniss, Stephann Makri
Citations43
SJR quartileQ1
SJR score1.39
SNIP2.34
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper presents CASSM, a method, which focuses on the quality of 'fit' between users and an interactive system and complements established evaluation methods by focusing on conceptual structures rather than procedures.
Abstract
There is a wealth of user-centred evaluation methods (UEMs) to support the analyst in assessing interactive systems. Many of these support detailed aspects of use – for example: Is the feedback helpful? Are labels appropriate? Is the task structure optimal? Few UEMs encourage the analyst to step back and consider how well a system supports users’ conceptual understandings and system utility. In this paper, we present CASSM, a method which focuses on the quality of ‘fit’ between users and an interactive system. We describe the methodology of conducting a CASSM analysis and illustrate the approach with three contrasting worked examples (a robotic arm, a digital library system and a drawing tool) that demonstrate different depths of analysis. We show how CASSM can help identify re-design possibilities to improve system utility. CASSM complements established evaluation methods by focusing on conceptual structures rather than procedures. Prototype tool support for completing a CASSM analysis is provided by Cassata, an open source development.
Keywords
Computer Science
Sage eBooksBasics of qualitative research : techniques and procedures for developing grounded theory
38,901 Citations1998Anselm L. Strauss, Juliet Corbin
The Psychology of Human-Computer Interaction
4,079 Citations2018Stuart K. Card, Thomas P. Moran +1 more
This work focuses on applying Psychology to Design Reprise, and the GOMS Model of Manuscript Editing, as an application to Information-Processing Psychology.
Cognitive Work Analysis: Toward Safe, Productive, and Healthy Computer-Based Work
1,690 Citations1999Kim J. Vicente
Cognitive Work Analysis
1,349 Citations1999Kim J. Vicente
Journal of Visual Languages & ComputingUsability Analysis of Visual Programming Environments: A ‘Cognitive Dimensions’ Framework
1,240 Citations1996T.R.G. Green, Marian Petre
This paper applies the cognitive dimensions framework to two commercially-available dataflow languages and concludes that it is effective and insightful; other HCI-based evaluation techniques focus on different aspects and would make good complements.
Human-Computer InteractionDamaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods
484 Citations1998Wayne D. Gray, Marilyn Salzman
In this review, the design of 5 experiments that compared usability evaluation methods (UEMs) are examined, showing that small problems in the way these experiments were designed and conducted call into serious question what the authors thought they knew regarding the efficacy of various UEMs.
Cognitive dimensions of notations
472 Citations1990T. R. G. Green
The dimensions concept can be justified by pointing to areas where progress is slow, as I have done above; or by criticising existing systems, or by raising design issues in relation to present growth areas.
International Journal of Human-Computer InteractionThe Evaluator Effect: A Chilling Fact About Usability Evaluation Methods
419 Citations2001Morten Hertzum, Niels Ebbe Jacobsen
Human Factors The Journal of the Human Factors and Ergonomics SocietyUse of the Critical Decision Method to Elicit Expert Knowledge: A Case Study in the Methodology of Cognitive Task Analysis
404 Citations1998Robert R. Hoffman, Beth Crandall +1 more
The Critical Decision Method research illustrates the sorts of knowledge representation products that can arise from cognitive task analysis and shows how one can approach methodological issues surrounding Cognitive task analysis, including questions about data quality and method reliability, efficiency, and utility.
International Journal of Human-Computer InteractionCriteria For Evaluating Usability Evaluation Methods
404 Citations2001H. Rex Hartson, Terence S. Andre +1 more
ACM Transactions on Information SystemsGetting around the task-artifact cycle
389 Citations1992John M. Carroll, Mary Beth Rosson
The approach leverages development practices of current HCI with methods and concepts to support a shift toward using broad and explicit design rationale to reify where in a design process, why the authors are there, and to guide reasoning about where they might go from there.
Cambridge University Press eBooksResearch Methods for Human-Computer Interaction
375 Citations2008Paul Cairns, Paul Cairns +12 more
This is the first book to provide a single resource through which a range of commonly used research methods in HCI are introduced and how to develop new methodologies and write up your research.
Elsevier eBooksHCI Models, Theories, and Frameworks
375 Citations2003
HCI Models, Theories, and Frameworks fills a huge void in the education and training of advanced HCI students by providing a thorough pedagogical survey of the multidisciplinary science of HCI.
People and Computers
277 Citations1994Gilbert Cockton, Stephen Draper +1 more
What do usability evaluators do in practice?
230 Citations2006Mie Nørgaard, Kasper Hornbæk
The study shows how practical realities rarely discussed in the literature on usability evaluation influence sessions, and discusses implications for usability researchers and professionals, including techniques for fast-paced analysis and tools for capturing observations during sessions.
interactionsEvaluating usability methods
182 Citations2003Dennis Wixon
Dennis Wixon argues that consideration of the factors that determine success of usability efforts in product development organizations will fundamentally change the terms of the debate about the relative effectiveness of different methodologies.
Information Systems JournalFormalized systems development methodologies: a critical perspective
180 Citations1996Brian Fitzgerald
This paper identifies a number of arguments and pressures which question the value of methodologies, and reports the results of a field study which investigated the role ofmethodologies in practice.
Behaviour and Information TechnologyTracking the effectiveness of usability evaluation methods
164 Citations1997Bonnie E. John, Steven J. Marks
A case study that tracks usability problems predicted with six usability evaluation methods through a development process and concludes that Predictive methods are not as effective as the HCI field would like.
Programmable user models for predictive evaluation of interface designs
131 Citations1989Richard M. Young, T. R. G. Green +1 more
The research findings include analyses of the cognitive processes involved in the use of interactive computer systems, and a number of issues to be resolved in future cognitive models.
Elsevier eBooksThe Usability Engineering Framework for Product Design and Evaluation
124 Citations1997Dennis Wixon, Chauncey Wilson
The chapter discusses the issues and pitfalls that provide background for establishing, marketing, and refining a usability engineering process in a development organization and lists the ways in which other design models and methods complement the usability engineering framework.
Getting into a system
111 Citations1983Thomas P. Moran
A task analysis technique, called ETIT analysis, is introduced and it is shown how this analysis can be used to assess the potential transfer of knowledge from one system to another, i.e., how much knowing one system helps with learning another.
Information Systems JournalFormalized systems development methodologies: a critical perspective
107 Citations1996Brian Fitzgerald
A number of arguments and pressures which question the value of methodologies are identified, and the results of a field study which investigated the role ofmethodologies in practice are reported.
Learning and using the cognitive walkthrough method
76 Citations1995Bonnie E. John, Hilary Packer
International Journal of Human-Computer StudiesThe skull beneath the skin: entity-relationship models of information artifacts
71 Citations1996T. R. G. Green, David Benyon
It is concluded that, as a cheap, coarse-grained, and easy-to-learn modelling technique, ERMIA usefully fills a gap in the range of available HCI analysis techniques.
Human-Computer InteractionThe Nature of Device Models: The Yoked State Space Hypothesis and Some Experiments With Text Editors
59 Citations1990Stephen A. Payne, Helen Ruth Squibb +1 more
The yoked state space (YSS) model allows an important distinction concerning the mental representation of procedures, and three experiments addressed the claims of this model, illustrated through a simple description of a device model for a cut-and-paste text editor.
New Methodological Developments in FTA
43 Citations2008Fabiana Scapolo, Alan L. Porter
Human-Computer InteractionScoping Analytical Usability Evaluation Methods: A Case Study
42 Citations2008Ann Blandford, Joanne Hyde +2 more
A detailed comparison of eight analytical UEMs applied to a robotic arm interface and the findings were systematically compared against video data of the arm in use, yielding insights into the scope of each method.
Software Engineering JournalSpecifying user knowledge for the design of interactive systems
33 Citations1996Ann Blandford, Richard M. Young
The Instruction Language and associated principles provide a means of evaluating system design in relation to user knowledge prior to implementation and formalising that description and adding principles about users' cognitive processes.
Behaviour and Information TechnologyCASSM and cognitive walkthrough: usability issues with ticket vending machines
31 Citations2004Iain Connell, Ann Blandford +1 more
This work focuses on the ability of two analytical usability evaluation methods, namely CASSM and Cognitive Walkthrough, to identify usability issues underlying the use made of two London Underground ticket vending machines, and assess the similarities and differences between the issues depicted by the two methods.
Multiple Viewpoints on Computer Supported Team Work: A Case Study on Ambulance Dispatch
29 Citations2002Ann Blandford, B. L. William Wong +2 more
This analysis shows how OSM can be used to identify misfits between domain and device and aid additional reasoning about the way in which a new or proposed computer system can both support and enhance existing work structures.
International Journal of Human-Computer StudiesTraining software engineers in a novel usability evaluation technique
26 Citations1998Ann Blandford, Simon Buckingham Shum +1 more
Work to train software engineering students to use a user-centred language for describing and analysing interface designs called the “Programmable User Model Instruction Language”, or IL shows that after 6 h of training, students have a good grasp of the syntax of the notation, and start using notational affordances to support their reasoning, but that their reasoning is still limited by a poor grasp ofThe underlying cognitive theory.
Ontological Sketch Models: Highlighting User—System Misfits
13 Citations2004Iain Connell, Thomas Green +1 more
It is shown how OSM can identify potential misfits between user and system, and how an OSM analysis can be edited, conventionalized and viewed in tabular form, thereby allowing automatic highlighting of user-system misfits.
Formalising an understanding of user-system misfits
7 Citations2005Ann Blandford, Trg Green +1 more
Measuring the conceptual fitness of an application in a computing ecosystem
7 Citations2004Idris Hsi
The computing ecosystem framework with its associated concepts, use niches, use potential, and activation potential is presented and it is shown how the ecosystem framework allows us to characterize the usefulness of an application through the concept of fitness.
Enhancing Contextual Analysis to Support the Design of Development Tools
4 Citations2007Chris Roast, Andy Dearden +1 more
This paper focuses on interactive systems that support certain types of ‘knowledge work’, which requires the use of abstractions and pre-planning, which can impose considerable cognitive burdens on the user.
UCL Discovery (University College London)Concept-based Analysis of Surface and Structural Misfits (CASSM) Tutorial notes
4 Citations2004Ann Blandford, Iain Connell +1 more
The name CASSM emphasises the importance of misfits and provides a ‘bridge’ between the core ideas underpinning work on mental models and design issues, and may make prior work onmental models more readily accessible to design practice.