Evaluating computer-supported cooperative work
Published 6 November 2004
Dennis C. Neale, John M. Carroll, Mary Beth Rosson
Citations219
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The problems involved in evaluating remote collaborations are described, and a multifaceted evaluation framework is presented that approaches the problem from the relationships underlying joint awareness, communication, collaboration, coordination, and work coupling.
Abstract
Evaluating distributed CSCW applications is a difficult endeavor. Frameworks and methodologies for structuring this type of evaluation have become a central concern for CSCW researchers. In this paper we describe the problems involved in evaluating remote collaborations, and we review some of the more prominent conceptual frameworks of group interaction that have driven CSCW evaluation in the past. A multifaceted evaluation framework is presented that approaches the problem from the relationships underlying joint awareness, communication, collaboration, coordination, and work coupling. Finally, recommendations for carrying out multifaceted evaluations of remote interaction are provided.
Keywords
PsychologyComputer Science
Journal of Environmental PsychologyQualitative Data Analysis: An Expanded Sourcebook
45,597 Citations1994
This book presents a step-by-step guide to making the research results presented in reports, slideshows, posters, and data visualizations more interesting, and describes how coding initiates qualitative data analysis.
The American Journal of PsychologyPlans and Situated Actions: The Problem of Human-Machine Communication
5,333 Citations1990Barry H. Kantowitz, Lucy Suchman
This paper presents a meta-modelling architecture for human-machine communication that automates the very labor-intensive and therefore time-heavy and therefore expensive and expensive process of designing and implementing communication systems.
Educational Evaluation and Policy AnalysisToward a Conceptual Framework for Mixed-Method Evaluation Designs
5,311 Citations1989Jennifer C. Greene, Valerie J. Caracelli +1 more
Contemporary Sociology A Journal of ReviewsMixed Methodology: Combining Qualitative and Quantitative Approaches
5,250 Citations1999Shelia R. Cotten, Abbas Tashakkori +1 more
This chapter discusses research design issues for Mixed Method and Mixed Model Studies, as well as applications, applications, and future directions of mixed model research.
Journal of the American Statistical AssociationMeasurement, Design, and Analysis: An Integrated Approach.
4,066 Citations1992Sandra G. Funk, Elazar J. Pedhazur +1 more
ACM Computing SurveysThe interdisciplinary study of coordination
3,439 Citations1994Thomas W. Malone, Kevin Crowston
This survey characterizes an emerging research area, sometimes called coordination theory, that focuses on the interdisciplinary study of coordination, that uses and extends ideas about coordination from disciplines such as computer science, organization theory, operations research, economics, linguistics, and psychology.
Choice Reviews OnlineContext and consciousness: activity theory and human-computer interaction
2,510 Citations1996
Part 1 Activity theory basics: introduction activity theory and human-computer interaction, Bonnie A. Nardi activity theory as a potential framework for human- computer interaction research, and Victor Kaptelinin studying context.
Awareness and coordination in shared workspaces
2,468 Citations1992Paul Dourish, Victoria Bellotti
A study of shared editor use is discussed which suggests that awareness information provided and exploited passively through the shared workspace, allows users to move smoothly between close and loose collaboration, and to assign and coordinate work dynamically.
Human-Computer InteractionDistance Matters
1,928 Citations2000Gary M. Olson, Judith S. Olson
This work focuses on the sociotechnical conditions required for effective distance work and brings together the results with four key concepts: common ground, coupling of work, collaboration readiness, and collaboration technology readiness.
Long Range PlanningConnections: New ways of working in the networked organization
1,775 Citations1992
Small Groups as Complex Systems: Formation, Coordination, Development, and Adaptation
810 Citations2000Holly Arrow, Joseph McGrath +1 more
This paper discusses the theory of group formation, history, and some research strategies for Studying Groups as Complex Systems.
Mixed methodology combining qualitative and quantitative approaches
787 Citations1998Tashakkori, Abbas, Teddlie, Charles
Addison-Wesley Longman Publishing Co., Inc. eBooksThe mythical man-month (anniversary ed.)
655 Citations1995Jr. Frederick P. Brooks
The added chapters contain a crisp condensation of all the propositions asserted in the original book, including Brooks' central argument in The Mythical Man-Month: that large programming projects suffer management problems different from small ones due to the division of labor.
Readings in human-computer interaction : toward the year 2000
450 Citations1995Ronald M. Baecker
This book discusses how to design to Fit Human Capabilities to Software Development Contexts, and the challenges of doing so in the rapidly changing environment.
International Journal of Human-Computer StudiesNotification and awareness: synchronizing task-oriented collaborative activity
370 Citations2003John M. Carroll, Dennis C. Neale +3 more
This work analysed awareness breakdowns in use of the Virtual School system--stemming from problems related to the collaborative situation, group, task and tool support--to motivate the concept of activity awareness.
Decision Support SystemsThe impact of technological support on groups: An assessment of the empirical research
310 Citations1989Alain Pinsonneault, Kenneth L. Kraemer
Five Major implications stem from the analysis of empirical findings on the impacts of technological support on group: there is lack of research on some important “formal” factors of groups, there is a need to move away from laboratory settings to field study in organization settings, and more research is needed to understand how the structure imposed by the technological supports affect group processes.
ACM Transactions on Computer-Human InteractionTask analysis for groupware usability evaluation
303 Citations2003David Pinelle, Carl Gutwin +1 more
Collaboration Usability Analysis (CUA) is a new modeling technique that focuses on the teamwork that goes on in a group task rather than the taskwork and provides evaluators with a framework in which they can simulate the realistic use of a groupware system and identify usability problems that are caused by the groupware interface.
Elsevier eBooksMETHODOLOGY MATTERS: DOING RESEARCH IN THE BEHAVIORAL and SOCIAL SCIENCES
291 Citations1995Joseph E. McGrath
This chapter focuses on the tools with which researchers in the social and behavioral sciences go about doing research and presents some of the inherent limits, as well as the potential strengths, of various features of the research process by which behavioral and social scientists do research.
Psychology Press eBooksTime Matters in Groups
255 Citations2014Joseph E. McGrath
This chapter deals with some temporal features of group work, and with how those temporal matters are affected when technological tools are added, and an array of potential technological tools that can mediate and modify cooperative work in groups are considered.
International Journal of Occupational Safety and ErgonomicsHandbook of human factors and ergonomics
253 Citations2023Krzysztof Kȩdzior
What Are Workplace Studies For?
236 Citations1995Lydia Plowman, Yvonne Rogers +1 more
The role of workplace studies from the CSCW literature are considered, categorised according to which phase of the design process they most inform, and the tensions between providing explanatory accounts and usable design recommendations are discussed.
Computer-Supported Cooperative Work: Introduction to Distributed Applications
177 Citations2010Uwe M. Borghoff, Johann Schlichter
The following chapter demonstrates typical usage scenarios and their characteristics of computer-supported cooperative work and presents several classification models, in particular time space taxonomy.
The development of cooperation
173 Citations2000John M. Carroll, George Chin +2 more
It is suggested that there may be a long-term developmental unfolding of roles and relationships in participatory design.
Computer-Supported Cooperative Work
156 Citations2000Uwe M. Borghoff, Johann Schlichter
Situated evaluation for cooperative systems
127 Citations1994Michael B. Twidale, David Randall +1 more
It is argued that informal procedures are a powerful and cost effective method for dealing with specific evaluation issues in the context of CSCW but that wider issues are more problematic.
A review of groupware evaluations
113 Citations2002David Pinelle, Carl Gutwin
The main conclusions from the review are that more attention must be paid to evaluating groupware systems and that there is room for additional evaluation techniques that are simple and low in cost.
Communications of the ACMGroup dynamics and ubiquitous computing
113 Citations2002Jonathan Grudin
From "Here and Now" to "Everywhere and Forever," these are the songs that have captured the zeitgeist of the 21st Century in song.
Classroom BRIDGE
82 Citations2003Craig H. Ganoe, Jacob Somervell +5 more
This paper demonstrates and develops an object replication approach to integrating synchronous and asynchronous distributed work for a platform incorporating both desktop and large-screen devices.
ACM SIGCHI BulletinThe Denver model for groupware design
77 Citations1996Tony Salvador, Jean Scholtz +1 more
The Denver Model consists of three submodels: goals and requirements, design and technology, and a description of the framework is provided and evaluation strategies are described in this paper.
Elsevier eBooksResearch on Computer Supported Cooperative Work
70 Citations1997Gary M. Olson, Judith S. Olson
This chapter considers a conceptual perspective that provides a framework for describing the human use of CSCW systems and discusses some general research issues for the future.
Computer Supported Cooperative Work (CSCW)Meetings of the Board: The Impact of Scheduling Medium on Long Term Group Coordination in Software Development
63 Citations1999Steve Whittaker, Heinrich Schwarz
It is found that the medium of the schedule has a major impact on coordination problems, and design recommendations are made about how the benefits of material tools could be incorporated into electronic groupware systems.
A laboratory method for studying activity awareness
61 Citations2004Gregorio Convertino, Dennis C. Neale +3 more
Evidence is presented showing that this method represents a valid model of real collaboration, based on participants' active engagement, lively negotiation, and awareness difficulties, and the ability to define, reproduce, and systematically manipulate collaborative situations allowed us to assess the effect of realistic conditions on activity awareness in remote collaboration.
Computer supported cooperative workCSCW Requirements and Evaluation
51 Citations1996Peter Thomas
Computer concooperative work Activity theory 92 ferencing system 50, 57, 107, 157 Adaptive structuration Centralisation of power Computer-mediated 123 75 communication 107 Adelaide-Loughborough Change management Computerised knowstudies 161 strategies 143 ledge base 149 Agent provocateur 100 change requests 24.
Human Factors The Journal of the Human Factors and Ergonomics SocietyA Framework for Epistemological Analysis in Empirical (Laboratory and Field) Studies
50 Citations2000Yan Xiao, Kim J. Vicente
A framework for epistemological analysis is described that can help serve this purpose and its application to two prototypical cases of cognitive engineering research: laboratory experiments and field studies is discussed.
Educational Technology & SocietyThe Virtual School: An integrated collaborative environment for the classroom
49 Citations2000Philip L. Isenhour, John M. Carroll +3 more
The types of collaborative activities for which the Virtual School has been used, the features of the environment designed to address the constraints of the classroom, and experiences with using the environment for a variety of cross-age and cross-school collaborative projects in six classrooms over several school years are described.
Interacting with ComputersPETRA: participatory evaluation through redesign and analysis
47 Citations1995Susi Ross, Magnus Ramage +1 more
This study investigated collaborative writing, both in a face-to-face context, and supported by a computer-based group editor, and found that the collaborative tool must be particularly sensitive to issues of awareness, communication, focus and ownership.
Contemporary Sociology A Journal of ReviewsGroups, Teams, and Social Interaction: Theories and Applications.
45 Citations1993H. Andrew Michener, A. Paul Hare
ACM Computing SurveysEvaluation for collaborative systems
37 Citations1999Laurie Damianos, Lynette Hirschman +6 more
The Evaluation Working Group in the Defense Advanced Research Projects Agency (DARPA) Intelligent Collaboration and Visualization (IC&V) program has developed a methodology for evaluating collaborative systems, which consists of a framework for classification of CSCW systems, metrics and measures related to the various components in the framework, and a scenario-based evaluation approach.
Human Factors The Journal of the Human Factors and Ergonomics SocietyCOMMENTARY Heeding the Legacy of Meister, Brunswik, & Gibson: Toward a Broader View of Human Factors Research
34 Citations1997Kim J. Vicente
The fact that one has to justify a study of human behaviour in a natural setting, where humans ordinarily behave, as opposed to in a laboratory setting, is certainly extraordinary.
International Journal of Human-Computer StudiesDesign guidelines for dealing with breakdowns and repairs in collaborative work settings
33 Citations2000Mark Hartswood, Rob Procter
The ways in which the organization of breast screening work and the design of artefacts such as report forms afford the detection of breakdowns and situations calling for departures from the “standard” procedures are focused on.
Multi-faceted evaluation for complex, distributed activities
29 Citations1999Dennis C. Neale, John M. Carroll
This paper overviews and discusses this emerging methodology for computer-supported cooperative learning, developing techniques to address challenges including systematic sampling, collation of multiple evaluation methods and data, and the use of collaborative critical incidents.
International Journal of Human-Computer StudiesInter-personal awareness and synchronization: assessing the value of communication technologies
26 Citations1996Leon Watts, Andrew Monk +1 more
Two groups of measures are described that assess someone's awareness of the attentional status of their conversational partner, such awareness is necessary for the successful coordination of conversation.
Proceedings of the Human Factors and Ergonomics Society Annual MeetingCollaborative Critical Incident Development
17 Citations2000Dennis C. Neale, Daniel Dunlap +2 more
A Web-based forum called the collaborative critical incident tool (CCIT) is developed, where users (teachers) and researchers collaboratively post, rate incident criticality, and elaborate on critical incidents through sustained dialog, contributing to the understanding of underlying conditions, causes, and implications.
Proceedings of the Human Factors and Ergonomics Society Annual MeetingData Logging: Higher-Level Capturing and Multi-Level Abstracting of User Activities
13 Citations2000Jim Helms, Dennis C. Neale +2 more
This paper extends traditional logging approaches to collaborative multi-user (groupware) systems and shows how higher-level data abstracted from logging can be more effectively combined with data from other usability methods.
Proceedings of the Human Factors and Ergonomics Society Annual MeetingLaboratory Simulation Methods for Studying Complex Collaborative Tasks
7 Citations2004William D. Humphries, Dennis C. Neale +2 more
A new laboratory model developed for studying complex CSCW phenomena is described and findings indicated that there was room for improving the model by making the simulation more ecological and engaging for participants.