The Measurement of Observer Agreement for Categorical Data
BiometricsPublished 1 March 1977Open access
J. Richard Landis, Gary G. Koch
Citations78,934
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies is presented and tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interob server agreement are developed as generalized kappa-type statistics.
Abstract
This paper presents a general statistical methodology for the analysis of multivariate categorical data arising from observer reliability studies. The procedure essentially involves the construction of functions of the observed proportions which are directed at the extent to which the observers agree among themselves and the construction of test statistics for hypotheses involving these functions. Tests for interobserver bias are presented in terms of first-order marginal homogeneity and measures of interobserver agreement are developed as generalized kappa-type statistics. These procedures are illustrated with a clinical diagnosis example from the epidemiological literature.
Keywords
Decision SciencesMathematics
BMJRoB 2: a revised tool for assessing risk of bias in randomised trials
30,746 Citations2019Jonathan A C Sterne, Jelena Savović +26 more
The Cochrane risk-of-bias tool has been updated to respond to developments in understanding how bias arises in randomised trials, and to address user feedback on and limitations of the original tool.
Systematic ReviewsPreferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement
26,507 Citations2015PRISMA-P Group, David Moher +7 more
A reporting guideline is described, the Preferred Reporting Items for Systematic reviews and Meta-Analyses for Protocols 2015 (PRISMA-P 2015), which consists of a 17-item checklist intended to facilitate the preparation and reporting of a robust protocol for the systematic review.
BMJROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions
19,002 Citations2016Jonathan A C Sterne, Miguel A. Hernán +33 more
Risk of Bias In Non-randomised Studies - of Interventions is developed, a new tool for evaluating risk of bias in estimates of the comparative effectiveness of interventions from studies that did not use randomisation to allocate units or clusters of individuals to comparison groups.
Psychological BulletinWeighted kappa: Nominal scale agreement provision for scaled disagreement or partial credit.
8,537 Citations1968Jacob Cohen
The Kw provides for the incorpation of ratio-scaled degrees of disagreement (or agreement) to each of the cells of the k * k table of joi.
JAMAIncidence of Adverse Drug Reactions in Hospitalized Patients
4,782 Citations1998Jason Lazarou, Bruce Pomeranz +1 more
The incidence of serious and fatal adverse drug reactions in US hospitals was found to be extremely high, and data suggest that ADRs represent an important clinical issue.
New England Journal of MedicineIncidence of Adverse Events and Negligence in Hospitalized Patients
4,666 Citations1991Troyen A. Brennan, Lucian L. Leape +7 more
The incidence of adverse events, defined as injuries caused by medical management, and of the subgroup of such injuries that resulted from negligent or substandard care, in New York State in 1984 were estimated.
BiometricsAn Application of Hierarchical Kappa-type Statistics in the Assessment of Majority Agreement among Multiple Observers
3,808 Citations1977J. Richard Landis, Gary G. Koch
A subset of 'observers who demonstrate a high level of interobserver agreement can be identified by using pairwise agreement statistics betweeni each observer and the internal majority standard opinion on each subject.
Journal of the American Statistical AssociationEpidemiologic Research: Principles and Quantitative Methods.
2,942 Citations1984Lawrence S. Mayer, David G. Kleinbaum +2 more
The Annals of Mathematical StatisticsTransformations Related to the Angular and the Square Root
2,615 Citations1950Murray F. Freeman, John W. Tukey
Canadian Medical Association JournalThe Canadian Adverse Events Study: the incidence of adverse events among hospital patients in Canada
2,350 Citations2004G. Ross Baker
The overall incidence rate of AEs of 7.5% in this study suggests that, of the almost 2.5 million annual hospital admissions in Canada similar to the type studied, about 185 000 are associated with an AE and close to 70 000 of these are potentially preventable.
Archives of Public HealthMetaprop: a Stata command to perform meta-analysis of binomial data
2,175 Citations2014Victoria Nyawira Nyaga, Marc Arbyn +1 more
Metaprop was applied on two published meta-analyses of HPV-infection in women with a Pap smear showing ASC-US and cure rate after treatment for cervical precancer using cold coagulation, both of which showed a pooled HPV-prevalence of 43%.
BMJ OpenPlea for routinely presenting prediction intervals in meta-analysis
2,141 Citations2016Joanna IntHout, John P. A. Ioannidis +2 more
The prediction interval reflects the variation in treatment effects over different settings, including what effect is to be expected in future patients, such as the patients that a clinician is interested to treat, in meta-analyses.
BMJAdverse events in British hospitals: preliminary retrospective record review
2,074 Citations2001Charles Vincent, Graham Neale +1 more
These results suggest that adverse events are a serious source of harm to patients and a large drain on NHS resources.
BMJ Quality & SafetyThe incidence and nature of in-hospital adverse events: a systematic review
1,784 Citations2008Eefje N. de Vries, Maya A. Ramrattan +3 more
A systematic review of the literature on in-hospital adverse events found that a substantial part of these events are preventable and interventions aimed at preventing these events have the potential to make a substantial difference.
BiometricsAnalysis of Categorical Data by Linear Models
1,522 Citations1969James E. Grizzle, C F Starmer +1 more
The special cases of linear functions and logarithmic functions of the 7rin are developed in detail, and some examples of how the general approach can be used to analyze various types of categorical data are presented.
Psychological BulletinLarge sample standard errors of kappa and weighted kappa.
1,504 Citations1969Joseph L. Fleiss, Jacob Cohen +1 more
Annals of Internal MedicineEvaluation of the Quality of Prognosis Studies in Systematic Reviews
1,452 Citations2006Jill A. Hayden, Pierre Côté +1 more
A review of reviews to describe methods used to assess the quality of prognosis studies and to describe how well current practices assess potential biases is to develop recommendations to guide future quality appraisal.
BiometricsA General Methodology for the Analysis of Experiments with Repeated Measurement of Categorical Data
620 Citations1977Gary G. Koch, J. Richard Landis +3 more
This paper is concerned with the analysis of multivariate categorical data which are obtained from repeated measurement experiments and appropriate test statistics are developed through the application of weighted least squares methods.
New England Journal of MedicineIatrogenic Illness on a General Medical Service at a University Hospital
574 Citations1981Knight Steel, Paul M. Gertman +2 more
PubMedCritical appraisal of the health research literature: prevalence or incidence of a health problem.
550 Citations1998Patricia Loney, L W Chambers +3 more
Criteria that can be used by health professionals to critically appraise research articles that estimate the prevalence or incidence of a disease or health problem are identified and discussed.
BiometricsMeasuring Agreement between Two Judges on the Presence or Absence of a Trait
504 Citations1975Joseph L. Fleiss
Using the binary (positive-negative) case as a model, some of the proposed indexes for measuring agreement between two judges on a categorical scale are presented and critically evaluates.
PubMedAdverse events in New Zealand public hospitals I: occurrence and impact.
491 Citations2002Peter Davis, Roy Lay-Yee +4 more
The study provides representative base parameters that can contribute to the wider understanding, and potential improvement, of patient safety and the quality of care in New Zealand public hospitals.
BMC Health Services ResearchThe occurrence, types, consequences and preventability of in-hospital adverse events – a scoping review
414 Citations2018René Schwendimann, Catherine Blatter +3 more
Evidence regarding the occurrence of AEs confirms earlier estimates that a tenth of inpatient stays include adverse events, half of which are preventable, and indicates automated methods for identifying AE using electronic health records have the potential to overcome various methodological issues and biases related to retrospective medical record review studies.
BMJComparison of three methods for estimating rates of adverse events and rates of preventable adverse events in acute care hospitals
372 Citations2004Philippe Michel, Jean Luc Quenon +2 more
The prospective method of data collection may be more appropriate for epidemiological studies that aim to convince clinical teams that their errors contribute significantly to adverse events, to study organisational and human factors, and to assess the impact of risk reduction programmes.
Journal of General Internal MedicineWhat can hospitalized patients tell us about adverse events? Learning from patient-reported incidents
332 Citations2005Saul N. Weingart, Odelya E. Pagovich +6 more
Engaging hospital inpatients as partners in identifying medical errors and injuries is a potentially promising approach for enhancing patient safety.
BMJ Quality & SafetyPatient safety research: an overview of the global evidence
306 Citations2010Ashish K. Jha, Nittita Prasopa-Plaizier +2 more
An overview of the world's literature on patient safety research was commissioned, finding considerable gaps in knowledge about the structural and process factors that underlie unsafe care globally make solutions difficult to identify, especially in resource-poor settings.
American Journal of Health-System PharmacySeverity-Indexed, Incident Report-Based Medication Error-Reporting Program
262 Citations1991Steven C. Hartwig, Sharon D. Denger +1 more
The use of a severity-indexed medication error-reporting program based on incident reports and managed through a relational database has revealed problems that are then addressed by other quality assurance measures.
IHI Global Trigger Tool for Measuring Adverse Events (Second Edition)
240 Citations2015Marianne Tinnå
BMJ Quality & SafetyPrevalence of adverse events in the hospitals of five Latin American countries: results of the ‘Iberoamerican study of adverse events’ (IBEAS)
235 Citations2011Jesús María Aranaz Andrés, C Aibar-Remón +11 more
The high rate of prevalent AE found suggests that PS may represent an important public-health issue in the participating hospitals, and may already be useful to inspire new PS-improvement policies in those settings.
Canadian Medical Association JournalOttawa Hospital Patient Safety Study: incidence and timing of adverse events in patients admitted to a Canadian teaching hospital
226 Citations2004Alan J. Forster
Characteristics of adverse events affecting patients admitted to a Canadian teaching hospital, paying particular attention to timing, were evaluated, finding that most occurred before the hospitalization, and most were avoidable.
Journal of the American Statistical AssociationA Note on the Equivalence of Two Test Criteria for Hypotheses in Categorical Data
225 Citations1966V. P. Bhapkar
Patient Safety in SurgeryThe incidence, root-causes, and outcomes of adverse events in surgical units: implication for potential prevention strategies
205 Citations2011Marieke Zegers, Martine C. de Bruijne +5 more
Surgical AEs occur more often than other types of AEs, are more often preventable and their consequences are more severe, therefore, surgical AEs have a major impact on the burden of A Es during hospitalizations.
Journal of the American Statistical AssociationOn Estimating Precision of Measuring Instruments and Product Variability
197 Citations1948Frank E. Grubbs
Pharmacoepidemiology and Drug SafetyFrequency of adverse drug reactions in hospitalized patients: a systematic review and meta‐analysis
179 Citations2012Ana Miguel, Luís Filipe Azevedo +2 more
To perform a comprehensive systematic review of prospective studies about frequency of adverse drug reactions occurring during hospitalization (ADRIn), including a thorough study quality assessment, meta‐analysis and heterogeneity evaluation.
Adverse Events in New Zealand Public Hospitals: Principal Findings from a National Survey
150 Citations2003Peter Davis, Roy Lay‐Yee +5 more
The study provides the base parameters necessary to inform the understanding of patient safety and the quality of care in New Zealand public hospitals and points to the importance of similar research on the sources and characteristics of adverse events outside public hospitals.
TechnometricsErrors of Measurement, Precision, Accuracy and the Statistical Comparison of Measuring Instruments
136 Citations1973Frank E. Grubbs
Statistica NeerlandicaA review of statistical methods in the analysis of data arising from observer reliability studies (Part I)*
135 Citations1975J. Richard Landis, Gary G. Koch
International Journal for Quality in Health CareImpact and preventability of adverse events in Spanish public hospitals: results of the Spanish National Study of Adverse Events (ENEAS)
135 Citations2009Jesús María Aranaz Andrés, C. Aibar-Remon +6 more
In Spanish hospitals, AEs associated with health care cause distress, disability, death, lengthen hospital stay and cause increased consumption of health-care resources, and a relatively high percentage of AEs in Spain may be preventable with improvements in medical care.
BMJ Quality & SafetyInternational recommendations for national patient safety incident reporting systems: an expert Delphi consensus-building process
133 Citations2016Ann-Marie Howell, Elaine M. Burns +4 more
To establish international, expert consensus on the purpose of PSRS regarding monitoring and learning from incidents and developing recommendations for their future role, a survey-based questionnaire was developed and subsequently completed by a larger expert panel.
Computer Programs in BiomedicineA computer program for the generalized chi-square analysis of categorical data using weighted least squares (GENCAT)
123 Citations1976J. Richard Landis, William M. Stanish +2 more
GENCAT is a computer program which implements an extremely general methodology for the analysis of multivariate categorical data which produces minimum modified chi-square statistics, obtained by partitioning the sums of squares as in ANOVA.
BMC Health Services ResearchWhat is preventable harm in healthcare? A systematic review of definitions
117 Citations2012Mohammed Nabhan, Tarig Elraiyah +11 more
There is limited empirical evidence of the validity and reliability of the available definitions of preventable harm, such that no single one is supported by high quality evidence.
Journal of the American Medical Informatics AssociationA Risk Analysis Method to Evaluate the Impact of a Computerized Provider Order Entry System on Patient Safety
113 Citations2008Pascal Bonnabry, Christelle Despont-Gros +9 more
The use of risk analysis helps to quantitatively evaluate the relationship between a system and patient safety and to build a strategy for continuous quality improvement, by selecting the most appropriate improvements to the system.
Journal of the American Geriatrics SocietyMeta‐analysis of Interventions to Reduce Adverse Drug Reactions in Older Adults
97 Citations2017Shelly L. Gray, Laura A. Hart +4 more
To examine the effect of interventions to optimize medication use on adverse drug reactions (ADRs) in older adults, a large number of patients were randomly assigned to receive either a dummy pill or a prescription drug.
BMJ OpenCharacterisations of adverse events detected in a university hospital: a 4-year study using the Global Trigger Tool method
86 Citations2014Hans Rutberg, Madeleine Borgstedt Risberg +4 more
The level, preventability and categories of adverse events (AEs) identified by medical record review using the Global Trigger Tool (GTT) and to compare voluntary AE reporting with medical record reviewing were described.
Research Synthesis MethodsA simple method to estimate prediction intervals and predictive distributions: Summarizing meta‐analyses beyond means and confidence intervals
82 Citations2019Chia‐Chun Wang, Wen‐Chung Lee
A simple method with a ready‐to‐use spreadsheet file to estimate prediction intervals and predictive distributions nonparametrically and the advantage and real‐world significance of this approach is illustrated with a meta‐analysis evaluating the protective effect of vaccination against tuberculosis.
BiometricsAn Analysis for Compounded Functions of Categorical Data
71 Citations1973Ronald N. Forthofer, Gary G. Koch
In each of the previously mentioned papers, primary emphasis was given to the formulation of models and the problems of analysis under various conditions of "no interaction" (see Roy and Kastenbaum [1956] or Bhapkar).
Health Services ResearchCharacterization of Adverse Events Detected in a Large Health Care Delivery System Using an Enhanced Global Trigger Tool over a Five‐Year Interval
70 Citations2014Donald A. Kennerly, Rustam Kudyakov +5 more
AEs are common and potentially amenable to prevention, and GTT-identified AEs are seldom caught by commonly used AE-detection systems.
Patient Safety in SurgeryIs detection of adverse events affected by record review methodology? an evaluation of the “Harvard Medical Practice Study” method and the “Global Trigger Tool”
68 Citations2013Maria Unbeck, Kristina Schildmeijer +5 more
Differences in review methodology, perception of less severe adverse events and context knowledge may explain the observed difference between two expert review teams in the detection of adverse events.
BiometricsThe Analysis of Categorical Data from Mixed Models
68 Citations1971Gary G. Koch, D W Reinfurt
Revista Médica Clínica Las CondesSEGURIDAD DEL PACIENTE Y CULTURA DE SEGURIDAD
64 Citations2017Cristián Rocco, Alejandro Garrido
International Journal for Quality in Health CareWhat makes hospitalized patients more vulnerable and increases their risk of experiencing an adverse event?
59 Citations2011Jesús María Aranaz Andrés, R. Limon +5 more
The true risk resides in the number of exposures to potentially iatrogenic actions, rather than being intrinsic to age or the presence of certain comorbidities.
American Journal of EpidemiologySTUDIES ON MULTIPLE SCLEROSIS IN WINNIPEG, MANITOBA, AND NEW ORLEANS, LOUISIANA12
59 Citations1953Knut Westlund, Leonard T. Kurland
Jornal de PediatriaErro médico em pacientes hospitalizados
58 Citations2002Manoel de Carvalho, Alan Araújo Vieira
Recente estudo revela que apos um plantao de 24 horas, sem dormir, o desempenho psicomotor de um profissional de saude e semelhante ao de um individuo legalmente bebado contribuem para a ocorrencia of erros.
International Journal for Quality in Health CareVariation in the rates of adverse events between hospitals and hospital departments
56 Citations2011Marieke Zegers, Martine C. de Bruijne +4 more
Monitor and comparing the performance of hospitals should not be limited to the hospital level, but should be extended to the individual department since there can be significant differences in the rates of preventable AEs between different departments within the same hospital.
TechnometricsA General Approach to the Estimation of Variance Components
56 Citations1967Gary G. Koch
RadiographicsRoot Cause Analysis: Learning from Adverse Safety Events
54 Citations2015Olga R. Brook, Jonathan B. Kruskal +2 more
The Joint Commission now requires that a credible RCA be performed within 45 days for all sentinel or major adverse events, emphasizing the need for all radiologists to understand the processes with which an effective RCA can be performed.
Revista de Salud PúblicaIncidencia y Evitabilidad de Eventos Adversos en Pacientes Hospitalizados en tres Instituciones Hospitalarias en Colombia, 2006
53 Citations2008Hernando Gaitán‐Duarte, Javier Eslava‐Schmalbach +4 more
An important incidence of AE was revealed in three Colombian hospitals, these being mainly preventable, and ongoing monitoring as a part of risk management systems could reduce costs and AE-associated morbidity and mortality.
BiometricsOn the Hypotheses of 'No Interaction' in Contingency Tables
47 Citations1968V. P. Bhapkar, Gary G. Koch
TechnometricsSome Further Remarks Concerning “A General Approach to the Estimation of Variance Components”
46 Citations1968Gary G. Koch
BMJ OpenIncidence of adverse events in Sweden during 2013–2016: a cohort study describing the implementation of a national trigger tool
45 Citations2018Lena Nilsson, Madeleine Borgstedt-Risberg +4 more
The rate of AEs in Swedish somatic hospitals has decreased from 2013 to 2016 and it was found that the economic burden of preventable AEs is high.
TechnometricsHypotheses Of ‘No Interaction’ In Multi-dimensional Contingency Tables
45 Citations1968V. P. Bhapker, Gary G. Koch
The statistical analysis of multi-dimensional contingency tables is discussed from the point of view of the associated underlying model and different formulations of hypotheses of ‘no interaction’ are considered.
BiometricsOn the Analysis of Contingency Tables with a Quantitative Response
41 Citations1968V. P. Bhapkar
BMJ OpenExploring similarities and differences in hospital adverse event rates between Norway and Sweden using Global Trigger Tool
37 Citations2017Ellen Catharina Tveter Deilkås, Madeleine Borgstedt Risberg +5 more
The differences between the countries in the rates of several types of AEs provide new incentives for Norwegian and Swedish governing bodies to address patient safety issues.
Patient Safety in SurgeryIncidence of adverse events in an integrated US healthcare system: a retrospective observational study of 82,784 surgical hospitalizations
31 Citations2014Muhammad Fazal Zeeshan, Allard E. Dembe +2 more
Detecting variations in AEs among surgical categories can be useful for surgeons and for hospital quality assurance personnel, and determining AE severity and age distributions within surgical categories will enable officials to better identify specific patient safety needs and develop appropriately targeted interventions for improvement.
TechnometricsThe Measuring Process
31 Citations1959John Mandel
A theoretical framework is proposed for the mathematical expression of the sources of variation in measuring methods and a suitable method of statistical analysis is described.
Asian BiomedicineIdentifying adverse events in hospitalized patients using global trigger tool in Thailand.
27 Citations2009Santawat Asavaroengchai, Jiruth Sriratanaban +2 more
Adverse events using the GTT detected more events than previous reports, most events had low severity and were preventable, and assessment of validity is needed before using GTT in medical service.
Patient Safety in SurgeryQuality review of an adverse incident reporting system and root cause analysis of serious adverse surgical incidents in a teaching hospital of Scotland
24 Citations2012Maziar Khorsandi, Christos Skouras +2 more
The results of this study demonstrate the need for improvement in the quality of incident reporting and sufficient support and resources are required for the implementation of RCA recommendations in clinical practice.
Journal of the American Statistical AssociationAssessing the Accuracy of Multivariate Observations
20 Citations1966Joseph L. Fleiss
Risk AnalysisA Hybrid Methodology for Modeling Risk of Adverse Events in Complex Health‐Care Settings
17 Citations2017Reza Kazemi, Ali Mosleh +1 more
A generic methodology for evaluating dynamic changes in AE risk in acute care hospitals as a function of organizational and nonorganizational factors is developed, using a combination of modeling formalisms.
Revista de Calidad AsistencialEstudio de incidencia de eventos adversos en una clínica privada en Chile
16 Citations2014M.L. Lancis-Sepúlveda, C. Asenjo-Araya
This is the first study conducted in a private hospital in this country that allowed us to identify the incidence, overall rate, and the rate by departments of adverse events, their severity, preventability and causal factors.
BiomédicaCostos directos e impacto sobre la morbimortalidad hospitalaria de eventos adversos prevenibles a medicamentos en una institución de tercer nivel de Bogotá
13 Citations2011José Fernando Pinzón, Carlos Maldonado +2 more
Determinar el porcentaje de eventos adversos prevenibles a medicamentos, reportados en una institucion de tercer nivel, para establecer su impacto sobre the morbimortalidad and el economico desde the optica del pagador y mostrando the relevancia of usar un metodo that permita identificarlos y evitar o disminuir su presentacion.
Colombian Journal of AnesthesiologyReportable hospital events
12 Citations2019Kelly Estrada-Orozco, Hernando Gaitán‐Duarte +2 more
The incidence, preventability, and severity of REUE are similar to those reported in the literature, and the analysis of the contributing factors shows areas that are susceptible to intervention and can be considered as opportunities for improvement.
BiometricsReliability of Measurements for Studies of Cerebrov Vascular Atherosclerosis
12 Citations1972Ruth B. Loewenson, Jacob E. Bearman +1 more
The development of the methodology for assessing reliability of data is presented and within and between coder variability is estimated.
Clinical TrialsThe automation of clinical trial serious adverse event reporting workflow
10 Citations2009Jack London, Karl J. Smalley +2 more
The automated system described facilitated analysis of individual investigator reporting performance, as well as the aggregation and analysis of the nature of reported adverse events.
International Journal of Healthcare ManagementQuality in private health companies in Peru: The relation of QMS & ISO 9000 principles on TQM factor
10 Citations2019Jorge Benzaquen, Marlene Carlos +3 more
Leadership in health servicesError proofing healthcare: an analysis of low cost, easy to implement and effective solutions
8 Citations2013Jamison V. Kovach, Lee Revere +1 more
This study aims to provide healthcare managers with a meaningful synthesis of state of the art knowledge on error proofing strategies to provide a foundation for understanding medical error prevention, to support the strategic deployment of errorProofing strategies, and facilitate the development and implementation of new error Proofing strategies.
Revista de Calidad AsistencialPlan de seguridad clínica del Complejo Hospitalario Universitario de Albacete (CHUA). Diagnóstico inicial: estudio de eventos adversos
6 Citations2012Francisco Javier Lucas-Imbernón, Jesús María Aranaz Andrés +4 more
The study of AEs in the CHUA represents an improvement within the hospital quality program andidences of patients with AEs and those related to health care fall within the range of those found in the studies where the objective was quality improvement.
Training of workers as an important safety measure for risk management
4 Citations2016Maja Ljubic, Bozidar Rakovic +2 more
Cirugía y Cirujanos (English Edition)Costs of serious adverse events in a community teaching hospital, in Mexico
3 Citations2015Luis Meave Gutiérrez-Mendoza, Abraham Torres‐Montes +3 more
Serious adverse events in a Teaching Hospital setting need to be analysed to learn and deploy interventions to prevent and improve patient safety.
…