THE VALIDITY AND RELIABILITY OF EVALUATION OF SCHOLARLY PERFORMANCE
Elsevier eBooksPublished 1 January 1988
A. J. Nederhof
Citations50
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
A survey is given of the merits of the contributions of peer review and bibliometric methods to evaluation in science. First, the role of evaluation in the scientific process is outlined, with a focus on the evaluation of research proposals and research products such as journal manuscripts. An analysis is made of the validity and reliability of both peer review and citation and publication measures of research performance. The evidence indicates that in all phases of research the outcomes of peer review are usually more than fifty per cent due to chance. It is much easier to cope with random error in bibliometric evaluation. Validity studies of citation measures have, in majority, dealt with more or less eminent scholars. Recent research has shown that bibliometric indicators are also capable of discriminating among starting scholars. Finally, the use and limitations of bibliometric indicators in the humanities and the social and behavioral sciences is discussed.
Keywords
Decision Sciences
ScienceThe Matthew Effect in Science
6,270 Citations1968Robert Κ. Merton
The psychosocial conditions and mechanisms underlying the Matthew effect are examined and a correlation between the redundancy function of multiple discoveries and the focalizing function of eminent men of science is found—a function which is reinforced by the great value these men place upon finding basic problems and by their self-assurance.
PubMedThe Matthew effect in science. The reward and communication systems of science are considered.
4,872 Citations1968Robert Κ. Merton
Behavioral and Brain SciencesPeer-review practices of psychological journals: The fate of published articles, submitted again
1,080 Citations1982Douglas Peters, Stephen J. Ceci
ScientometricsIs citation analysis a legitimate evaluation tool?
906 Citations1979Eugene Garfield
It is concluded that as the scientific enterprise becomes larger and more complex, and its role in society more critical, it will become more difficult, expensive and necessary to evaluate and identify the largest contributors.
Social Studies of ScienceSome Results on the Function and Quality of Citations
690 Citations1975Michael J. Moravcsik, Poovanalingam Murugesan
ScienceChance and Consensus in Peer Review
624 Citations1981Stephen Cole, Jonathan R. Cole +1 more
An experiment in which 150 proposals submitted to the National Science Foundation were evaluated independently by a new set of reviewers indicates that getting a research grant depends to a significant extent on chance.
Research PolicyThe use of bibliometric data for the measurement of university research performance
482 Citations1985Henk F. Moed, W. J. M. Burger +2 more
Journal of Information ScienceA review of bibliometric and other science indicators and their role in research evaluation
337 Citations1987Jean King
In this state-of-the- art review the various methodologies that have been developed are outlined in terms of their strengths, weaknesses and par ticular applications.
American Sociological ReviewScholarly Consensus and Journal Rejection Rates
236 Citations1988Lowell L. Hargens
Journal of the American Society for Information ScienceEvidence of complex citer motivations
185 Citations1986Terrence A. Brooks
American PsychologistImproving manuscript evaluation procedures.
160 Citations1972Donald D. Bowen, Robert Perloff +1 more
Convention program committees face a similar problem in selecting papers for presentation, although the published complaints usually focus exclusively on frustrations encountered in dealings with the journals.
ScientometricsAssessing the usefulness of bibliometric indicators for the humanities and the social and beha vioural sciences: A comparative study
154 Citations1989A. J. Nederhof, Rolf A. Zwaan +2 more
Although many publications were written in English, only experimental psychology, general linguistics, anthropology, and genrral literature were internationally oriented regarding output media.
ScienceCitation Indexing and Evaluation of Scientific Papers
147 Citations1967James W. Margolis
An increased awareness of the usefulness of citation indexing as a tool for retrieval and evaluation will make this aspect of refereeing more important, and what now passes for minor carelessness or discourtesy could easily come to be regarded as serious malpractice.
Social Studies of ScienceQuantitative Measures of Communication in Science: A Study of the Formal Level
145 Citations1986Michael H. MacRoberts, Barbara R. MacRoberts
This work examined papers to see how much influence appears as references in bibliographies and found that very little does.
ScientometricsSpecialties and disciplines in science and social science: An examination of their structure using citation indexes
134 Citations1979Hamish Small, D. Russell Crane
Certain common structural characteristics of the social science and natural science groups suggest that knowledge is developing in parts of theSocial Sciences Citation Index disciplines in a manner similar to the natural sciences.
Social Studies of SciencePeer Review at the NSF: A Dialectical Policy Analysis
124 Citations1979Ian I. Mitroff, Daryl E. Chubin
The Library QuarterlyThe Use of Citations in Literary Research: A Preliminary Classification of Citation Functions
103 Citations1979Carolyn O. Frost
Differences in research methods between scientific and humanistic disciplines which might have an influence on citation practices are considered and a scheme for the classification of citations in literary research is proposed as a preliminary model.
Journal of Clinical PsychologyThe citation index: Another case of spurious validity
84 Citations1977Frederick C. Thorne
The Library QuarterlyCharacteristics of References in Selected Scholarly English Literary Journals
82 Citations1980Richard Heinzkill
Journal of the American Society for Information ScienceThe adequacy of the science citation index (SCI) as an indicator of international scientific activity
80 Citations1981Mark P. Carpenter, Francis Narin
The SCI appears to be an excellent, internationally balanced data source for the core of the physical and biological sciences, particularly for the English-speaking countries, but there still are significant differences between the SCI and other sources in national coverage of fields with a more dispersed literature, especially in the case of journals from countries with non-Roman alphabets.
Social Studies of ScienceDifferences in Impact of Scientific Publications: Some Indices Derived from a Citation Analysis
75 Citations1976Dennis Dieks, Hans Chang
A mathematical model is introduced which describes citing as a probabilistic process, while taking into account that in general the probability for a paper to be cited decreases with age.
The Library QuarterlyIs Information Retrieval in the Arts and Humanities Inherently Different from That in Science? The Effect That ISI®'S Citation Index for the Arts and Humanities Is Expected to Have on Future Scholarship
68 Citations1980Eugene Garfield
American Sociological ReviewDo Journal Rejection Rates Index Consensus?
66 Citations1988Stephen Cole, Gary Simon +1 more
Hargens argues that the lower rejection rate in natural sciences journals is evidence that these fields have more cognitive consensus, and believes that journal rejection rates are a better indicator of field differences in cognitive consensus than CR (defined formally below), the extent to which independent evaluators of a scientific product like a journal article or a research proposal reach the same conclusion.
ScientometricsPeer review and bibliometric indicators of scientific performance: A comparison of cum laude doctorates with ordinary doctorates in physics
60 Citations1987A. J. Nederhof, Anthony F. J. van Raan
The results offer little evidence for the Matthew effect and the Ortega hypothesis, but support the validity of both peer review outcomes and bibliometric impact assessments of scientific performance.
Social Studies of ScienceAssessing Basic Research: The Case of the Isaac Newton Telescope
58 Citations1983John M. Irvine, Ben R. Martin
Research PolicyCERN: Past performance and future prospects
58 Citations1984Ben R. Martin, John M. Irvine
Research PolicyBetween dirigism and laissez-faire: Effects of implementing the science policy priority for biotechnology in the Netherlands
51 Citations1986Arie Rip, Anton J. Nederhof
ScientometricsA comparative study of bibliometric past performance analysis and peer judgement
51 Citations1985Henk F. Moed, W. J. M. Burger +2 more
The work of Dutch National Survey Committees on Chemistry and on Biology is compared with the authors' bibliometric results for research groups in these disciplines at the University of Leiden, revealing a serious lack of agreement between the two types of past performance analysis.
Journal of the American Society for Information ScienceSome aspects of citation behavior: A pilot study in business administration
49 Citations1983Chandra G. Prabha
Ninety‐six percent of the sources cited had been consulted by the authors, which indicates little evidence of secondhand citation, but criticality is no guarantee that it has been used heavily by the author.
ScientometricsUsing the archive: Derek Price's theory of differences among the sciences
41 Citations1985Susan E. Cozzens
Several bodies of evidence point to the desirability of further application of the basic concepts Derek Price introduced for the purpose he proposed: as diagnostic tools to describe and compare processes of knowledge growth in the sciences.
NatureNational performance in basic research
31 Citations1986Drew Smith, Peter Collins +2 more
A newly published report reviews the techniques available for evaluating national performance in basic research and applies these techniques to solid state physics and genetics.
Proceedings of the Royal Society of London. Series B, Biological sciencesThe Bernal Lecture, 1983 - The collectivization of science
30 Citations1983John Ziman
Journal of Information ScienceAgreement and divergence on referencing practice
28 Citations1981Blaise Cronin
A small experiment is described which illustrates the extent to which referencing habits and expectations can vary, even within a single professional sub-group.
ScientometricsChanges in publication patterns of biotechnologists: An evaluation of the impact of government stimulation programs in six industrial nations
24 Citations1988A. J. Nederhof
The effects of a government stimulation program on the development of Dutch biotechnology have been studied scientometrically in comparison with world-wide averages, and with the effects of programs of five important large Western industrial nations, in the period 1976–1985.
ScientometricsA validation study of bibliometric indicators: The comparative performance of cum laude doctorates in chemistry
22 Citations1989A. J. Nederhof, Anthony F. J. van Raan
The results support the concurrent validity of bibliometric indicators with peer review indicators of quality of the research project and little evidence was found in favor of the Ortega hypothesis and the Matthew effect.
ScientometricsEvaluating research output through life work citation counts
12 Citations1985A. J. Nederhof
A study of a sample of Dutch full professors of sociology shows that simple counting of citations may lead to unreliable results, and longitudinal analysis seems to be called for for fine-tuned assessment of research output.
American Sociological ReviewFurther Evidence on Field Differences in Consensus from the NSF Peer Review Studies
6 Citations1988Lowell L. Hargens