Caveats for using statistical significance tests in research assessments

Concerns are raised about the advantages of using statistical significance tests in research assessments as has recently been suggested in the debate about proper normalization procedures for citation indicators by Opthof and Leydesdorff (2010).

Abstract

This paper raises concerns about the advantages of using statistical\nsignificance tests in research assessments as has recently been suggested in\nthe debate about proper normalization procedures for citation indicators.\nStatistical significance tests are highly controversial and numerous criticisms\nhave been leveled against their use. Based on examples from articles by\nproponents of the use statistical significance tests in research assessments,\nwe address some of the numerous problems with such tests. The issues\nspecifically discussed are the ritual practice of such tests, their dichotomous\napplication in decision making, the difference between statistical and\nsubstantive significance, the implausibility of most null hypotheses, the\ncrucial assumption of randomness, as well as the utility of standard errors and\nconfidence intervals for inferential purposes. We argue that applying\nstatistical significance tests and mechanically adhering to their results is\nhighly problematic and detrimental to critical thinking. We claim that the use\nof such tests do not provide any advantages in relation to citation indicators,\ninterpretations of them, or the decision making processes based upon them. On\nthe contrary their use may be harmful. Like many other critics, we generally\nbelieve that statistical significance tests are over- and misused in the social\nsciences including scientometrics and we encourage a reform on these matters.\n