Agreement between observers when the categories are not specified in advance

Abstract

To test the agreement between two observers who categorize a number of objects when the categories have not been specified in advance, Brennan & Light (1974) developed a statistic A ′ and suggested a normal approximation for its distribution. In this paper it is shown that this approximation is inadequate particularly when one, or both, of the observers place a fairly equal number of objects in all of their categories. A chi‐squared approximation to the distribution of A ′ is developed and is shown to work well in a variety of situations. The relative powers of A ′ and the ordinary X 2 test for association are dependent on the type of ‘agreement between the observers’ that is assumed. However a simulation for a fairly general type of agreement indicates that the X 2 test is more powerful. As the X 2 test is also much easier to apply, it would seem preferable in most situations.