A new permutation-based method for assessing agreement between two observers making replicated binary readings.

A new coefficient for assessing agreement between two observers using a permutation-based method is introduced in this article. When observations are binary, this coefficient compares the observed disagreement (probability of discordance) between the observers with its expected value under the hypothesis of individual equivalence. This hypothesis states that for each subject, the conditional distributions of the readings of the two observers are identical, and therefore from a statistical viewpoint it does not matter which observer makes the reading on this subject. Let K and L denote the numbers of replicated observations that are available from observers X and Y, respectively, on a given subject. Then the expected disagreement under individual equivalence for a subject is based on the (K + L) choosing K possible assignments of X's and Y's to the K + L observations made on this subject. Simple methods for the estimation of the new coefficient and its standard error are derived. The new coefficient is compared with kappa and the coefficient of individual agreement, which is based on comparing the inter and intra observer disagreements. Simulation studies confirm the validity of the estimated coefficient and its standard error. Data from a study involving the evaluation of mammograms by 10 radiologists are used to illustrate this new approach to the evaluation of observer agreement.