Screening for amblyopia in preverbal children with photoscreening photographs: IV. Interobserver variability in photograph grading: origin and method of reduction.

BACKGROUND

Earlier detection of childhood vision disorders is a prominent goal of vision screening. The Medical Technology and Innovation (MTI) PhotoScreener addresses this objective. Use of this camera does not require verbal feedback and may be administered early in a child's development. Decreasing the variability in photograph grading results will boost the utility of any photoscreening system. This report aims to understand and to decrease intra- and inter-observer variability in grading photoscreening photographs.

METHODS

We dissected the photograph grading process and quantified the intra- and inter-observer agreement using intraclass plot correlation coefficients. We evaluated the outcome of a two grader verification system vs. adjudicated measurements with Receiver Operator Characteristic (ROC) curves.

PARTICIPANTS

Data on 955 children under 5 years of age, normal except for refractive error, each with complete photoscreening and eye examination data, culled from two previous studies.

MAIN OUTCOME MEASURES: Intra- and inter-observer agreement in measuring bright crescent dimensions and pupillary diameters. Sensitivity and specificity of detection of hyperopia.

RESULTS

Measurements of bright crescents are associated with greater variability than are measurements of pupillary diameters. Recognition and omission of light "rim" measurements from photograph grading will result in superior inter-observer agreement. Photograph independent errors increase variability and may be corrected by remeasurement. A verification system in which the most discrepant 5% of measurements are redone results in ROC curves similar to adjudicated dimension.

CONCLUSIONS

We conclude: 1) two novices grading photographs can do as well as one expert in most cases; 2) the proposed grading methodology has undergone statistical validation and can be used in other areas of ophthalmology and medicine; and 3) inter-observer variability, one of the limitations of photoscreening photograph grading, can be reduced. For 95% of the photographs, two novices achieve similar true positive and true negative values with or without adjudication.