Testing the test: Item response curves and test quality

Abstract:

We present a simple technique for evaluating multiple-choice questions and their answer choices beyond the usual measures of difficulty and the effectiveness of distractors. The technique involves the construction and qualitative consideration of item response curves (IRCs) and is based upon Item Response Theory from the field of education measurement. Item response curves relate the percentage of students who select each possible answer choice to overall ability level. To demonstrate the technique, we apply IRC analysis to three questions from the Force Concept Inventory (FCI). IRC analysis allows us to characterize qualitatively whether these questions are efficient, where efficient is defined in terms of the construction, performance, and discrimination of a question and its answer choices. Such analysis can be useful both in the development of future multiple-choice examination questions and in the development of a better understanding of results from existing diagnostic instruments such as the FCI.