Exploring the feasibility of using the Item Descriptor (ID) method to set cut scores on a B2 level test
Supervised by Claudia Harsch at the University of Bremen
The Item Descriptor (ID) matching method is a relatively new method that has not yet been widely used in the CEFR context (see Harsch & Hartig, 2015, for one example). In the ID method panellists focus on identifying the skills and subskills each item aims at measuring and on determining the amount of knowledge test takers need to have to answer test items correctly (Ferrara, Perie, & Johnson, 2008; Zieky, Perie, & Livingston, 2012). The Item Descriptor (ID) matching method appears to be free from some of the abstract notions affecting the other standard setting methods (i.e. the notion of the “borderline candidate” and/or the notion of response probability (RP)) and as such is rather appealing to educators. Despite its appeal, this method has not been well researched in the field of language testing and in particular in standard setting workshops aiming to establish cut scores on tests that aim at operationalising one CEFR level.
For the PhD study, the qualitative and quantitative methods employed to collect data followed the embedded mixed methods design described by Creswell (2014). In this design, qualitative and quantitative data were collected in a convergent manner, i.e. data collected during the standard setting workshop, and in a sequential manner, i.e. after the main data collection project ended (Clark & Ivankova, 2016; Creswell, 2014).
The study seeks to contribute to the limited research in the ID method in three ways: (1) theoretical; (2) methodological and (3) practical. The first contribution of the study is to explore the theoretical possibility of setting cut scores on a CEFR B2 level instrument and to provide a quantitative framework for calculating cut scores through the ID method. The next contribution of the study is to provide a methodological framework for conducting stimulated recall interviews after a standard setting workshop has been completed, where there often is no possibility to record judges’ thoughts or capture their cognitive processes. The final contribution of the study is to provide practical guidance on conducting a virtual standard setting workshop using the ID matching method.