A Learner Language Based Validation of the CEFR CV Grid of Phonological Control

Supervised by Claudia Harsch at the University of Bremen

The aim of this study is to empirically validate the scale of CEFR CV grid of phonological control using real-life learner language. The study addresses the critique that arises due to the calibration process of the scales and levels, that “were derived from expert estimation and quantitative analysis” (Harsch & Malone, 2020, p. 37) rather than real-life learner language. This missing empirical underpinning with learner language leads to the dilemma, that the descriptors and scales neither describe proficiency development, nor do they present a theory-based description of SLA (Harsch & Malone, 2020).
The gap between empirical groundings, SL pronunciation theory and scale content was particularly evident in the original scale of phonological control of the CEFR (Council of Europe [CoE], 2001), which was harshly criticized by experts for a variety of reasons (cf. e.g. Galaczi et al., 2012; Harding, 2017a; Hirschfeld & Reinke, 2018; Horner, 2013). Even the authors (Council of Europe [CoE], 2020, p. 133) agreed, that “the phonology scale was the least successful of those calibrated in the original research behind the descriptors published in 2001” and was therefore explicitly mentioned as one of the main foci to be revised in the CV, being the only scale that was nearly completely renewed (North & Piccardo, 2019). While the revision (Piccardo, 2016) led to an overall improvement (cf. e.g. Chudoba et al., 2020; Dahmen, 2019) the critique regarding the missing empirical underpinning was not addressed. It remains unclear to which extent the descriptors calibrated during the process are suitable to adequately describe learners’ speech samples (Harsch, 2014), a gap that can only be closed by underpinning the CV grid of phonological control with real-life learner language. Only by examining the capability of the grid to address real-life phonological criteria of learners’ speech the applicability of the grid can be assured, which is a necessary condition to add real-world meaning to the scale (Knoch et al., 2021).
The underpinning follows the argument-based validation framework of (Chapelle et al., 2009), an adapted version of Kane’s (2006, 2013) seminal Interpretation/Use Argument (IUA). By “specifying an interpretive argument that lays out the inferences and assumptions leading from the test scores to the conclusions and decisions”(Kane, 2006, p. 135), the interpretive argument presents a chain of four inferential links that, in order to claim validity, have to be intact and to be supported by evidence (Harding, 2017b).
The chosen validation process consists of a multi-level, mixed-methods approach in six steps:

1. Identifying key criteria of SLA theory and the CEFR CV grid in two separate Content Analyses.
2. Analysing speech samples using criteria the SLA theory criteria.
3. Matching the same speech samples to the three phonological scales in the CV using the item-descriptor matching method.
4. Many Facet Rasch Modelling for revising and adjusting obtained data.
5. Conducting an ordinal logistic regression to determine predictors for level discrimination
6. Calculating Spearman correlations to examine the relationship between detail and overall control scales of the grid.

References

• Chapelle, C. A., Enright, M. K., & Jamieson, J. M. (Eds.). (2009). ESL & applied linguistics professional series.
  Building a validity argument for the Test of English as a Foreign Language TM (reprinted.). Routledge.
• Piccardo, E. (2020). Der GERS Neu (2018) und der Ausspracheunterricht DaF/DaZ. ÖDaF-Mitteilungen, 36(1),
  9–21. https://doi.org/10.14220/odaf.2020.36.1.9
• Council of Europe. (2001). Common European Framework of References for Languages: Learning, Teaching and Assessment. Council of Europe.
• (2020). Common European Framework of Reference for Languages: Companion Volume.
  Council of Europe. https://ebookcentral.proquest.com/lib/kxp/detail.action?docID=6692342
• Dahmen, S. (2019). Die Skalen zu Phonetik / Phonologie im GeR und seinem Begleitband. Deutsch Als
  Fremdsprache. Advance online publication. https://doi.org/10.37307/j.2198-2430.2019.04.02
• Galaczi, E., Post, B., Li, A., & Graham, C. (2012). Measuring L2 English Phonological Proficiency: Implications
  for Language Assessment. In J. Angouri, M. Daller, & J. Treffers-Daller (Eds.), The Impact of Applied Linguistics: Proceedings of the 44th Annual Meeting of the British Association for Applied Linguistics (pp. 67–72). Scitsiugnil Press.
• Harding, L. (2017a). Second Language Pronunciation Assessment: Interdisciplinary Perspectives. In T. Isaacs &
  P. Trofimovich (Eds.), Second Language Acquisition. Second Language Pronunciation Assessment: Interdisciplinary Perspectives (pp. 12–34). Multilingual Matters / Channel View Publications. https://doi.org/10.21832/9781783096855-004
• Harding, L. (2017b). Validity In Pronunciation Assessment. In O. Kang & A. Ginther (Eds.), Assessment in Second Language Pronunciation (pp. 30–48). Routledge.
• Harsch, C. (2014). General Language Proficiency Revisited: Current and Future Issues. Language Assessment
  Quarterly, 11(2), 152–169. https://doi.org/10.1080/15434303.2014.902059
• Harsch, C., & Malone, M. (2020). Language Proficiency Frameworks and Scales. In P. Winke & T. Brunfaut
  (Eds.), The Routledge Handbook of Second Language Acquisition and Language Testing (pp. 33–44).
  Routledge.
• Hirschfeld, U., & Reinke, K. (2018). Phonetik im Fach Deutsch als Fremd- und Zweitsprache: Unter
  Berücksichtigung des Verhältnisses von Orthografie und Phonetik (2., neu bearbeitete Auflage). ESV basics:
  Vol. 1. Erich Schmidt Verlag. https://esv-elibrary.de/book/99.160005/9783503177578
• Horner, D. (2013). Towards a new phonological control grid. In E. Galaczi & C. J. Weir (Eds.), Studies in language testing: Vol. 36. Exploring language frameworks: Proceedings of the ALTE Kraków Conference, July 2011. Cambridge University Press.
• Kane, M. (2006). Content-Related Validity Evidence in Test Development. In S. M. Downing & T. M. Haladyna
  (Eds.), Handbook of test development (pp. 131–153). L. Erlbaum.
• Kane, M. (2013). Validating the Interpretations and Uses of Test Scores. Journal of Educational Measurement,
  50(1), 1–73. https://doi.org/10.1111/jedm.12000
• Knoch, U., Deygers, B., & Khamboonruang, A. (2021). Revisiting rating scale development for rater-mediated
  language performance assessments: Modelling construct and contextual choices made by scale developers.
  Language Testing, 38(4), 602–626. https://doi.org/10.1177/0265532221994052
• North, B., & Piccardo, E. (2019). Developing new CEFR descriptor scales and expanding the existing ones:
  constructs, approaches and methodologies. Zeitschrift Für Fremdsprachenforschung, 30(2), 143–161.
• Piccardo, E. (2016). Phonological Scale Revision: Process Report