Assisted and unassisted rating of source-text use in the context of integrated academic writing
Supervised by Claudia Harsch at the University of Bremen
Successful academic writing requires a broad range of competences and includes the processing of multiple sources. When writing from sources, students are expected to have efficient paraphrasing skills along with the knowledge of proper source attribution. The assessment of these competencies calls for multi-trait rating with the help of an analytic rating scale where the required skills are operationalized in separate criteria (Knoch & Sitajalabhorn, 2013, p. 305). Some of the assessment criteria in an analytic rating scale are related to the use of a source text. However, proper source-text-use identification and evaluation pose difficulties for raters (Gebril & Plakans, 2014). There is a need to assist raters with an automatic tool that highlights copied passages from a source text. This doctorate project aims to investigate the effects of such assistance on the quality of analytic ratings. The empirical enquiry encompasses two stages: explorative and confirmatory. During the explorative stage, the differences between unassisted and assisted rating will be examined. Here, with the help of think-aloud protocols and interviews I expect to draw conclusions about cognitive challenges for the raters while rating source text use as well as solutions how to address these challenges. The confirmatory stage includes quantitative analysis of the inter-rater reliability based on ratings produced in assisted and unassisted modes. Both stages also include qualitative and quantitative investigation of the impact that assisted and unassisted ratings have on the rating of other criteria in the analytic rating scale. This project has potential implications of facilitating the rating process and improving rating quality in the context of academic English assessment.
In June 2021, the doctoral thesis is awarded an Impulse grant by the Central Research Development Fund (CRDF), University Bremen.