Rater uncertainty and its effects on the quality of the proficiency rating in the Finnish speaking test assessment
Keywords:
language assessment, language test, speaking, uncertaintyAbstract
The study examines the uncertainty of trained raters in speaking test assessment and its effect on the
quality of the proficiency ratings of adult learners of Finnish. The data consists of 12,059 task-specific
speaking proficiency ratings and notes taken by the raters during the assessment, as well as questionnaire data. Qualitative content analysis as well as statistical methods were used to answer the research
questions. In the studied group, certainty dominated the assessment. All the speaking raters remained,
however, at least occasionally uncertain of the proficiency ratings they gave. When uncertainty did
occur, it typically covered only one task-specific rating in the set of tasks of a given speaker. The raters
named several reasons for their uncertainty. The level of uncertainty varied at different stages of the
assessment, with uncertainty being highest at the beginning of the assessment and increasing also as
the second rater’s previously assessed performance entered the double-assessment. Uncertainty turned
out to be largely individual, as in double-assessed performances, the raters’ uncertainty was mainly
focused on the performances of different speakers. The length of the rating experience did not explain
the variation in rater (un)certainty. Perceived uncertainty did not affect the quality of the assessment,
i.e. its consistency or severity/leniency. Yet, it is important to take care of raters and assessment quality
factors and so that uncertainty does not burden raters and so that the validity and reliability of assessment is maintained.
