EFL Thai Students’ Rating Performances in Self- and Peer-Assessments of Writing: A Many-Facets Rasch Analysis

Main Article Content

Apichat Khamboonruang

Abstract

Despite the value of self- and peer-assessments in supporting learning, research reports mixed findings about self- and peer-assessments of writing, raising intriguing queries about the validity and utility of student-led assessment in a writing classroom. Applying a many-facets Rasch measurement approach, this quantitative study investigated 53 EFL Thai undergraduates’ rating performances in self- and peer-assessments of writing during an ongoing classroom in a university setting. Main findings revealed that self-assessors tended to rate more leniently and heterogeneously than peer-assessors whose severity levels were far more homogeneous and closer to the level of teacher severity. Across self- and peer-ratings, students could observably distinguish writing performance quality, thereby assigning varying scores across rubric criteria, and lower-ability students were more variable in their levels of severity than those of high ability. The findings highlight the main and interaction effects of student-led assessment type and writing ability level on students’ rating performances. That is, students’ rating performances vary according to different types of student-led assessment and different levels of ability. More specifically, assessment type exercises a greater bearing on lower-ability raters’ rating variability, while ability level exerts a stronger influence on self-raters’ rating variability. The findings provide implications for student-led assessment design and research in the classroom context.

Downloads

Download data is not yet available.

Article Details

How to Cite
Khamboonruang, A. (2023). EFL Thai Students’ Rating Performances in Self- and Peer-Assessments of Writing: A Many-Facets Rasch Analysis. LEARN Journal: Language Education and Acquisition Research Network, 16(1), 221–245. Retrieved from https://so04.tci-thaijo.org/index.php/LEARN/article/view/263440
Section
Research Articles
Author Biography

Apichat Khamboonruang, Department of Western Languages and Linguistics, Faculty of Humanities and Social Sciences, Mahasarakham University, Thailand

A lecturer of English in the Department of Western Languages and Linguistics at Mahasarakham University. He holds an MA in English as an International Language from Chulalongkorn University and a PhD in Applied Linguistics from the University of Melbourne. His research interests include rater-mediated language assessment and language test development and validation. (ORCID iD: 0000-0002-7182-3501)

References

Al-Mahrooqi, R., & Denman, C. (2018). Alternative assessment. In J. L. Liontas & M. DelliCarpini (Eds.), The TESOL encyclopedia of English language teaching (pp. 1–6). Wiley-Blackwell. https://doi.org/10.1002/9781118784235.eelt0325

Andrade, H. L. (2019). A critical review of research on student self-assessment. Frontiers in Education, 4(87), 1–13. https://doi.org/10.3389/feduc.2019.00087

Andrade, H. L., & Heritage, M. (2018). Using formative assessment to enhance learning, achievement, and academic self-regulation. Routledge.

Barkaoui, K. (2014). Multifaceted Rasch analysis for test evaluation. In A. J. Kunnan (Ed.), The companion to language assessment (pp. 1–22). John Wiley & Sons, Inc. doi: 10.1002/9781118411360.wbcla070

Brown, G. T. L., & Harris, L. R. (2013). Student self-assessment. In J. H. McMillan (Ed.), Sage handbook of research on classroom assessment (pp. 367–393). SAGE Publications, Inc.

Cheng, W., & Warren, M. (2005). Peer assessment of language proficiency. Language Testing, 22(1), 93–121. https://doi.org/10.1191/0265532205lt298oa

Christison, M. (2018). Student involvement in assessment. In J. L. Liontas & M. DelliCarpini (Eds.), The TESOL encyclopedia of English language teaching (pp. 1–7). Wiley-Blackwell. https://doi.org/10.1002/9781118784235.eelt0357

Eckes, T. (2015). Introduction to many-facet Rasch measurement: Analyzing and evaluating rater-mediated assessments (2nd ed.). Peter Language.

Edwards, J. G. H. (2013). Peer assessment in the classroom. In A. J. Kunnan (Ed.), The companion to language assessment (pp. 730–750). Wiley-Blackwell. https://doi.org/10.1002/9781118411360.wbcla002

Erman Aslanoglu, A., Karakaya, I., & Sata, M. (2020). Evaluation of university students’ rating behaviors in self and peer-rating process via many facet Rasch model. Eurasian Journal of Educational Research, 89, 25–46.

Esfandiari, R., & Myford, C. M. (2013). Severity differences among self-assessors, peer-assessors, and teacher assessors rating EFL essays. Assessing Writing, 18(2), 111–131. https://doi.org/10.1016/j.asw.2012.12.002

Fan, J., & Bond, T. (2019). Applying Rasch measurement in language assessment: Unidimensionality and local independence. In V. Aryadoust & M. Raquel (Eds.), Quantitative data analysis for language assessment volume I: Fundamental techniques (pp. 83–102). Routledge.

Han, C., & Riazi, M. (2018). The accuracy of student self-assessments of English-Chinese bidirectional interpretation: A longitudinal quantitative study. Assessment & Evaluation in Higher Education, 43(3), 386–398. https://doi.org/10.1080/02602938.2017.1353062

Harris, L. R., & Brown, G. T. L. (2018). Using self-assessment to improve student learning. Routledge.

Hung, Y.-j., Samuelson, B. L., & Chen, S.-c. (2016). Relationships between peer- and self-assessment and teacher assessment of young EFL learners’ oral presentations. In M. Nikolov (Ed.), Assessing young learners of English: Global and local perspectives (pp. 317–338). Springer, Cham. https://doi.org/10.1007/978-3-319-22422-0

Kim, S. C., & Wilson, M. (2009). A comparative analysis of the ratings in performance assessment using Generalizability theory and the many-facet Rasch model. Journal of Applied Measurement, 10(4), 408–423.

Khamboonruang, A. (2020). Development and validation of a diagnostic rating scale for formative assessment in a Thai EFL university writing classroom: A mixed methods study [Doctoral dissertation, The University of Melbourne]. Minerva Access. http://hdl.handle.net/11343/252672

Li, M., & Zhang, X. (2021). A meta-analysis of self-assessment and language performance in language testing and assessment. Language Testing, 38(2), 189–218. https://doi.org/10.1177/0265532220932481

Linacre, J. M. (1989). Many-facet Rasch measurement. MESA.

Linacre, J. M. (2021). Facets computer program for many-facet Rasch measurement, version 3.83.6. Winsteps.com

Matsuno, S. (2009). Self-, peer-, and teacher-assessments in Japanese university EFL writing classrooms. Language Testing, 26(1), 075–100. https://doi.org/10.1177/0265532208097337

McNamara, T., Knoch, U., & Fan, J. (2019). Fairness, justice, and language assessment: The role of measurement. Oxford University Press.

Midraj, J. (2018). Self-assessment. In J. L. Liontas & M. DelliCarpini (Eds.), The TESOL encyclopedia of English language teaching (pp. 1–7). Wiley-Blackwell. https://doi.org/10.1002/9781118784235.eelt0331

Myford, C. M., & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet Rasch measurement: Part I. Journal of Applied Measurement, 4(4), 386–422.

Oscarson, M. (2013). Self-assessment in the classroom. In A. J. Kunnan (Ed.), The companion to language assessment (pp. 712–729). Wiley-Blackwell. https://doi.org/10.1002/9781118411360.wbcla046

Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Danish Institute for Educational Research.

Ross, S. (1998). Self-assessment in second language testing: a meta-analysis and analysis of experiential factors. Language Testing, 15(1), 1–20. https://doi.org/10.1177/026553229801500101

Saito, H. (2008). EFL classroom peer assessment: Training effects on rating and commenting. Language Testing, 25(4), 553–581. https://doi.org/10.1177/0265532208094276

Saito, H., & Fujita, T. (2004). Characteristics and user acceptance of peer-rating in EFL writing classrooms. Language Teaching Research, 8(1), 31–54. https://doi.org/10.1191/1362168804lr133oa

Sun, Y., & Doman, E. (2018). Peer assessment. In J. L. Liontas & M. DelliCarpini (Eds.), The TESOL encyclopedia of English language teaching (pp. 1–7). Wiley-Blackwell. https://doi.org/10.1002/9781118784235.eelt0330

Topping, K. J. (2013). Peers as a source of formative and summative assessment. In J. H. McMillan (Ed.), SAGE handbook of research on classroom assessment (pp. 395–412). SAGE Publications, Inc.