Effects of Raters’ L1, Assessment Experience, and Teaching Experience on their Assessment of L2 English Speech: A Study Based on the ICNALE Global Rating Archives
Article Sidebar
Main Article Content
Abstract
TESOL practitioners, especially in Asia, tend to believe that reliable assessment of students’ L2 English speech can be done solely by L1 English native speakers with sufficient teaching and assessment experiences. Such a belief, however, may need to be reconsidered from a new perspective of “diversity and inclusivity.” This study used data from the ICNALE Global Rating Archives, a newly compiled assessment dataset, to examine the degree of the effects of raters’ L1, assessment experience, and teaching experience on their assessment of Chinese, Japanese, and Thai learners’ L2 English speech. The quantitative analyses showed that (1) raters’ L1 significantly influenced the assessment scores for all three learner groups, but the difference between native speaker raters and non-native speaker raters was not clear, (2) raters’ assessment experience influenced the scores only for Japanese learners, and (3) raters’ teaching experience did not significantly influence the assessment scores for any of the three learner groups. These findings, which cast doubt on the dependence on native speaker raters alone, suggest the need to involve a greater variety of raters in L2 speech assessment.
Article Details
References
ACTFL (2019). Diversity and inclusion in world language teaching & learning. https://www.actfl.org/advocacy/actfl-position-statements/diversity-and-inclusion-world-language-teaching-learning
Bachman, L. F., & Palmer, A. S. (1996). Language testing in practice: Designing and developing useful language tests. Oxford University Press.
Brown, J. D. (2022). Classical test theory. In G. Fulcher & L. Harding (Eds.), The Routledge handbook of language testing (2nd ed.) (pp. 323–335). Routledge.
Chau, M. H., Lie, A., Jacobs, G. M., & Renandya, W. A. (2022). Introduction: Promoting diversity and inclusion in language education through research and practice in global Englishes and translanguaging. TESL-EJ, 26(3). https://doi.org/10.55593/ej.26103a0
Dalman, M., & Kang, O. (2019). Listener background in L2 speech evaluation. In N. Feza (Ed.), Metacognition in learning (pp. 1-14). Intech Open. DOI: 10.5772/intechopen.89414
Gui, M. (2012). Exploring differences between Chinese and American EFL teachers’ evaluations of speech performance. Language Assessment Quarterly, 9, 186–203.
Hall, C. S., & Hope, A. K. (2016). Tips for testing speaking. TESOL Connections 2016 April, 1–4.
http://newsmanager.commpartners.com/tesolc/issues/2016-04-01/3.html
Han, Q. (2016). Rater cognition in L2 speaking assessment: A review of the literature. Teachers College, Columbia University Working Papers in TESOL & Applied Linguistics, 16(1), 1–24.
https://doi.org/10.7916/salt.v16i1.1261
Holliday, A. R. (2005). The struggle to teach English as an international language. Oxford University Press.
Holliday, A. R. (2006). Native-speakerism. ELT Journal, 60(4), 385–387. https://doi.org/10.1093/elt/ccl030
Honda, Y., & Tsuchidate, S. (2022, November 28). Tokyo includes 1st speaking test for English in entrance exam. The Asahi Shimbun. https://www.asahi.com/ajw/articles/14779198
Hsieh, C. N. (2011). Rater effects in ITA testing: ESL teachers’ versus American undergraduates’ judgments of accentedness, comprehensibility, and oral proficiency. Spaan Fellow Working Papers in Second or Foreign Language Assessment, 9, 47–74.
Huang, L., Kubelec, S., Keng, N., & Hsu, L. (2018). Evaluating CEFR rater performance through the analysis of spoken learner corpora. Language Testing in Asia, 8(1), 1–17.
https://doi.org/http://dx.doi.org/10.1186/s40468-018-0069-0
Isaacs, T., & Thomson, R. I. (2013). Rater experience, rating scale length, and judgements of L2 pronunciation: Revisiting research conventions. Language Assessment Quarterly, 10, 135–159.
Ishikawa, S. (2013). The ICNALE and sophisticated contrastive interlanguage analysis of Asian learners of English. Learner Corpus Studies in Asia and the World, 1, 91–118.
Ishikawa, S. (2019). The ICNALE Spoken Dialogue: A new dataset for the study of Asian learners’ performance in L2 English interviews. English Teaching, 74(4), 153–177.
Ishikawa, S. (2020). Aim of the ICNALE GRA project: Global collaboration to collect ratings of Asian learners’ L2 English essays and speeches from an ELF perspective. Learner Corpus Studies in Asia and the World, 5, 121–144.
Ishikawa, S. (2023). The ICNALE Guide: An introduction to a learner corpus study on Asian learners’ L2 English. Routledge.
Kang, O. (2012). Impact of rater characteristics and prosodic features of speaker accentedness on ratings of international teaching assistants’ oral performance. Language Assessment Quarterly, 9(3), 249–269.
Kang, O., Rubin, R., & Kermad, A. (2019). The effect of training and rater differences on oral proficiency assessment. Language Testing, 36(4), 481–504.
Kasper, G., & Dahl, M. (1991). Research methods in interlanguage pragmatics. Studies in Second Language Acquisition, 12, 215–247.
Lee, H. (2017). The effects of rater’s familiarity with test taker’s L1 in assessing accentedness and comprehensibility of independent speaking tasks. SNU Working Papers in English Linguistics and Language, 15, 93–111.
Mohd Noh, M.F., & Mohd Matore, M.E.E. (2022). Rater severity differences in English language as a second language speaking assessment based on rating experience, training experience, and teaching experience through many-faceted Rasch measurement analysis. Frontiers in Psychology, 13: 941084. https://doi.org/10.3389/fpsyg.2022.941084
Mohd Noh, M.F., Mohd Matore, M.E.E., Niusila Faamanatu-Eteuati, & Norhidayu Rosman (2021). Rating quality in rater mediated language assessment: A systematic literature review. Journal of Contemporary Issues in Business and Government, 27(2), 6096–6116. https://doi.org/10.47750/cibg.2021.27.02.606
Price, P. C., Jhangiani, R. S., Chiang, I-C., A., Leighton, D. C., & Cuttler, C. (2017). Research methods in psychology. (3rd American ed.). Pressbooks. https://opentext.wsu.edu/carriecuttler/
Saito, K., & Shintani, N. (2016). Foreign accentedness revisited: Canadian and Singaporean raters’ perception of Japanese-accented English. Language Awareness, 25(4), 305–317.
https://doi.org/10.1080/09658416.2016.1229784
UCLA Advanced Research Computing (n.d.). What does Cronbach’s alpha mean? SPSS FAQ. https://stats.oarc.ucla.edu/spss/faq/what-does-cronbachs-alpha-mean/
Wei, J., & Llosa, L. (2015). Investigating differences between American and Indian raters in assessing TOEFL iBT speaking tasks. Language Assessment Quarterly, 12(3), 283–304.
Wind, S. A., & Peterson, M. E. (2018). A systematic review of methods for evaluating rating quality in language assessment. Language Testing, 35(2), 161–192. https://doi.org/10.1177/0265532216686999
Winke, P., Gass, S., & Myford, C. M. (2011). The relationship between raters’ prior language study and the evaluation of foreign language speech samples. ETS Research Report Series 2011(2), i–67. https://doi.org/10.1002/j.2333-8504.2011.tb02266.x
Xi, X., & Mollaun, P. (2009). How do raters from India perform in scoring the TOEFL iBT speaking section and what kind of training helps? ETS Research Report Series 2009(2), i–37.
https://doi.org/10.1002/j.2333-8504.2009.tb02188.x
Yan, X., & Fan, J. (2022). Reliability and dependability. In G. Fulcher & L. Harding (Eds.), The Routledge handbook of language testing (2nd ed.) (pp. 477–494). Routledge.
Zhang, Y., & Elder, C. (2014). Investigating native and non-native English-speaking teacher raters’ judgements of oral proficiency in the College English Test-Spoken English Test (CET-SET). Assessment in Education: Principles, Policy & Practice, 21(3), 306–325. https://doi.org/10.1080/0969594X.2013.845547