Mapping STOU-EPT Scores with the CEFR Framework: A Standard Alignment Approach
Article Sidebar
Main Article Content
Abstract
This study examined the alignment of the Sukhothai Thammathirat Open University English Proficiency Test (STOU-EPT) with the Common European Framework of Reference for Languages (CEFR) using Webb’s alignment method, and mapped STOU-EPT scores to CEFR levels with a modified Angoff method. Ten panelists evaluated alignment, ten panelists participated in standard setting, and 1,272 STOU-EPT test-takers were used. The STOU-EPT aligned with the CEFR as follows. Categorical Occurrence: Items across listening, structure, and reading were aligned to the A2–B2. Listening was more evenly distributed across levels, more structure items were included in A2–B1, and more reading items were included in B1–B2. All three sections met Webb’s minimum requirement, with high rater agreement on listening and moderate agreement on structure and reading. Depth of Knowledge (DOK): Cognitive demand on listening and structure aligned with CEFR expectations at B1 and B2 but did not at A2. Reading showed appropriate cognitive demand across all targeted levels. Balance of Representation: The distribution of items across CEFR levels was balanced on listening, structure, and reading. CEFR-mapped scores were identified for each skill and overall performance, categorizing total scores as A2 (1–55 items), B1 (56–76), and B2 (77–100).
Article Details
References
Al Lawati, Z. A. (2023). Investigating the characteristics of language test specifications and item writer guidelines, and their effect on item development: A mixed-method case study. Language Testing in Asia, 13(1), 21. https://doi.org/10.1186/s40468-023-00233-5
Alderson, J. C. (2005). Diagnosing foreign language proficiency: The interface between learning and assessment. Continuum.
Alderson, J. C. (2007). The CEFR and the need for more research. The Modern Language Journal, 91(4), 659–663. http://www.jstor.org/stable/4626093
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. https://www.testingstandards.net/uploads/7/6/6/4/76643089/standards_2014edition.pdf
Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 508–600). American Council on Education.
Athiworakun, C., & Wudthayagorn, J. (2018). Mapping Srinakharinwirot University - Standardized English Test (SWU-SET) onto the Common European Framework of Reference (CEFR). Suranaree Journal of Social Science, 12(2), 69–84.
Baron, P. A., & Papageorgiou, S. (2016). Setting language proficiency score requirements for English-as-a-second-language placement decisions in secondary education (Research Report ETS RR-16-17). Educational Testing Service. https://doi.org/10.1002/ets2.12102
British Council & EAQUALS. (2015). British Council – EAQUALS core inventory for general English (2nd ed.). https://www.teachingenglish.org.uk/sites/teacheng/files/pub-british-council-eaquals-core-inventoryv2.pdf
Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Sage.
Council of Europe. (2020). Common European Framework of Reference for Languages: Learning, teaching, assessment. Companion volume. Council of Europe Publishing. https://www.coe.int/lang-cefr
DeMars, C. (2010). Item response theory. Oxford University Press.
EAQUALS. (2008). EAQUALS bank of descriptors – As checklists. https://www.eaquals.org/wp-ontent/uploads/EAQUALS_Bank_as_checklists.pdf
Evans, J. D. (1996). Straightforward statistics for the behavioral sciences. Thomson Brooks/Cole.
Figueras, N. (2012). The impact of the CEFR. ELT Journal, 66(4), 477–485. https://doi.org/10.1093/elt/ccs037
Forte, E. (2017). Evaluating alignment in large-scale standards-based assessment systems. Council of Chief State School Officers.
Green, A. (2020). Exploring language assessment and testing: Language in action (2nd ed.). Routledge.
Hambleton, R. K., & Pitoniak, M. J. (2006). Setting performance standards. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 433–470). American Council on Education/Praeger.
Hulstijn, J. H. (2007). The shaky ground beneath the CEFR: Quantitative and qualitative dimensions of language proficiency. The Modern Language Journal, 91(4), 663–667. https://doi.org/10.1111/j.1540-4781.2007.00627_5.x
Jiraro, S., & Angsuchoti, S. (2022). The concurrent validity of Sukhothai Thammathirat Open University’s English proficiency test with the computer system under the Common European Framework of Reference for Languages. ASEAN Journal of Open and Distance Learning (AJODL), 14(1), 98–107.
Kane, M. T. (2006). Validation. In R. L. Brennan (Ed.), Educational measurement (4th ed., pp. 17–64). American Council on Education
/Praeger.
Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50(1), 1–73. https://doi.org/10.1111/jedm.12000
Livingston, S. A., & Zieky, M. J. (1982). Passing scores: A manual for setting standards of performance on educational and occupational tests. Educational Testing Service. https://www.ets.org/Media/Research/pdf/passing_scores.pdf
Ngudgratoke, S. (2018). Assessment and evaluation for standard-based education management. Chatuporn Design.
North, B. (2000). The development of a common framework scale of language proficiency. https://www.researchgate.net/publication/312489070
Office of the Higher Education Commission. (2016). Policy on enhancing English standards in higher education institutions 2016. http://www.dqe.mhesi.go.th/front_home/Data%20Bhes_2559/04052559.pdf
Papageorgiou, S., Wu, S., Hsieh, C.-N., Tannenbaum, R. J., & Cheng, M. (2019). Mapping the TOEFL iBT®test scores to China’s standards of English language ability: Implications for score interpretation and use (Research Report No. RR–19-44). Educational Testing Service. https://doi.org/10.1002/ets2.12281
Plake, B. S., & Cizek, G. J. (2012). The modified Angoff, extended Angoff, and yes/no standard setting methods. In G.J. Cizek (Ed.), Setting performance standards: Foundations, methods, and innovations (pp. 181–253). Routledge.
Roach, A. T., & Elliott, S. N. (2004). Alignment analysis and standard-setting procedures for alternate assessments (WCER Working Paper No. 2004–1). Wisconsin Center for Education Research, University of Wisconsin–Madison.
Roach, A. T., Elliott, S. N., & Webb, N. L. (2005). Alignment of an alternate assessment with state academic standards: Evidence for the content validity of the Wisconsin alternate sssessment. The Journal of Special Education, 38(4), 218–231.
Rovinelli, R. J., & Hambleton, R. K. (1977). On the use of content specialists in the assessment of criterion-referenced test item validity. Tijdschrift voor Onderwijsresearch, 2, 49–60.
Tangsakul, S., & Poonpon, K. (2024). Aligning academic reading tests to the Common European Framework of Reference for Languages (CEFR). rEFLections, 31(2), 614–638. https://doi.org/10.61508/refl.v31i2.275057
Tannenbaum, R. J. & Baron, P. A. (2011). Mapping TOEIC® ITP scores onto the Common European Framework of Reference (Research Memorandum ETS RM—11–33). Educational Testing Service. https://www.ets.org/Media/Research/pdf/RM-11-33.pdf
Traynor, A., Webb, N. L., Christopherson, S., & Sato, E. (2020). Using the results of content alignment analyses to inform ongoing item-level improvements to an assessment program: A guide for state departments of education and for assessment vendors. Wisconsin Center for Education Products and Services.
Webb, N. L. (1997). Criteria for alignment of expectations and assessments in mathematics and science education (Research Monograph No. 6). Council of Chief State School Officers.
https://files.eric.ed.gov/fulltext/ED414305.pdf
Webb, N. L. (2007). Issues related to judging the alignment of curriculum standards and assessments. Applied Measurement in Education, 20(1), 7–25. http://dx.doi.org/10.1207/s15324818ame2001_2
Wudthayagorn, J. (2018). Mapping the CU-TEP to the Common European Framework of Reference (CEFR). LEARN Journal: Language Education and Acquisition Research Network, 11(2), 163–180. https://so04.tci-thaijo.org/index.php/LEARN/article/view/161641
