Examining and controlling rater severity and leniency effects on alignment evaluation between science items and science learning indicators using many-facets Rasch modeling

Authors

  • Budsayarat Janprasert Department of Educational Research and Psychology, Faculty of Education, Chulalongkorn University, Pathumwan, Bangkok 10330 Thailand
  • Nuttaporn Lawthong Department of Educational Research and Psychology, Faculty of Education, Chulalongkorn University, Pathumwan, Bangkok 10330 Thailand
  • Sungworn Ngudgratoke School of Education, Sukhothai Thammathirat Open University, Nonthaburi 11120, Thailand

Keywords:

alignment, many-facets Rasch modeling, rater severity and leniency effects

Abstract

This research aimed to (1) examine rater severity and leniency effects on alignment evaluation between the science items and the science learning indicators in junior secondary education, and (2) investigate the alignment evaluation between the science items and the learning indicators when rater severity and leniency effects were controlled. Research subjects were (1) 1,089 in-class science items collected from junior secondary schools under the Office of the Basic Education Commission in Bangkok, and (2) 20 expert panelists participating on alignment evaluation process. Research instrument was a rating scale questionnaire. The inter-rater reliability examined by intra-class correlation was .94 (95% confident interval [CI]: .91–.97). Data analysis was performed using the application of Many-facet Rasch Modeling (MFRM) in FACETS software, version 3.80.3 (Linacre, 2018) and paired-samples t-test. The findings revealed that (1) there were rater severity and leniency effects on alignment evaluation between science items and learning indicators; and (2) when the rater effects were controlled by the MFRM, there was statistically significant difference at the .01 level between non-controlled and controlled mean alignment scores, and alignment evaluation scores of some items were shifted when the effects
were controlled.

Downloads

Published

01-12-2020

How to Cite

Janprasert, B. ., Lawthong, N., & Ngudgratoke, S. (2020). Examining and controlling rater severity and leniency effects on alignment evaluation between science items and science learning indicators using many-facets Rasch modeling. Kasetsart Journal of Social Sciences, 41(3), 592–597. Retrieved from https://so04.tci-thaijo.org/index.php/kjss/article/view/247652

Issue

Section

Research articles