Examining and controlling rater severity and leniency effects on alignment evaluation between science items and science learning indicators using many-facets Rasch modeling

Abstract

This research aimed to (1) examine rater severity and leniency effects on alignment evaluation between the science items and the science learning indicators in junior secondary education, and (2) investigate the alignment evaluation between the science items and the learning indicators when rater severity and leniency effects were controlled. Research subjects were (1) 1,089 in-class science items collected from junior secondary schools under the Office of the Basic Education Commission in Bangkok, and (2) 20 expert panelists participating on alignment evaluation process. Research instrument was a rating scale questionnaire. The inter-rater reliability examined by intra-class correlation was .94 (95% confident interval [CI]: .91–.97). Data analysis was performed using the application of Many-facet Rasch Modeling (MFRM) in FACETS software, version 3.80.3 (Linacre, 2018) and paired-samples t-test. The findings revealed that (1) there were rater severity and leniency effects on alignment evaluation between science items and learning indicators; and (2) when the rater effects were controlled by the MFRM, there was statistically significant difference at the .01 level between non-controlled and controlled mean alignment scores, and alignment evaluation scores of some items were shifted when the effects
were controlled.