•  
  •  
 

Keywords

Rasch analysis; constructvalidity; differential itemfunctioning; local itemdependence; Englishreading comprehension

Document Type

Article

Abstract

This study reports the psychometric evaluation of an item bank for an Assessment for Learning (AfL) tool to assess primary school students’ reading comprehension skills. A pool of 46 primary 1 to 6 reading passages and their accompanying 522 multiple choice and short answer items were developed based on the Progress in International Reading Literacy Study (PIRLS) assessment framework. They were field-tested at 27 schools in Singapore involving 9834 students aged between 7 and 13. Four main com- prehension processes outlined in PIRLS were assessed: focusing on and retrieving ex- plicitly stated information, making straightforward inferences, interpreting and inte- grating ideas and information, and evaluating and critiquing content and textual ele- ments. Rasch analysis was employed to examine students’ item response patterns for (1) model and item fit; (2) differential item functioning (DIF) about gender and test platform used; (3) local item dependence (LID) within and amongst reading passages; and (4) distractor issues about options within the multiple-choice-type items. Results showed that the data adequately fit the unidimensional Rasch model across all test levels with good internal consistency. Psychometric issues found amongst items were primarily related to ill-functioning distractors and local dependence on items. Problem- atic items identified were reviewed and subsequently amended by a panel of assess- ment professionals for future recalibration. This psychometrically and theoretically sound item bank is envisaged to be valuable to developing comprehensive classroom AfL tools that provide information for the English reading comprehension instruc- tional design in the Singaporean context.

Page Range

18-34

Issue

1

Volume

10

Digital Object Identifier (DOI)

10.21831/reid.v10i1.65284

Source

https://journal.uny.ac.id/index.php/reid/article/view/65284

References

Andrich, D., & Marais, I. (2019). A course in Rasch measurement theory. Springer texts in education. Springer Singapore. https://doi.org/10.1007/978-981-13-7496-8_4

Bond, T. G. (2003). Validity and assessment: A Rasch measurement perspective. Metodologia de las Clecias del Comportamientro, 5(2), 179-194.

Bond, T. G., & Fox. C. M. (2015). Applying the Rasch model: Fundamental measurement in the human sciences (3rd ed.). Routledge. https://doi.org/10.4324/9781315814698

Cantó-Cerdán, M., Cacho-Martínez, P., Lara-Lacárcel, F., & García-Muñoz, Á. (2021). Rasch analysis for development and reduction of Symptom Questionnaire for Visual Dysfunctions (SQVD). Scientific Reports, 11(1), 14855. https://doi.org/10.1038/s41598-02194166-9

Christensen, K. B., Makransky, G., & Horton, M. (2017). Critical values for Yen’s Q 3: Identification of local dependence in the Rasch model using residual correlations. Applied psychological measurement, 41(3), 178-194. https://doi.org/10.1177/0146621616677520

Fan, J., & Bond, T. (2019). Applying Rasch measurement in language assessment. In V. Aryadoust & M. Raquel (Eds.), Quantitative data analysis for language assessment volume I: Fundamental techniques (pp. 83-102). Routledge. https://www.taylorfrancis.com/chapters/edit/10.4324/9781315187815-5/applying-raschmeasurement-language-assessment-jason-fan-trevor-bond

Gierl, M. J., Bulut, O., Guo, Q., & Zhang, X. (2017). Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review. Review of Educational Research, 87(6), 1082–1116. https://doi.org/10.3102/0034654317726529

Hagell, P., & Westergren, A. (2016). Sample size and statistical conclusions from tests of fit to the Rasch Model according to the Rasch Unidimensional Measurement Model (RUMM) Program in Health Outcome Measurement. Journal of Applied Measurement, 17(4), 416 – 431.

Hwang, G. J., & Wu, P. H. (2014). Applications, impacts, and trends of mobile technologyenhanced learning: A review of 2008–2012 publications in selected SSCI journals. International Journal of Mobile Learning and Organisation, 8(2), 83-95. https://doi.org/10.1504/IJMLO.2014.062346

Korhonen, J., Linnanmäki, K., & Aunio, P. (2014). Learning difficulties, academic well-being, and educational dropout: A person-centred approach. Learning and Individual Differences, 31, 1-10. https://doi.org/10.1016/j.lindif.2013.12.011

Livingstone, S., Mascheroni, G., & Staksrud, E. (2018). European research on children’s internet use: Assessing the past and anticipating the future. New Media & Society, 20(3), 1103-1122. https://doi.org/10.1177/1461444816685930

Messick, S. (1995). Standards of validity and the validity of standards in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5-8. https://doi.org/10.1111/j.17453992.1995.tb00881.x

Messick, S. (1994). Validity of psychological assessment: Validation of inferences from persons' responses and performances as scientific inquiry into score meaning. ETS Research Report Series, 1994(2), i-28. https://doi.org/10.1002/j.2333-8504.1994.tb01618.x

Ministry of Education of Singapore. (2020). Pedagogy: Teaching and learning English. In English language syllabus – Primary foundation English - Secondary normal (technical) course. Curriculum Planning and Development Division (CPDD), pp. 30–32. https://www.moe.gov.sg//media/files/secondary/syllabuses-nt/eng/felnt_els-2020_syllabus.pdf

Mugnaini, D., Lassi, S., La Malfa, G., & Albertini, G. (2009). Internalizing correlates of dyslexia. World Journal of Pediatrics, 5, 255-264. https://doi.org/10.1007/s12519-009-0049-7

Mullis, I. V. S., & Martin, M. O. (Eds.). (2021). PIRLS 2021 Assessment frameworks. Retrieved from Boston College, TIMSS & PIRLS International Study Center website: https://pirls2021.org/frameworks/wpcontent/uploads/sites/2/2019/04/P21_FW_Ch1_Assessment.pdf

Oakley, G. (2011). The assessment of reading comprehension cognitive strategies: Practices and perceptions of Western Australian teachers. The Australian Journal of Language and Literacy, 34(3), 279-293. https://doi.org/10.1007/BF03651863

OECD. (2016). PISA 2015 results (Volume I): Excellence and equity in education. OECD Publishing. https://doi.org/10.1787/9789264266490-en

Pallant, J. F., & Tennant, A. (2007). An introduction to the Rasch measurement model: An example using the Hospital Anxiety and Depression Scale (HADS). British Journal of Clinical Psychology, 46(1), 1-18. https://doi.org/10.1348/014466506X96931

Pearson, P. D., & Hamm, D. N. (2005). The assessment of reading comprehension: A review of practices—Past, present, and future. In S. G. Paris & S. A. Stahl (Eds.), Children's reading comprehension and assessment, 31-88. https://doi.org/10.4324/9781410612762

Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen Danmarks Paedagogiske Institut.

Ravand, H., & Firoozi, T. (2016). Examining the construct validity of the Master’s UEE using the Rasch model and the six aspects of Messick’s framework. International Journal of Language Testing, 6(1), 1-23. https://www.ijlt.ir/article_114414.html

RAND Reading Study Group (2002). Reading for understanding: Toward an R & D program in reading comprehension. RAND Corporation. https://www.rand.org/pubs/monograph_reports/MR1465.html

Smith, R., Snow, P., Serry, T., & Hammond, L. (2021). The role of background knowledge in reading comprehension: A critical review. Reading Psychology, 42(3), 214-240. https://doi.org/10.1080/02702711.2021.1888348

Download

Share

COinS