Abstract
Purpose: The curriculum-based measurement (CBM) metric, correct minus incorrect writing sequences (CIWS) tends to show higher reliability than other rubric measures and higher validity than other CBM metrics (Romig et al., 2017). However, CIWS relies extensively on grammar rules that raters explicitly and implicitly consider "correct." In the present study we: 1) re-wrote rules of CBM-WE that might misclassify dialect variation as grammatical or spelling errors for speakers of nonmainstream American English (NMAE), and 2) explored the differences in the two sets of scoring rules for students who had NMAE dialect features in their writing.
Purpose: The curriculum-based measurement (CBM) metric, correct minus incorrect writing sequences (CIWS) tends to show higher reliability than other rubric measures and higher validity than other CBM metrics (Romig et al., 2017). However, CIWS relies extensively on grammar rules that raters explicitly and implicitly consider "correct." In the present study we: 1) re-wrote rules of CBM-WE that might misclassify dialect variation as grammatical or spelling errors for speakers of nonmainstream American English (NMAE), and 2) explored the differences in the two sets of scoring rules for students who had NMAE dialect features in their writing.
Method: New scoring rules were aggregated from multiple studies of the morphosyntactical and phonological features commonly present in NMAE oral language. To examine the second research aim, written compositions from 432 students in grades 3 through 5 in a southern state were examined for the presence of NMAE features. Of these compositions, 113 samples used NMAE features and these samples were scored using both the standard, unmodified and NMAE-modified CIWS scoring rules.
Results: Scoring rules were developed that differentiated between rule-governed grammatical NMAE features (reclassified from incorrect to correct sequences) and NMAE features that overlap with clinical indicators of language disorder (Lee-James & Johnson, 2022), which remained classified as "incorrect" sequences. When comparing student scores using the two different scoring rules the modified rules improved students’ CIWS scores (Hedges's g = 0.168).
Conclusions: Additional research is needed to determine the predictive consequences of the new scoring rules and its potential impact on scoring bias.
Johnson, L. J., ^Bourgeois, S., ^Friedman, E., Truckenmiller, A. J., Johnson, L. C., Hennenfent, L., Sarmiento, C. M., Troia, G., Cho, E. (2023). Exploring the effects of culturally responsive scoring modifications in curriculum-based measurement in written expression. [Manuscript in preparation]. Department of Counseling, Educational Psychology, and Special Education, Michigan State University.