VALUE Research Hub

Measuring Writing as a Representation of Disciplinary Knowledge

[RETURN_MARKER:207046944]

Citation

Burke, K., Ouellette, J., Miller, W., Leise, C., & Utschig, T. (2012). Measuring Writing as a Representation of Disciplinary Knowledge. International Journal of Process Education, 4(1), 13–27. http://www.ijpe.online//2012/writingh.pdf

Abstract

Most students are exposed to different writing genres as well as to grammar and other writing-specific skills during their required composition course(s). Discipline-specific writing skills are often not taught, rather they are assumed to be prerequisite knowledge by disciplinary instructors. Rubrics are often used by instructors to measure these skills; however, few rubrics integrate measurement of both general and discipline-specific skills. One such rubric has been designed by the Academy of Process Educators. The present research examines the empirical reliability and validity of this rubric. It was hypothesized that consistency of ratings would be achieved for three different types of writing by four raters from different disciplines when read in a counterbalanced pattern. The Written Communication Value Rubric from the Association of American Colleges and Universities (AAC&U, 2012) was also utilized to make possible the evaluation of concurrent validity of the two rubrics. Different statistical methods were utilized to analyze the ratings. The research methodology utilized provides a model for empirically exploring the reliability and validity of other rubrics. Overall, our findings indicate that individually raters were consistent on both rubrics, but the inter-rater reliability was not strong. The ratings were less dispersed on the Academy rubric than on the AAC&U rubric but “concurrent” validity, that is significant positive correlations between the rubrics, were clearly present. Some of the implications of the study include the importance of encouraging careful rater and inter-rater training, using counter-balanced sequences to avoid bias from repetition, ensuring that writing assignments reflect clear performance criteria in regards to rubric performance areas, and assuring that writing pre-requisites are carefully integrated throughout the program’s curriculum and assessment strategy