Citation
Burke, K., Ouellette, J., Miller, W., Leise, C., & Utschig, T. (2012). Measuring Writing as a Representation of Disciplinary Knowledge. International Journal of Process Education, 4(1), 13–27. http://www.ijpe.online//2012/writingh.pdf
Abstract
Most students are exposed to different writing genres as well as to grammar and other writing-specific skills during their
required composition course(s). Discipline-specific writing skills are often not taught, rather they are assumed to be prerequisite
knowledge by disciplinary instructors. Rubrics are often used by instructors to measure these skills; however, few rubrics
integrate measurement of both general and discipline-specific skills. One such rubric has been designed by the Academy of
Process Educators. The present research examines the empirical reliability and validity of this rubric. It was hypothesized
that consistency of ratings would be achieved for three different types of writing by four raters from different disciplines when
read in a counterbalanced pattern. The Written Communication Value Rubric from the Association of American Colleges and
Universities (AAC&U, 2012) was also utilized to make possible the evaluation of concurrent validity of the two rubrics. Different
statistical methods were utilized to analyze the ratings. The research methodology utilized provides a model for empirically
exploring the reliability and validity of other rubrics. Overall, our findings indicate that individually raters were consistent
on both rubrics, but the inter-rater reliability was not strong. The ratings were less dispersed on the Academy rubric than
on the AAC&U rubric but “concurrent” validity, that is significant positive correlations between the rubrics, were clearly
present. Some of the implications of the study include the importance of encouraging careful rater and inter-rater training, using
counter-balanced sequences to avoid bias from repetition, ensuring that writing assignments reflect clear performance criteria in
regards to rubric performance areas, and assuring that writing pre-requisites are carefully integrated throughout the program’s
curriculum and assessment strategy