AAC&U News | December 2004

Portfolios Transform Writing Assessment at Carleton College

Editor's Note: In September 2004, AAC&U's board of directors released Our Students' Best Work: A Framework for Accountability Worthy of Our Mission, a statement that urges all colleges and universities to focus their educational planning and assessment efforts on a set of outcomes essential for success in today's world. The statement noted that there is a strong consensus emerging among educators as well as business, policy, and civic leaders that higher education must "embrace a small number of highly valued and widely affirmed educational goals, establish high standards for each, and assess their achievement across the curriculum." Many AAC&U member institutions have already made progress in organizing their curriculum and assessment programs around these goals. These institutions' innovations provide valuable models for how to effectively advance and comprehensively assess the outcomes that form the core of a contemporary liberal education. Over the next several months, AAC&U News will highlight several of these models, focusing on the key learning outcomes described in Our Students' Best Work.

A recent study by the National Commission on Writing has shed light on a disturbing trend: at a time when writing increasingly is seen as a fundamental skill for professionals, many employers are reporting that college graduates have difficulty communicating in clear, grammatical prose. This finding, drawn from a national survey of business leaders, has economic as well as educational implications. According to the study, businesses in the United States may spend as much as $3.1 billion on remedial writing instruction each year, with many additional dollars lost as a result of inefficiencies related to poor writing. To prepare graduates for the contemporary workplace, the commission concludes, educators need to develop more comprehensive and sustained ways of teaching and assessing writing.

Carleton College, a private, liberal arts institution in Northfield, Minnesota, is one of a growing number of schools developing new strategies to improve and assess students' writing skills. Carleton has long emphasized "writing across the curriculum"—distributing writing instruction in many classes and throughout the disciplines—and in 2001, the college introduced sophomore writing portfolios to broaden its assessment of student writing. Required midway through every student's college career, the writing portfolios enable the college to ensure that undergraduates can write competently in a range of styles and contexts. At the same time, by encouraging students to reflect on—and revise—their writing, the portfolios themselves constitute an important educational experience.

A New Writing Requirement

Carol Rutz, the director of the writing program at Carleton College, has overseen the school's transition from its old "single course" model of writing assessment to the use of writing portfolios. By the time Carleton decided to abandon the old model, she says, there was a broad consensus among students and faculty that assessments based on writing from a single class were inadequate. But when the school decided on portfolios as a new assessment vehicle, it faced a different set of challenges. Portfolio assessment requires large amounts of time and money and significant administrative oversight. "The wealth of data comes at a price," Rutz says, "but we are finding—even after a very short period of implementation—that gains made in faculty development and student self-awareness are well worth the investment."

To meet the school's current portfolio requirement, students at the end of their sophomore year must submit three to five papers demonstrating their ability to write effectively in different rhetorical and disciplinary contexts. The papers must be "authenticated" by instructors, who certify that the papers were written for their classes and indicate if they have since been revised. Finally, students write reflective essays about their writing to introduce the portfolios.

Requiring such a breadth of written work—as opposed to the narrower sample of student writing produced in single course writing assessments—tends to foster healthier attitudes toward writing. Clara Shaw Hardy, a professor in the Department of Classical Languages who is active in the portfolio program, notes that the breadth requirement "eliminates the perception that writing is something to be checked off after one course rather than a skill that underlies most of an undergraduate's education." Moreover, she says, "it honors the fact that different disciplines value different kinds of writing skills."

Professor Hardy stresses that her own approach to teaching—like that of many of her colleagues—has also been "radically affected" by participation in the writing program. "I now regularly require drafts, use peer feedback, and incorporate a wide range of formal and informal writing assignments in my courses," she says. Faculty development programs, including grants for course development, workshops, and speaker series, contribute greatly to such successes. Indeed, in the 2003-04 academic year, 41.5 percent of tenured and tenure-track faculty participated in grant-funded activities related to the writing program.

Evaluating the portfolios, a task Carleton assigns to a volunteer group of "faculty readers," provides another opportunity for faculty development. Associate Dean Elizabeth Ciner, who has been involved in the writing program at Carleton for almost three decades, points out that faculty readers gain a broad perspective on the state of student writing on campus. And these faculty members inevitably learn from what they find: "To see what others are doing in their writing assignments, to see how well students can write, to see what gives students general difficulty . . . all of this is possible reading portfolios."

Assessing the Portfolios

During assessment, faculty readers assign one of three scores to every writing portfolio: "pass," "exemplary," or "needs work." These scores are determined according to locally developed standards, says Jacqulyn Lauer-Glebov, Carleton's coordinator of educational assessment. Such locally developed assessments have several advantages over national standards-based assessments. Unlike high-stakes, "snapshot" assessments such as the new SAT writing test, Lauer-Glebov says, portfolios offer a flexible assessment model that can document a range of ability.

Carleton's evaluation criteria emphasize the demonstration of this range. Each portfolio must represent at least two of the college's four curricular divisions (Arts and Literature, Humanities, Social Sciences, and Mathematics/Natural Sciences) and must include at least one paper from the student's "Writing Requirement" course. Together, the papers must also demonstrate their author's mastery of each of several key writing skills—the ability to report on observation, to analyze complex information, to provide interpretation, to use and document sources, and to articulate and support a thesis-driven argument.

In addition to assessing the student's success in these areas, faculty readers also provide feedback on the quality of writing. Each portfolio is rated on whether it "rarely," "usually," or "consistently" demonstrates attention to audience and purpose, clarity of prose, clear organization, effective use of evidence, distinctive voice, appropriate diction, and control of error.

At the end of the assessment process, about 8 percent of portfolios typically are rated "needs work" (an average of 78 percent receive a passing grade, with 14 percent earning "exemplary"). Students who do not pass the writing assessment resubmit their portfolios after receiving written suggestions and, if needed, personalized counseling.

Looking Ahead

As more information becomes available about the strengths and weaknesses of portfolio-based writing assessments, Carleton's writing program will continue to evolve. Some aspects of the program, such as the treatment of failing portfolios, have already come under scrutiny. As Carol Rutz points out, assessing student writing midway through college affords Carleton "a chance to intervene with students who need help," but students whose portfolios do fail still receive help only on a case-by-case basis. Adopting a more formalized approach to working with these students could strengthen the program.

On the other hand, early data already is indicating some positive trends. For example, an analysis of portfolio contents suggests that Carleton's faculty development efforts have paid off: a statistically significant correlation has been found between faculty participation in development activities and the presence of papers in portfolios, with students more likely to include papers from courses taught by faculty who have been active in the writing program. This finding, Jacqulyn Lauer-Glabov explains, suggests that "as faculty become more conscious of the writing they ask of their students, students respond by selecting that work to represent their writing mastery."

Ciner hopes that the effects of such successes will extend beyond student writing. The writing program itself, she says, is "becoming a model for all sorts of curricular initiatives—and our experience in portfolios is guiding us as we consider other aspects of the curriculum." These initiatives—which target other key learning outcomes such as information literacy, interdisciplinary learning, quantitative reasoning, and critical thinking—promise to interact with the writing program in fruitful ways and continue the transformation of undergraduate learning at Carleton.

For further information about Carleton College's writing portfolios, visit the writing program's Web pages. The results of the survey from the National Commission on Writing are also online. AAC&U's recent accountability statement, Our Students Best Work, offers a blueprint for assessment efforts at American colleges and universities. For more information about innovative writing programs in colleges and universities today, see the special issue of Peer Review on that topic.