Liberal Education

Assessment: What Is It Good For? (For Faculty, It Turns Out)

To meet the rising expectations of accrediting organizations, colleges and universities are being urged to produce data on student learning outcomes. If faculty and administrators are to add assessment design and implementation to our many responsibilities, however, we need to see assessment as a worthwhile intellectual enterprise, not as a form of surveillance or reductive quantification of knowledge. How will the assessment process make us better teachers and our students better learners? And how can it possibly be compatible with, much less enhance, our work as scholars?

Here, we report on the results of a three-year study we conducted on assessment practices in general education courses at Duke University as part of a three-institution (Duke, the University of Kansas, and the University of Nebraska) research project funded by the Spencer Foundation. At all three institutions, researchers asked: Are data from the assessment process being communicated and acted on as they should? Above all, what might make the data gathered in the assessment process more likely to produce meaningful change? Our study addressed these questions, but our results spoke also to the concerns of many faculty about assessment’s intellectual rigor and its efficacy as a tool for better teaching. How can assessment be relevant, and even contribute in some way, to faculty members’ work as scholars in their disciplines?

The investigators began by acknowledging that many faculty remain alienated from and by assessment—a legacy of the top-down manner in which the assessment process started at most institutions, which continues to affect how data are gathered and reported. This top-down model persists despite considerable evidence that meaningful faculty participation is vital to the success of assessment processes. Our joint investigation addressed this problem by putting the focus on instructors in the classroom. The project asked: What happens when individual faculty not only determine what they want to learn from any assessment they conduct, but even anticipate what they might learn in order to make their assessment tools more robust and the likelihood of using the information greater? And what happens if they are invited to themselves draw meaning out of data from their courses? In short, what happens when faculty “close the loop” by drawing conclusions from their own data and then offering their conclusions to administrators?

At Duke, the study focused on the Writing-in-the-Disciplines (WID) requirement—a general education objective from the College of Arts and Sciences. We—the two faculty who ran the Duke study—are a historian and a cultural anthropologist who came to this project after having taken leadership roles in a university committee tasked with ensuring departments’ compliance in college-wide assessment used for accreditation by the Southern Association of Colleges and Schools Commission on Colleges—outward-facing assessment, in short. We were tapped for that assignment because we had been leaders in developing assessment protocols in our own departments (history and the Thompson Writing Program). In those roles, we had discovered firsthand the gap that has developed between compliance with demands for assessment, on the one hand, and subsequent use of assessment data for real educational improvement on the other. That disconnect was more obvious to us than it was to the other faculty who served alongside us because the two of us had also worked side by side as directors of the college’s interdisciplinary academic writing program. There, we treated assessment as an imperative partly because the writing program—which teaches every incoming student—has to justify its existence and compete for resources more energetically than does a conventional department.

In the writing program, we also treat assessment as an integral part of teaching; we train new postdocs in teaching writing in their disciplines, with the expectation that they will engage in self-reflection and use their experience and data to spur improvement. Thus, we treat teaching as an exercise in evidence-based inquiry. That culture—one that treats assessment not grudgingly as a fact of life but as an indispensable part of teaching that lies completely within the competence and purview of each faculty member—makes us outliers among our peers at Duke and ideally placed to lead other faculty in investigating their own teaching from that stance.

Study design

This study’s design assumes that decision making can be understood as “sense making,” a reiterative social process that involves framing, interpretation, argumentation, and persuasion.1 One of the implications of this framework is that those who will use the data (i.e., faculty and administrators) need to be involved in designing the assessment and interpreting the findings. In other words, those responsible for applying the data should be involved in making meaning out of them—in asking the questions that will generate them, in grappling with their implications, and in anticipating their uses. Thus the research question driving this study was as follows:

Does participation in a sense-making
simulation exercise improve the use of data
to inform educational decisions and
improve learning?2

To answer this question, the study followed a common protocol in all three institutions. It included a series of surveys designed to measure whether and to what extent the use of student assessment in academic decisions (the independent variable) was affected by any of four dependent variables: (1) personal characteristics, including knowledge of and disposition toward assessment; (2) organizational context, including institutional support for assessment; (3) information characteristics, including quality of data and compatibility with expectations; and (4) participation in the sense-making process.

At Duke, we offered a small stipend ($1,000 to $1,500) to faculty across campus who were then teaching writing-intensive courses and invited them to participate in either the control group ($1,000) or the experiment group ($1,500). Faculty in the control group received a rubric that they could choose to modify, and which they would then use to assess a writing assignment at the start of the semester and another at the semester’s end. They then reported on their experience of using the rubric and briefly reflected, in writing, on how that experience might inform their future teaching and/or assessment practices. The experiment group, identified in the study as the “simulation group,” participated in two meaning-making workshops bookending the semester. In the first, we gave faculty an overview of the study and led them through a series of brainstorming exercises and discussions aimed at helping them identify learning goals for their courses—not only for their students but, most importantly, for themselves.

Due to the small sample size at Duke, our survey results were inconclusive, but the qualitative data we gathered from participants, particularly in the experiment group, offered valuable insight into their experiences—especially those related to the fourth variable above, referred to in the study as “sense making.”

Experiment group: qualitative findings

In the first of the two workshops, we asked participants in the experiment group: What frustrated them about how students have responded to their assignments in the past, and how had they addressed it? What did they want to know about how students process and respond to their assignments, and about the roadblocks students may encounter?

A public policy professor wondered how she could get her students to “embrace risk” in their writing. A philosophy professor asked herself how she could help her students “think like philosophers.” A sociology professor wanted her students to frame good research questions. And an English professor wanted to pinpoint exactly what she found off-putting about her students’ personal essays. For some faculty, like the philosopher, the exercise in identifying the features they were looking for in their students’ written work was their first experience of naming the analytical or interpretive moves essential to their discipline. Some, like the English professor, needed first to attend to what was not working in students’ writing to move further.

Faculty shared their thoughts about issues they wished to assess in their respective courses, then discussed how they might assess those issues. Would they revise the language in their course materials? Create new assignments or adapt existing ones? Rethink their assignment sequencing? Would they use a grading or an evaluating rubric, and would they share it with their students? Using a workbook designed specifically for the workshop, participants then devised a plan for revising their course materials and created a rubric for assessing the learning goals they had identified.

After using their newly designed criteria to rate a set of early-semester and another set of late-semester student writing, faculty gathered for a second workshop to make meaning of their findings, reflect on what they had learned, and consider the implications of their data for future iterations of their courses. Not all participants generated numerical data, but they unanimously found the exercise useful in making their goals more evident to themselves and their expectations clearer to students. More importantly, faculty used their findings to set new learning goals, rethink their course materials, and continue reflecting on their teaching.

The key factor here is that faculty weren’t told what to assess, nor were they given a ready-made rubric to apply; their questions arose from their own curiosity and reflection, as prompted by the questions we posed for them and which they answered in writing. The sense-making meetings were structured by and around worksheets that each faculty member completed. Those are interesting documents because they tracked and revealed faculty thinking processes. This was by design: the questions to which faculty responded in each workshop overlapped, leading them to revisit what they most value in their courses for students, how best to elicit student success, and how to observe that success or failure. Both worksheets prompted them to anticipate how they would revise their goals for students based on what constructing the rubric had helped them see about what they value. In short, faculty learning about their own goals for students was iterative by nature. Thus, the process of identifying goals, crafting assignments to foster student learning, and collecting information about student success ended, even in the second workshop, in the presumption that it was necessary to start again with restated, reimagined goals.

Not surprisingly, in both the first and second sessions, faculty members’ worksheet responses alternated between overly general goals or grandiose claims about what “success” would look like and specific lists detailing what moves they could take that would be the building blocks of that success. For example, one public policy professor answered the (deliberately mixed, compound) question—“How will I know that learning outcomes have been achieved? How will these be assessed? How will I gather information about student learning?”—with only a forceful description of what high-quality work should display: “Students will build their own argument by fully engaging with relevant existing research and integrating it with their own reasoning.” In the drafted rubric, however, the professor anticipates what analytical and argumentative moves are necessary to build such a successful piece of writing.

Particularly helpful for many participants was the section of the worksheet that asked them to identify the “mechanisms of effect” likely to influence their students’ ability to learn. Mechanisms of effect include both “mediators” of learning—emotional, motivational, and cognitive qualities or dispositions, as well as prior knowledge—that would facilitate learning as well as “moderators”—other knowledge, qualities, or dispositions likely to impede learning in the given course. The worksheet required a detailed process of reflection, and faculty named mediators and moderators in each of five categories: prior knowledge, beliefs, emotional processes, cognitive processes, and motivational processes. Key here was the fine-grained approach that forced faculty to slow down and consider students as learners. This process enabled faculty to identify not merely potential problems but also potential student strengths as they affected learning in the teacher’s actual course. Thus, faculty were given a chance to think about precisely what cognitive processes, what kinds of reasoning and learning, students actually do in their courses. This enabled them to move more effectively into designing rubrics to guide themselves as they fashion assignments and measure student learning.

Conclusions

Our study confirms the importance of faculty defining their own goals, designing their own assessment strategies, and discussing their goals and findings with others. Our analysis suggests that, despite associated challenges, the process of creating reliable and informative rubrics—especially in a collaborative and multidisciplinary context—provides faculty with the self-assessment and metacognitive awareness crucial for student learning and for their own satisfaction as instructors. The workshop format gave faculty a space in which to take risks themselves—to be learners rather than experts. Articulating concerns about student learning and describing the epistemology of one’s discipline takes intellectual work. It is this kind of work, especially when done in dialogue with scholars across disciplines, that makes visible our own personal and disciplinary assumptions. When the particularities of our disciplines become evident by comparison, we become, collectively, more than inter- or multidisciplinary; we gain a metadisciplinary awareness that we can then share with our students. It surfaces in our assignments and in the ways we communicate our expectations to students, and it helps them develop their own metacognition, which we know to be important for learning and transfer.

In a research article published in 2014, the principal investigators at the three participating universities argued that when we seek to determine why assessment data are not being used for educational improvement, we need above all to rethink and broaden our notion of what “use” means.3 Among other things, they argued, individual faculty and their departments need a longer time window than current assessment-for-accreditation protocols allow to interpret assessment data, much less implement changes in how courses are taught. More importantly, “use” can and often does mean changing how faculty identify goals for students or craft and evaluate coursework and assignments to attain those goals. In short, conceptual change is a form of usefulness that goes unrecorded and undervalued in conventional assessment processes.4

In our research project with Duke faculty, we explored how working with writing fosters conceptual change in faculty. Grappling with student writing and planning rubrics to guide both students and faculty leads faculty to a greater meta-awareness of thinking processes in their disciplines, and thus to a greater ability to guide students through those processes. Engaging faculty in thoughtful practices surrounding the teaching of writing is thus one model for showing why conceptual change matters and how to effectuate it.

The research we conducted also created a protected space for faculty that is hard to come by, especially at a research university such as Duke. This space permitted attention to holistic questions about teaching and learning (rather than problem solving) and about the making of disciplinary knowledge, unfettered by institutional goals. It was a space we were willing to construct because it served one of the goals of the writing program: to support faculty across the university who teach writing-intensive courses. Our process of working with faculty in this research project was successful, in turn, because it mimicked what we have found successful in training new instructors in the writing program: the attention to identifying disciplinary values and translating those values into goals for students to reach through their writing. The workshops convened for the study mirrored the multidisciplinarity of our program, pushing participants to make explicit their own epistemological frameworks. But perhaps more importantly, we asked faculty to do their thinking through writing—reflecting not only what they would be asking of their students, but what is, after all, arguably the main mode of intellectual engagement and of “doing” scholarship in academia.

Our work with faculty on planning assessment in their courses and making meaning out of the information they glean has demonstrated to us how revolutionizing it is to put faculty at the center of this process. This means that to carry out assessment that succeeds in seeding improvement—assessment that closes the loop—we must democratize it and make it a more collaborative venture. Most important, it is clear to us now that assessment is potentially much more than a mode of accountability; if done correctly and collectively, it can be a powerful tool in our professional development as teachers and as intellectuals, a thinking process that has the potential to be attractive and productive, and not a burden, to faculty themselves.

NOTES

1. See Cynthia E. Coburn, Judith Toure, and Mika Yamashita, “Evidence, Interpretation and Persuasion: Instructional Decision Making in the District Central Office,” Teachers College Record 111, no. 4 (2009): 1115–61.

2. Robert J. Thompson Jr., A Study of the Use of General Education Assessment Findings for Educational Improvement, project funded by the Spencer Foundation for award period July 1, 2011, through June 30, 2014, identifying number: 201200046.

3. Jessica L. Jonson, Tim Guetterman, and Robert J. Thompson Jr., “An Integrated Model of Influence: Use of Assessment Data in Higher Education,” Research and Practice in Assessment 9 (Summer 2014): 18–30.

4. Jonson, Guetterman, and Thompson Jr., “An Integrated Model.”

To respond to this article, email liberaled@aacu.org with the authors’ names on the subject line.


KRISTEN NEUSCHEL is associate professor of history and director of the Language, Arts, and Media Program at Duke University. MÁRCIA REGO is associate professor of the practice and director for faculty development and assessment in the Thompson Writing Program at Duke University.

Previous Issues