Assessing Quality and Equity: Observations about the State of Signature Work

Learning experiences that require grappling with complex problems and projects that are important to students and to society are worth guaranteeing for all students. Bringing students through a learning experience that demands holistic integration is exactly the kind of practice all graduates need. The articles featured in this issue of Peer Review, which were written by faculty, administrators, and staff who participated in the capstones and signature work project from the Association of American Colleges and Universities (AAC&U), demonstrate first hand what signature work entails for students and faculty and illustrate how colleges and universities can define, organize, scale, and assess this work.

Signature work’s emphasis on providing students a substantial occasion to apply and integrate learning to a pressing issue in the larger world suggests a conceptual relationship to the popular high-impact practices (HIPs)—active learning experiences that promote deep learning and are associated with desirable student outcomes, including improved student retention, grades, and graduation rates. Signature work is clearly an extension of the well-established HIP—the capstone—which is a culminating experience that caps off the integration of educational experiences leading to a demonstration of mastery. The rising popularity of HIPs and the overlap between capstones and signature work suggest the time is right to reflect on these parallel activities. In this article, I offer some observations about the current state of signature work and their ancestor, capstones, and then discuss the extent to which institutions in the LEAP Challenge: Engaging in Capstones and Signature Work consortium, supported by a grant from the Arthur Vining Davis Foundations, are addressing the challenges of this important work, including a focus on the challenges of assessment.

Taking Stock of Capstones

Several years ago, in a Peer Review article titled “Taking Stock of Capstones and Integrative Learning” (Kinzie 2013), I summarized extant research on the nature, scope, and outcomes of culminating experiences and discussed integrative learning in capstones. At that time, there were some clear patterns. Students at smaller-enrollment, baccalaureate liberal arts, and private institutions were more likely to have done a capstone; the most common form of capstone was a major paper, project, or thesis; and most capstones were completed in the context of the major. As reported then, and affirmed by Kuh and O’Donnell (2013), capstone participation is positively associated with a range of educationally effective practices, in particular with greater gains in measures of reflective and integrative learning. I concluded that capstones are educationally rich, potentially integrative experiences, but also raised concerns that capstones that are very focused in the discipline may not contribute to desired outcomes including integration of ideas and perspectives across fields or applying learning to a wider context. I also asserted the need for institutions to be more intentional about fostering the level of integration that educators envision.

Revisiting the topic of capstones recently, I discovered that most of the earlier findings still hold true. National Survey of Student Engagement (NSSE) 2017 results show that about 44 percent of seniors have completed or are in the process of completing a capstone, which is consistent for the last five years. Students at baccalaureate and private institutions continue to be more likely to participate in a capstone. Corresponding with the racial/ethnic demographics at these institution types, white students participate in capstones more than students in all other racial/ethnic groups, and first-generation students have lower levels of participation than their non-first-generation peers. Updated analyses of the relationship between capstones and all ten NSSE Engagement Indicators (http://nsse.indiana.edu/html/engagement_indicators.cfm) and perceived gains and satisfaction are significant and positive. The positive association between participating in a capstone and measures of integrative learning, such as combining ideas from different courses, connecting learning to societal problems, and including diverse perspectives in assignments, is a particularly important indicator of the power of capstones and, by extension, signature work. These findings affirm that capstones are an educationally beneficial educational practice; however, they also show that participation is unequal.

Notably, several challenges I originally identified for integrative learning and capstones to move forward—including the need for student reflection on their learning, scaffolded experiences that build to the capstone, and capstones that foster coherence and integration—have been addressed by institutions in the capstones and signature work project. For example, the need for students to reflect on their learning, including for metacognition and integration, is demonstrated in capstones and signature work projects that specify reflection as pedagogy and that include reflection essays as part of an eportfolio. It is also demonstrated in required reflective components that ask students “why the project is meaningful and impactful,” “to reflect on their own development,” or more broadly “to reflect on the integration of their overall academic experience.” Connecting guided reflection to signature work helps students achieve learning goals.

Consortium institutions also addressed my call for greater scaffolding by designing experiences throughout the curriculum that build skills necessary for capstones and signature work including (1) in general education and introductory courses in the major, (2) via a connected curriculum that leads students down an integrated path, and (3) in a prerequisite course on complex problems that leads to the capstone. Finally, building on the value of having multiple HIPs, several consortium institutions explicitly connected other HIPs—including undergraduate research, service learning, and internships—to signature work experiences. Importantly, the experiences of the institutions in the consortium offer a critical reminder that the quality and depth of student learning is very much dependent on ensuring that students can practice complex work through carefully crafted, outcomes-based, scaffolded assignments and projects.

Although HIPs have been lauded for the beneficial effects they can produce for historically underrepresented students (Kuh 2008), they have also been critiqued for uneven levels of participation across racial/ethnic groups and for their association with deficit-minded campus practices that create barriers to equitable participation (Finley and McNair 2013). Fortunately, the scalability of signature work signals some hope for assuring access for underrepresented students. On another hopeful note for inclusive excellence, signature work’s focus on a problem or project important to the student can help validate students as “knowers” and honor learning that is relevant to students’ lives and experiences.

Assessing Signature Work

The institutions in the capstones and signature work project made significant progress to define, organize, and scale signature learning. However, by their own admission, most institutions made only modest headway on assessing signature work. At least one institution in the consortium demonstrated an important first step toward strengthening the assessment dimension of capstones and signature work by conducting an inventory of capstone examples, practices, and outcomes. At another institution, an in-depth assessment of the key qualities of signature work revealed unevenness in the embodiment of signature work, providing valuable information about where improvements were needed. Some institutions also used surveys to assess students’ preparation for signature work as well as their perceived learning gains. These assessment activities are all good first steps.

Much like capstones, which Banta and Palomba (2014) asserted were rich with direct information about student learning, signature work can be valuable for the assessment of both individual student learning outcomes and program effectiveness. Basic assessment activities toward these goals could include simple inventories counting courses with signature work, mapping signature work across the curriculum and in majors, and examining the equity of student participation in signature work. Syllabi audits could provide evidence of the alignment of signature work with intended outcomes. Signature work products could also be reviewed to classify and study the range of problems and projects that students chose to address. NSSE data could provide institutions a sense of the impact of signature work. For example, institutions with NSSE data could identify relevant items, such as reflective and integrative learning and solving complex problems, and compare results prior to implementing signature work and after implementation, or compare results for students who experienced signature work with those who did not, including whether differences exist by race/ethnicity or first-generation status.

More in-depth assessment could focus on assessing individual student learning in terms of integrative learning and the application of learning. For example, the extent to which signature work fostered students’ pursuit of important, complex problems could be assessed by categorizing and assessing the complexity of problems and projects, and by asking students about the extent to which the work empowered them to pursue projects that were important to them. Student reflection exercises could also be reviewed for the source and depth of student agency in defining an important problem and learning gains. Assessments could include indirect methods such as course evaluations and student surveys, while rubrics could be used as a direct approach to assess complex problem-solving, integrative learning, and application and synthesis demonstrated in signature work products. The assessment of signature work can provide institutions and programs with evidence of how well students are prepared for twenty-first-century learning needs.

Conclusion

Signature work is an inviting educational concept that holds promise for ensuring more students have learning experiences that prepare them for work and life beyond college. Concerted efforts to assess signature work would provide needed evidence about the extent to which the objectives of integration and application to complex problems are being achieved, as well as evidence that topics important to the student and to society are being pursued. Much like capstones, which have grown in popularity as vehicles for assessing student learning and program quality, signature work products can become valued, authentic embedded-assessment methods to assess educational effectiveness and students’ capacity to engage in a world rife with messy, unscripted problems.

 

References

Banta, Trudy W., and Catherine A. Palomba. 2014. Assessment Essentials: Planning, Implementing and Improving Assessment in Higher Education. 2nd ed. San Francisco, CA: Jossey-Bass.

Finley, Ashley, and Tia McNair. 2013. Assessing Underserved Students’ Engagement in High-Impact Practices. Washington, DC: Association of American Colleges and Universities.

Kinzie, Jillian. 2013. “Taking Stock of Capstones and Integrative Learning.” Peer Review 15 (4): 27–30.

Kuh, George D. 2008. High-Impact Educational Practices: What They Are, Who Has Access to Them, and Why They Matter. Washington DC: Association of American Colleges and Universities.

Kuh, George D., and Ken O’Donnell. 2013. Ensuring Quality & Taking High-Impact Practices to Scale. Washington, DC: Association of American Colleges and Universities.


Jillian Kinzie, Associate Director of the Indiana University Center for Postsecondary Research and the National Survey of Student Engagement Institute

Select any filter and click on Apply to see results