Assessing Signature Work

The complexity and varied outcomes associated with signature experiences have presented many assessment challenges for participating institutions in the LEAP Challenge: Engaging in Capstones and Signature Work consortium, led by the Association of American Colleges and Universities (AAC&U). Each college and university evaluated outcomes by developing customized approaches that were designed for both the type of experience and the specific institutional mission or culture. Higher education’s emphasis on signature work is a recent development, and most of the consortium schools believe their institution has yet to make significant progress in assessing such work. In this article, we will examine some of the early efforts to assess signature work with a particular focus on practices used at individual colleges to assess students’ reflections, as well as student and institutional preparedness for signature work. We will also discuss some of the challenges and issues faced as work on this front has moved forward.

Using Existing Institutional Data

Assessing undergraduate signature work may seem daunting at first. After all, institutions often vary in how they define and implement signature work within their curricula. These differences may lead administrators and professors to the conclusion that they must create new assessment mechanisms. Creating new assessments may be necessary, depending on the unique circumstances of each institution. However, it is often possible to assess signature work in meaningful ways using existing institutional assessment practices. In this section, we illustrate how our institutions have applied three distinct, existing assessment tools to help evaluate undergraduate signature work. In doing so, we hope that other institutions may use these examples to implement their own signature work assessment measures.

Among the institutions participating in AAC&U’s capstone and signature work project, the most common assessment tool was an institutional or departmental senior survey. Some institutions also employed an alumni survey or a general survey of current students. Through these assessment mechanisms, students self-report learning outcomes or rate their preparedness for signature work. Institutional research offices often administer senior surveys to measure a variety of outcomes within the curriculum, cocurriculum, or other aspects of campus life.

There are several benefits associated with the use of a senior survey. First, some institutions require students to complete a survey before graduation, thus ensuring a high response rate and robust data across academic disciplines. Second, given that many institutions already have a plan in place to administer a senior survey, adding targeted questions regarding signature work to an existing survey requires a modest amount of work on the part of administrators, faculty, and students.

The National Survey of Student Engagement (NSSE) is another survey assessment option available to many institutions. NSSE already includes several questions that can measure involvement in signature work and high-impact educational practices, depending on how signature work is defined at a given institution. There are several benefits of employing NSSE data in the assessment of signature work. First, NSSE includes questions regarding capstone courses and other high-impact practices that may constitute signature work. Given the breadth of NSSE questions, administrators and faculty can dig deep into NSSE data to better understand how signature work influences reflective and integrated learning, faculty-student interactions, student perceptions of their institution, and other important indicators. Second, NSSE provides comparative summary data for peer institutions. While these data may be somewhat basic in their scope, they nevertheless provide insight as to whether one’s home institution is excelling or falling behind relative to other colleges and universities. Such information may help to inform changes to the curriculum and signature work. Finally, NSSE collects data from students in different years, allowing for assessment of effective scaffolding or incremental changes in student experiences.

While assessments like NSSE and senior surveys are primarily concerned with outcomes, syllabus audits can help to assess inputs to determine if course assignments and activities meet an institution’s definition of signature work. Academic oversight committees can review course syllabi from time to time, and signature work syllabus audits could be built into the review process to ensure that curricular activities align with institutional definitions of signature work. This method provides another means of quality control to ensure that there is consistency across all offerings that are classified as signature work.

Admittedly, none of these assessment methods are perfect. For example, all of them are indirect approaches; they do not involve direct evaluation of students’ signature work artifacts or products. However, each provides a mechanism for assessing signature work using existing assessment frameworks at a variety of institutions. Furthermore, if an institution already makes use of one or more of these assessment mechanisms, they can measure signature work inputs and outcomes in an efficient manner.

What We Are Learning from Early Assessment Data

Students’ Readiness for Signature Experiences
Many schools involved in the project reported concerns about preparation for signature work. To date, none of the schools have developed a systematic, centralized way to assess student readiness for this type of complex learning, and most assessment of preparation is still in the early stages. In general, there is a paucity of research on undergraduates’ readiness for independent or self-directed work. The extant studies tend to be in applied fields like engineering or health (e.g., Sahoo 2016; Long and Afgyekum 1983), with little written on the liberal arts. Given the financial and human resources that increasingly are being directed toward signature work experiences in American liberal education, assessing student readiness and adapting projects to ensure they are academically appropriate and suited to student skills are critical. The four schools whose data are reported below offer signature experiences for all students and have decentralized approaches to both overseeing and evaluating this kind of work. Other schools that have more centralized approaches are in the initial phases (see the article by Budwig, Ratliff-Crain, and Reder, in this issue of Peer Review).

Two campuses participating in the project have ensured adequate preparation for signature work through curricular oversight at the department level. Augustana College, which requires Senior Inquiry (SI) experiences for all students, reviews the design of departmental curricula for coherence and sensibility. This process ensures intentional, developmental, and coherent design of the major, including the capstone, which in turn should adequately prepare students for the work. However, Augustana has not engaged in assessment of individual student readiness for SI.

Similarly, Elizabethtown College has all students participate in at least two signature learning experiences (SLEs)—internships, study abroad/away, research with faculty, community-based learning, and capstone courses and experiences. Assessment of the SLEs is decentralized, with preparedness assessed at the department level. With a general education revision underway, Elizabethtown is emphasizing the acquisition of skills that are most salient for SLEs, such as intercultural competence and problem-solving, as they consider scaffolding within majors.

Nebraska Wesleyan University and Bates College assess student performance in capstone experiences as one indicator of the students’ preparation. Nebraska Wesleyan’s first cohort of students completed the four-year Archway Curriculum in May 2018. Like Augustana College, Nebraska Wesleyan has relied on a well-designed curriculum to ensure preparedness. The Archway Curriculum emphasizes skill scaffolding, integrative learning, and experiential pursuits, and it was designed to ensure better preparation, autonomy, and academic self-awareness as students approach the capstone. At Nebraska Wesleyan, assessment of student learning in the capstone will provide insight into the effectiveness of the new curriculum in providing adequate preparation.

At Bates College, where there are a variety of signature experiences available that are largely decentralized, the assessment of student readiness has depended on the point of entry. For those doing community-engaged work, instructors and staff work with students to ensure that projects are appropriate and that students have the requisite skills to develop and support a community collaboration. A post-experience survey includes items on preparation (e.g., “I was adequately oriented to or prepared for community-engaged work.”).

In the context of senior thesis work, Bates College has collected institution-level data on thesis outcomes over the last several years, including narrative and survey data about student learning. The data suggest that not all students are well-prepared for thesis work. Figure 1 summarizes faculty assessments of student readiness for thesis work on a variety of dimensions. The challenges appear to be in the areas of development of topics/research ideas and insight into personal strengths and weaknesses.

Similarly, according to Bates’s senior survey data, about a quarter of students report that they are not well-prepared for a thesis. On average, faculty rate student preparation lower than students rate themselves (see figure 2); these data suggest that there is work to be done to provide students with the skills to successfully complete these experiences.

 

Figure 1. Bates College Faculty Ratings of Student Thesis Skills

Fig1_AssessingSignWork_Web.jpg

Figure 2. Bates College Survey Ratings of Thesis Preparation (1–5, with 5=strongly agree or definitely yes)

Fig2_AssessingSignWork_Web.jpg

Institutional Readiness for Signature Work

One key question that came up repeatedly in our consortium meetings had to do with faculty and staff workload and institutional readiness to scale signature and capstone work. While part of the issue pertaining to workload had to do with the rewards for and accounting of this work, another challenge is related to professional development and the extensive mentoring involved in signature projects. There has been a significant amount published on the importance of mentors and their impact on student learning. For instance, in studies stemming from the Survey of Undergraduate Research Experience (SURE), learning has been directly correlated with students’ evaluation of their mentor, and mentoring on a long-term project is one of the key indicators of student success after college, as reported in the 2014 Gallup Purdue Index Report (see Lopatto 2010; Nicholson et al. 2017).

While it has been noted that mentoring plays a formative role in student success, studies of undergraduate research note that students typically report more than one mentor for their work, and evidence suggests that different mentors in a given network play different roles (Bradley et al. 2017). Early on, as capstones and signature work were being scaled at Clark University, Michelle Bata, associate dean and LEEP (Liberal Education and Effective Practice) Center director, worked as part of the consortium of schools publishing on mentoring networks. Examining survey data from an exploratory study of 146 undergraduate researchers at several different schools, they found that co-mentored and multi-mentored projects were noted by many of the students. This was especially true for interdisciplinary projects. Furthermore, this study noted that although faculty members were most frequently identified as mentors, others were as well (e.g., peers, graduate students, postdocs). And it was reported that students often highlighted the distinct roles played by different sorts of mentors (e.g., providing disciplinary expertise, logistical support, emotional support).

Such exploratory work may help address issues raised by faculty as capstone requirements are considered. For instance, an analysis suggests that Clark students typically indicate that they had more mentors than students at other schools and 68 percent reported at least two faculty mentors, something that could prove difficult to sustain with scaling. Most important was the finding that students reported turning to each mentor at Clark for a variety of kinds of support, whereas at some other schools, students reported more differentiation, as mentor networks distributed mentoring functions to ensure student success. Early evaluation can help institutions design professional development programs that target evidence-based best practices for co-mentoring. This kind of iterative assessment is essential to the success of capstones and signature work.

Direct Assessment: Assessing Student Reflection

At this stage in our institutions’ signature work implementation, assessment data are still provisional. However, many of us have noticed gaps between goals and practices in the pedagogy of meaningful, iterative reflection. Some types of signature work (internships, study away, practicums) are more likely to incorporate midsemester self-assessment or iterative journaling; other types (independent research, theses) are less likely to include such reflective elements. Nonetheless, all students benefit from opportunities to reflect at multiple times in their signature projects and processes (e.g., see Peden, Reed, and Wolfe 2017).

Augustana College conducted a collective syllabus review for all Senior Inquiry courses to determine how consistently these courses addressed different signature work criteria and goals. There was strong evidence that the Senior Inquiry courses across campus met four of the signature work criteria, with a notable gap in areas related to reflection on learning (Peden 2015). Other campuses in the project had similar challenges with reflective activities, either anecdotally or in their formal assessment results.

At Nebraska Wesleyan University, many departments were unsure how best to include reflection in their capstone projects, even as students in a few departments had multiple opportunities for reflection, both vocational and self-evaluative. Bringing departments together to discuss signature work opened conversations among departments about their different capstones, which previously had been implemented in isolation from one another. These conversations both revealed disparities and provided mechanisms for departments to learn from their peers. Moreover, as a result of this process, Nebraska Wesleyan faculty revised the university’s definition of capstones to include the element of reflection explicitly.

Elizabethtown College has assessed the extent to which students who completed different signature learning experiences reported accomplishing different NSSE indicators. The NSSE indicators that most closely track students’ reflection ask students whether they “examined the strengths and weaknesses of your own views on a topic or issue” or “learned something that changed the way you understand an issue or concept.” Analysis of NSSE scores for juniors and seniors at Elizabethtown showed a positive correlation between community-based learning and study abroad for both of the reflection-related NSSE indicators (McClellan and Kopko 2015). It is worth noting that NSSE does not measure any other potential types of reflection (e.g., vocational exploration, strengths analysis, evaluation of cumulative academic or professional growth) that may occur within SLEs.

Faculty and staff who support signature work may not all prioritize reflection; the need for students to gain mastery of disciplinary content or professional skills can easily obscure the need to develop metacognition or self-knowledge. Moreover, many faculty members feel uncomfortable eliciting or evaluating student reflections. The uneven success in implementing reflection in capstones and signature work will likely need to be addressed prior to and within the capstones and signature work experiences. As Mary and Michael Ryan (2013) point out, work at different levels of sophistication can all be called reflection, so faculty members must make a concerted effort to scaffold the teaching of reflection if students are to progress from simple narration to a deeper and more meaningful analysis and synthesis. Among the schools involved in the project, there were no reports of using standardized instruments like the brief questionnaire that Kember et al. (2000) describe in their article on measuring the level of reflective thinking.

Institutions that make consistent use of eportfolios are well-positioned to make the connection between reflection and assessment across a student’s academic career in order to improve both individual and institutional outcomes (Peden 2015). Programs that incorporate students’ scaffolded and intentional use of eportfolios at multiple times and in multiple areas can develop students’ capacity for meaningful reflection within and about their academic and professional trajectories, reflection that takes into account both past work and future directions (Treuer 2014). For reflection to become a high-impact capstone practice, students must have carefully calibrated experiences to build this skill throughout their academic programs, and faculty must have scaffolded opportunities to practice mentoring student reflection.

Lessons Learned: Challenges Assessing Signature Experiences

Signature work not only provides a powerful pedagogy for student learning, but it also offers opportunities for students to demonstrate what they have learned and to apply their learning to unscripted situations. Thus, educational institutions and programs can use assessment techniques to ascertain student proficiency in applied skills such as integrative thinking and problem-solving, as well as in common outcomes such as written communication and analysis. To the extent that students pursue their own inquiries into real-world problems, signature work can also provide insight into how students develop personal and social responsibility.

In this issue of Peer Review, Egan, Kneas, and Reder note that—although signature work may require certain kinds of knowledge, dispositions, and skills—institutions and programs will vary in the outcomes they expect students to master. That is, what an educational organization or unit determines to be signature work will depend on that entity’s mission and culture, which in turn will have implications for assessment. One university may place a high value on global learning, while another will emphasize ethical training, perhaps from a particular value or faith-based tradition. An engineering or business program may expect students to apply their expertise in enterprises that create social value. Assessment of signature work in such contexts should tap into those intended outcomes.

The way in which institutions structure signature work and signature assignments also complicates assessment efforts. In this issue of Peer Review, Hayden-Roy et al. note that many of our schools are making signature work a culminating capstone experience in the major. One challenge has been to ensure that general knowledge, skills, and dispositions are assessed, not only knowledge, skills, and dispositions pertaining to capping off the specific learning outcomes of the major. In addition, while some colleges require a single project such as a senior thesis or an eportfolio, others permit students to choose the signature practice that best fits their academic and career plans. When signature work is defined to include high-impact practices, such as supervised research or community-based learning, students may complete multiple signature experiences at different points of the college career. This means assessment coordinators must make decisions about what kinds of signature work should be assessed, when, and by what means. These choices should be part of a larger campus conversation about how and toward what ends assessment results will be used.

Conclusion

Assessment of signature learning is still in the formative stages. Several institutions report using standardized surveys like NSSE to inform them about aspects of signature work, but they report challenges that include the variable definitions of signature experiences and the need to assess complex constructs like integrative learning or reflection. ePortfolios hold promise for both promoting reflection and providing material for assessment. Finally, participating schools expressed concerns about student preparation for signature work, given that not all students may be ready for original, self-authored projects. Early data on student and institutional preparedness have played a formative role in how schools are scaling this work.

 

References

Bradley, Evan D., Michelle Bata, Heather M. Fitz Gibbon, Caroline J. Ketcham, Brittany A. Nicholson, and Meagen Pollock. 2017. “The Structure of Mentoring in Undergraduate Research: Multi-Mentor Models.” Scholarship and Practice of Undergraduate Research 1 (2): 35–42.

Gallup. 2014. “Life in College Matters for Life After College: New Gallup-Purdue Study Looks at Links among College, Work, and Well-Being.” Retrieved from http://www.gallup.com/poll/168848/life-college-matters-life-college.aspx.

Kember, David, Doris Y. P. Leung, Alice Jones, Alice Yuen Loke, Jan McKay, Kit Sinclair, Harrison Tse, Celia Webb, Frances Kam Yuet Wong, Marian Wong, and Ella Yeung. 2000. “Development of a Questionnaire to Measure the Level of Reflective Thinking.” Assessment and Evaluation in Higher Education 25 (4): 381–95.

Long, Huey B., and Stephen K. Agyekum. 1983. “Guglielmino’s Self-Directed Learning Readiness Scale: A Validation Study.” Higher Education 12 (1): 77–87.

Lopatto, David. 2010. “Undergraduate Research as a High-Impact Student Experience.” Peer Review 12 (2): 27–30.

McClellan, Fletcher, and Kyle Kopko. 2015. “High Impact Practices in Political Science Major: Effects on Student Learning.” Conference paper, Annual Meeting of the American Political Science Association, September, San Francisco, CA.

Nicholson, Brittany A., Meagen Pollock, Caroline J. Ketcham, Heather M. Fitz Gibbon, Evan D. Bradley, and Michelle Bata. 2017. “Beyond the Mentor-Mentee Model: A Case for Multi-Mentoring in Undergraduate Research.” Perspectives on Undergraduate Research and Mentoring 5 (2).

Peden, Wilson. 2015. “Signature Work: A Survey of Current Practices.” Liberal Education 101 (1/2): 22–29.

Peden, Wilson, Sally Reed, and Kathy Wolfe. 2017. Rising to the LEAP Challenge: Case Studies of Integrative Pathways to Student Signature Work. Washington, DC: Association of American Colleges and Universities.

Ryan, Mary E., and Michael Ryan. 2013. “Theorising a Model for Teaching and Assessing Reflective Learning in Higher Education.” Higher Education Research and Development 32 (2): 244–57.

Sahoo, Soumendra. 2016. “Finding Self-Directed Learning Readiness and Fostering Self-Directed Learning through Weekly Assessment of Self-Directed Learning Topics during Undergraduate Clinical Training in Ophthalmology.” International Journal of Applied and Basic Medical Research 6 (3): 166–69.


Kathryn Low, Interim Dean of the Faculty and Vice President of Academic Affairs, Bates College; Sarah A. Kelen, Dean of the Colleges, Nebraska Wesleyan University; Kyle Kopko, Assistant Dean for Academic Achievement and Engagement, Associate Professor of Political Science, Elizabethtown College; Fletcher McClellan, Professor of Political Science, Elizabethtown College; Michelle Bata, Associate Dean and Director of the LEEP Center, Clark University

Select any filter and click on Apply to see results