Diversity and Democracy

Working across Institutions to Define and Assess Civic Learning

In December 2012, the Commonwealth of Massachusetts reaffirmed its historical commitment to civic engagement and learning with a report titled Renewing the Social Compact. Written by the commonwealth's newly formed Special Commission on Civic Engagement and Learning, the report supported the recent actions of the state's Board of Higher Education, which in September 2012 named civic education and engagement as the sixth goal of the state's Vision Project—an effort to improve student learning and achievement, college participation, degree production, and workforce development, with a special focus on college completion for low-income and racial and ethnic minority groups. For the state's higher education institutions, the Vision Project and the Renewing the Social Compact report served as catalysts for reexamining academic programs and campus goals related to civic engagement.

The renewed emphasis on civic learning validated an effort already in progress at Fitchburg State University and Mount Wachusett Community College (MWCC). Since March 2012, the two institutions have been collaborating through the Association of American Colleges and Universities' (AAC&U's) Quality Collaboratives (QC), a three-year project supported by Lumina Foundation and the William and Flora Hewlett Foundation. The QC project pairs dyads of two- and four-year institutions in nine states to develop recommended policies for assessing and reporting student achievement and related transfer designs, using both Lumina Foundation's Degree Qualifications Profile (DQP) and AAC&U's Liberal Education and America's Promise (LEAP) Essential Learning Outcomes as guides. The Fitchburg State–MWCC dyad has developed teams around four competencies: civic engagement, written communication, information literacy, and quantitative reasoning. The civic engagement team is facing one of the project's more challenging and critical tasks: reconciling two different institutional approaches to assessment within the broader context of a state-level commitment to civic education and engagement.

Institutional Contexts

Both Mount Wachusett Community College and Fitchburg State University entered the QC project with longstanding commitments to civic priorities.

For over a decade, MWCC has supported civic learning collaborations and community partnerships within the Gardner, Massachusetts, area. Through the Center for Civic Learning and Community Engagement (endowed in 2012), MWCC staff assist with program development and design, faculty and student support, strategic community collaborations, data collection and evaluation, and assessment of high-impact practices and programs. Service learning, one of the college's most treasured assets, occurs in nearly fifty courses, engaging more than forty instructors and over four hundred students each year. Some majors require service-learning internships, involving over one hundred students annually in hands-on experiences that benefit the community. Beginning in fall 2012, students majoring in general studies will enroll in a required service-learning capstone course. The college also participates with Fitchburg State in AmeriCorps Job Ready, engaging over ten thousand community members annually in job preparedness training and workshops, and in the United Way Youth Venture program, which involves over two thousand high school students per year in developing their own social entrepreneurial opportunities. The college is one of ten Massachusetts community colleges holding Carnegie classification in Curricular Engagement and Outreach Partnerships.

Fitchburg State University has also expressed its longstanding institutional commitment to civic education and engagement, including through its institutional mission of fostering students' civic and global responsibility. Identifying citizenship as one of five central learning objectives for all students, the campus liberal arts and sciences curriculum is designed to ensure that students will articulate the relationships among local, national, and global concerns and recognize opportunities to enact positive change. The Douglas and Isabelle Crocker Center for Civic Engagement, founded in 2007, has expanded the commitment to community by promoting engaged scholarship for students and faculty. Eight recent faculty-driven civic engagement projects have enlisted over one hundred students in community work, and over three hundred attendees have engaged in ten public workshops. The center has established resource-sharing and service-learning partnerships with organizations in areas from public safety to community outreach.

Developing a Work Plan

After joining the QC project in spring 2012, project directors from MWCC (Ruth Slotnick) and Fitchburg State (Christopher Cratsley) recruited and assembled four teams of faculty and staff leaders. Each team consisted of eight paid "assessment scholars" (three faculty members and one staff member from each institution), one of whom was named team leader. The teams focused on four areas—civic engagement, communication literacy, quantitative literacy, and information literacy—that represent priorities shared between the two institutions and reflected in both the DQP and the LEAP Essential Learning Outcomes. The project goal was to develop and implement assessment plans in these areas and generate recommendations for using the resulting data to inform transfer decisions.

In June 2012, project participants gathered for a two-day institute where they analyzed each institution's rubrics, aligned these rubrics with LEAP and DQP learning outcomes, and developed a work plan for the upcoming year. At this meeting, the civic engagement team received copies of the DQP, AAC&U's Civic Engagement VALUE (Valid Assessment of Learning in Undergraduate Education) Rubric, the MWCC Understanding Self Survey, the Fitchburg State Citizenship through Critical Analysis of Events Rubric, and a blank timeline for the upcoming year. The teams established a Blackboard site with wikis and blogs, scheduled monthly meetings beginning in September, and established work plans to collect and assess samples of student work.

In October 2012, the dyad held an all-project workshop where Susan Albertine, vice president for diversity, equity, and student success at AAC&U, described national efforts to improve outcomes-based assessment and to emphasize the critical role faculty play in developing a local philosophy and methodology to assess student learning in relation to transfer. The workshop provided an opportunity for Albertine and project directors Slotnick and Cratsley to help the assessment teams refine their rubrics, identify assignments appropriate for assessing student work, and address concerns about norming rubric scoring between assessors.

Artifacts and Scoring

Throughout the fall semester, the civic engagement team modified the Civic Engagement VALUE Rubric by reorganizing and revising some of the criteria. The team identified types of student artifacts that were potential sources of evidence for civic engagement and ruled out types of work that did not provide evidence related to rubric criteria. Having established these parameters, the team agreed to proceed with norming and scoring student work in the hopes that the data generated would prove illuminating.

The teams gathered in a second all-project meeting on January 24, 2013. Prior to this meeting, the project directors uploaded one set of artifacts to the Tk20 online assessment system for each team's review. Addressing the full group, each team leader presented an overview of the team's work since the fall all-project meeting and shared the selection of artifacts to be assessed. After the project directors explained how to use the Tk20 system, the teams logged on, reviewed the artifacts, completed any additional norming, and discussed their plans for assessing the artifacts.

Throughout the month of February, the civic engagement team used the Tk20 system to assess images MWCC students had created to represent what the United States Constitution means to them (figure 1). The team also assessed artifacts from Fitchburg State, including reflective essays written by human services students following a semester-long field experience, research papers from the same students on a contemporary issue in human services, and oral history projects created by students who had interviewed community members about an issue in the Fitchburg community.dd16_3_cratsley.jpg

The assessment scholars attempted to score each sample of student work using the modified Civic Engagement VALUE Rubric they had developed in the fall (to download the original rubric, visit www.aacu.org/value/). In addition to modifying the rubric criteria, they had revised the rubric to rename the four levels of achievement—mastery, proficiency, emerging, and foundation—and to add two columns: one representing no demonstration of each criterion, and one indicating that the assignment was not appropriate for assessment using a particular criterion. The most telling finding was arguably related to this final category, as a high percentage of scorers rated many criteria as not applicable to the assignment (see table 1).

Table 1. Percentage (and Number) Indicating "No Response Applicable" for Student Artifacts

Rubric Criteria Images Representing US Constitution Reflection on Field Experience Paper on Human Services Issue Report on Fitchburg Oral History Project
Diversity of Communities and Cultures
52.6% (20)
44.4% (8)
32.3% (11)
33.3% (2)
Analysis of Knowledge
34.2% (13)
22.2% (4)
17.6% (6)
0%
Civic Identity and Reflection
65.8% (25)
22.2% (4)
64.7% (22)
33.3% (2)
Civic Communication
47.4% (18)
38.9% (7)
23.5% (8)
16.7% (1)
Involvement in the Community Infrastructure
76.3% (29)
44.4% (8)
70.6% (24)
16.7% (1)
Civic Action and Commitment
76.3% (29)
27.8% (5)
70.6% (24)
50% (3)

These results, which the project directors aggregated in Tk20 and compiled for the team, suggest that no one type of assignment sampled effectively captured all rubric criteria. Evidence of Analysis of Knowledge and Civic Communication appeared to be the easiest to find across all assignments. Self-reflection assignments showed the most potential for effectively capturing elements like Civic Identity and Reflection as well as Civic Action and Commitment, but provided insight into the Diversity of Communities and Cultures or Involvement in the Community Infrastructure only in certain cases. In these areas, analytical papers (such as the oral history project reports and human services issues papers) provided additional information, particularly when they focused on local community issues. Finally, images presented without reflection or analysis seem to be poor sources of data for assessing the criteria.

Lessons Learned

In March 2013, the civic engagement team met to review the data and discuss the assessment process. Project directors Cratsley and Slotnick used this meeting to collect qualitative data on faculty and staff perspectives. The team identified a number of important outcomes and established next steps for the project.

The civic engagement team described challenges related to reconciling diverse views of civic education, as represented in institutional assessment approaches at Fitchburg State and MWCC, in the outcomes identified by the Commonwealth of Massachusetts, in the DQP and the LEAP Essential Learning Outcomes, and in the VALUE rubrics. For example, Fitchburg State's definition of citizenship requires students to recognize opportunities to enact change but does not necessarily require them to act. The institution's Citizenship through Critical Analysis of Events rubric thus measures only their ability to reflect on diversity and make connections between academic knowledge, civic engagement, and their own civic lives. In contrast, the MWCC Understanding Self rubric asks students to rate their own level of civic engagement in terms of volunteer work and political activity, as well as to rate their civic knowledge, but falls short of measuring civic learning.

Assessment scholars working with the modified Civic Engagement VALUE Rubric found that the student artifacts they collected, while often providing evidence of civic knowledge and civic cognitive skills, were less consistent in providing evidence of civic participatory skills or civic dispositions. They also identified a need to create better alignment between the modified rubric and various civic frameworks. The DQP uses the broad term "civic learning" to describe separate civic outcomes (for example, the ability to explain and describe in ways that reflect civic knowledge and cognitive skills and the ability to provide evidence of active, engaged collaborations that reflect participatory skills). While the original LEAP Essential Learning Outcome—Civic Knowledge and Engagement (Local and Global)—reflects a similar balance between knowledge and active engagement, the Civic Engagement VALUE Rubric focuses heavily on assessing participatory skills and dispositions. Therefore, the team's goal moving forward is to further revise the rubric to more clearly distinguish between civic knowledge, cognitive and participatory skills, and dispositions, as well as to provide guidelines and sample assignments that better capture the range of civic learning competencies.

Next Steps

In the grant's final year (2013–14), Fitchburg State University and Mount Wachusett Community College will focus on encouraging cross-disciplinary discussions in high-transfer areas such as business, early childhood education, and liberal arts and sciences. The teams will also create assignment prompts that mirror their adapted VALUE rubrics, with the eventual goal of embedding assignments and curriculum mapping within each institution for program-level benchmarking and cross-institutional comparisons. The civic engagement team's challenge will be reconciling the DQP's focus on civic learning with that of the Civic Engagement VALUE Rubric, which stresses participatory skills and civic dispositions.

Reference

Special Commission on Civic Engagement and Learning. 2012. Renewing the Social Compact. Boston, MA: Department of Higher Education.


Christopher Cratsley is director of assessment at Fitchburg State University, and Ruth C. Slotnick is director of articulation and learning assessment at Mount Wachusett Community College.

Previous Issues