Diversity and Democracy

Assessing Diversity, Global, and Civic Learning: A Means to Change in Higher Education

Although pedagogies and programs focused on diversity, global, and civic learning are not new in higher education, their salience is increasing, and they are likely to continue growing and becoming more coordinated and integrated as the academy faces unprecedented demographic and social change. This issue of Diversity & Democracy adds an important perspective to these developments. Assessment of student outcomes (that is, knowledge, skills, and attitudes) and related outcomes (for example, community outcomes and campus climate) in these three domains—within and across courses, programs, and institutions—poses both daunting challenges and intriguing possibilities. In this essay, we suggest questions for faculty and administrators to pose in considering approaches to assessment, and we explore the role of assessment as a catalyst for institutional transformation.

Established Assessment Issues

Assessment in higher education has grown significantly in recent decades. Institutions are allocating more resources, adding professional staff, and enhancing techniques; in addition, accrediting bodies as well as funders and the public are calling for both improvement and accountability. These developments have helped provide evidence-based feedback to improve teaching and learning and advance program and institutional goals, but they have also been marked by underlying tensions. Reflecting on four common points of tension, we offer questions for practitioners to pose of any assessment strategy.

Is the assessment based on self-reports of learning or direct, authentic evidence of learning? Self-reports can provide insights into learners’ understandings of the learning process, including the presumed causes of outcomes, attitudes toward learning, particular motivations, and behavioral intentions. In addition, self-reports can be used in combination with direct evidence of learning. However, self-reports are known to be inaccurate measures of learning outcomes (i.e., knowledge, skills) (Dunning, Heath, and Suls 2004). The complexities of learning in the diversity, global, and civic domains may make self-reports particularly susceptible to response shift bias (Howard and Daily 1979), in which learners overestimate their knowledge and skills on pretests, leading to distorted pre-post comparisons. A pre-post-then design, in which students retrospectively assess their competency at the time of the pre-test—“back then”—may be a useful modification when using self-reports for assessment. In contrast, products (such as essays, project reports, presentations, and portfolios of written and audiovisual artifacts) or demonstrations of learning can provide authentic evidence of knowledge and skills in the diversity, global, and civic domains. For example, student essays that examine the intertwining political, cultural, social, historical, economic, and ecological dimensions of an international policy, when evaluated with a rubric from any of the three domains, can provide authentic evidence of knowledge or skills. Partners in the learning process can also assist in assessing students’ learning, as when a community partner evaluates students’ communication skills or sensitivities to multiple perspectives within an organization.

Is the assessment both practical and meaningful? Although self-reports are relatively inexpensive and efficient, approaches that focus on assessing products or behaviors typically involve greater investments in time, professional staff, and course management systems. Gathering authentic evidence of learning may, therefore, pose a variety of costs, especially if infrastructure and human resources are required. Participants at all levels of the assessment process need to consider costs and benefits in light of the short- and long-term purposes and outcomes of the process. Professional development that helps educators integrate diversity, global, and civic learning with other learning goals and embeds associated assessment in courses may enhance both efficiency and effectiveness over time.

Do the assessment procedures support multiple levels of analysis, and what is distinctive at each level? Robust assessment can provide information that is relevant at multiple levels: individual student outcomes (e.g., competencies on course learning goals), metacourse analysis (e.g., cumulative learning across courses), institutional analysis (e.g., information required for accreditation), and cross-institutional comparisons (e.g., data for benchmarking or rankings). For example, through the application of common rubrics to problem-solving narratives assigned in first-year and capstone courses, an institution can gather evidence—useful at many levels—about how well students understand the sources and significance of cultural and political differences or about their level of critical thinking skills. Integrated frameworks, such as Lumina Foundation’s Degree Qualifications Profile, can help structure assessment that serves distinct purposes at each level while contributing to a cumulative, cross-level evidence base.

Is the assessment theoretically grounded and generalizable to other contexts?Any assessment procedure rests on a set of assumptions—sometimes implicit—about the nature of a content domain, the purpose of instructional or programmatic interventions, and the intended learning outcomes. Battistoni’s (2002) discipline-linked frameworks for civic learning and Deardorff’s (2006) model of intercultural competence are two examples of frameworks related to the domains of diversity, global, and civic learning that can help practitioners make those assumptions explicit and coherent. Such frameworks and theories elevate assessment activities so that they not only evaluate particular programs but also contribute to a generalizable knowledge base. Through collaborative, scholarly approaches, practitioners can mobilize appropriate theory to inform assessment in the diversity, global, and civic domains while also using assessment to refine and generate theory.

Emerging Assessment Issues

Beyond these four established areas, assessment of diversity, global, and civic learning in particular highlights considerations that can enhance approaches to assessment more generally. Diversity, global, and civic learning may well be most powerfully cultivated in academically-grounded experiential learning contexts—giving rise to three additional questions.

Whose voices and perspectives are included in the assessment process?Diversity, global, and civic learning are often associated with pedagogies—such as domestic and international service learning—that, at their best, embody a democratic epistemology. These pedagogies position members of various constituencies as co-educators, co-learners, and co-generators of knowledge, with learning goals that vary appropriately from context to context. When assessing the outcomes of these pedagogies, a broader-than-usual range of stakeholders should participate in establishing the specific nature of learning goals, in determining what constitutes meaningful evidence of attainment, and in deciding how best to generate learning and gather evidence. Furthermore, these stakeholders should participate in gauging the quality of learning and in the design and evaluation of the assessment process itself. By decentralizing and coconstructing assessment approaches, practitioners can ensure that assessment is shaped by and is useful to multiple constituencies within and beyond the academy.

Does the assessment approach align with the nature of the learning process? In academically-based experiential education, critical reflection on experience—not experience alone—both generates learning and provides evidence of learning. Faculty may encourage students to use course concepts as lenses through which to examine their experiences, challenging and deepening their knowledge, skills, and dispositions in various domains of learning. Such an approach integrates critical reflection and assessment to generate, deepen, and document learning. As an example of facilitating and assessing integrative learning and coconstruction of knowledge, this approach has widespread implications that extend beyond experiential education pedagogies or learning in these three domains (Ash and Clayton 2009).

Does the assessment process integrate all relevant learning contexts?Experiential education, which is well-suited to cultivating diversity, global, and civic learning, often involves out-of-classroom academic activities. These cocurricular contexts may lack the structures and norms of course-embedded assessment, and educators may have little experience collaborating across curricular and cocurricular settings or with partners beyond higher education to conceptualize learning goals, design assessment procedures, or interpret evidence. In the diversity, global, and civic domains, higher-order learning may depend on multiple, iterative opportunities for students to reflect critically on their experiences. Insofar as such learning is transformative in nature, it may develop over extended periods of time, across multiple courses, and in a variety of contexts. Assessing such learning in a significant way may therefore require developmentally designed and integrated strategies that depend on collaboration across multiple teaching and assessment contexts—a welcome deepening and broadening of traditional assessment practices.

Catalyzing Institutional Change

Assessment of diversity, global, and civic learning holds the potential not only to enhance assessment practices more generally, but also to catalyze institutional change in higher education. To date, pedagogies and programs focused on learning in these domains have been somewhat peripheral in the academy, amounting to what Cuban (1988) describes as “first-order change” in that they have largely failed to alter the fundamental cultures or organizational features of colleges and universities. However, these initiatives may eventually evoke second-order change—change that transforms the purposes, policies, identities, structures, and operations of institutions. In the realm of assessment, second-order change would mean much more than adding learning objectives to a syllabus or incorporating additional items into established assessment instruments and procedures. Rather, it would involve altering the assumptions and norms that underlie how institutions conceive of and approach not only assessment itself, but also teaching, learning, scholarship, partnerships, and engagement.

The imperative for institutional change in these areas is growing more urgent as the population of the United States becomes more diverse; global interdependency demands greater knowledge of and experience with others; and sectarian and ideological divisions, ethnic rivalries, and economic as well as ecological crises challenge communities. These powerful, interconnected, and fast-paced forces call higher education to establish now what diversity, global, and civic learning should look like; how to generate learning in these domains across the full range of students, disciplines, and institutional types; what infrastructure would allow change to occur; and what impacts students can have in the world if they have attained such learning and developed the capacities to continue it throughout their lives. We need to know now how best to integrate learning across diversity, global, civic, and other domains, and we need to know how to leverage pedagogical designs focused on integrative learning for broader and deeper institutional change. The scholarly attention increasingly paid to high-impact pedagogies can provide a vibrant context for such inquiry into teaching and learning, contributing to a knowledge base and informing institutional transformation that is so timely yet so challenging.

One key challenge facing higher education institutions as they reconsider their core purposes is the relationship between education for economic (career) and for civic (citizenship) purposes. Colleges and universities certainly stimulate economic development by teaching job-related knowledge and skills; but a focus on private gain (credentialing for employment) may displace public good (educating for citizenship) as the primary raison d’être of the academy—to the detriment of our students, our communities, and our democracy. By collaboratively conceptualizing and developing the means to integrate and then assess diversity, global, and civic learning, practitioners may contribute new insights into the connections between learning for work and learning for citizenship. By generating and assessing learning across these three domains, colleges and universities committed to nurturing the development of civic-minded graduates and professionals may advance second-order institutional transformation.

Work on assessing diversity, global, and civic learning—and especially their integration—promises to inform higher education’s understanding not only of graduates’ knowledge, skills, dispositions, perspectives, and identities, but also of what higher education institutions should become. We applaud those who have initiated inquiry in these areas, and we challenge colleagues throughout and beyond the academy to continue this important work.

References

Ash, Sarah L., and Patti H. Clayton. 2009. “Generating, Deepening, and Documenting Learning: The Power of Critical Reflection in Applied Learning.”Journal of Applied Learning in Higher Education 1: 25–48.

Battistoni, Richard M. 2002. Civic Engagement Across the Curriculum.Providence, RI: Campus Compact.

Cuban, Larry. 1988. “A Fundamental Puzzle of School Reform.” Phi Delta Kappan 69 (5): 341–44.

Deardorff, Darla K. 2006. “Identification and Assessment of Intercultural Competence as a Student Outcome of Internationalization.” Journal of Studies in International Education 10 (3): 241–66.

Dunning, David, Chip Heath, and Jerry M. Suls. 2004. “Flawed Self-Assessment: Implications for Health, Education, and the Workplace.”Psychological Science in the Public Interest 5 (3): 69–106.

Howard, George S., and Patrick R. Dailey. 1979. “Response-Shift Bias: A Source of Contamination of Self-Report Measures.” Journal of Applied Psychology 64 (2): 144–50.

Robert G. Bringle is Kulynych/Cline Visiting Distinguished Professor of Psychology at Appalachian State University, Patti H. Clayton is practitioner–scholar at PHC Ventures and senior scholar at Indiana University–Purdue University Indianapolis, and William M. Plater is Chancellor’s Professor Emeritus at Indiana University–Purdue University Indianapolis.

Previous Issues