Diversity and Democracy

Designing a Model for International Learning Assessment

Calls to internationalize higher education have grown more urgent over the past decade as universities endeavor to meet the shifting social, political, and economic exigencies of our interconnected societies. Pressing issues such as global warming, the yawning chasm between rich and poor, and international violence compel educators to equip students with the global competencies necessary to address complex challenges in both local and international contexts. At the heart of this process is the need to evaluate global education’s impact on the institution and its students.

Assessment can provide a range of benefits: meaningful information to ensure effective programs; a rationale for advocacy of international education; and knowledge of the learning and developmental processes that underpin transformation, to name just a few. In the context of international education, assessment can examine factors such as language acquisition, content knowledge (geography, history, cultural customs and practices), and intercultural competence, communication, and sensitivity. When planned and implemented effectively, assessment moves beyond traditional notions of inputs and outputs to capture the complex impact of international learning. This article invites you to set your own agenda for assessing international learning outcomes at your institution.

Assessment Basics

Table 1: Assessment Team’s Checklist

__ Aligned and Articulated: Are goals, objectives, and assessment measures aligned and articulated?
__ Intentional: Is assessment intentionally addressed?
__ Developed: Have assessment issues been carefully analyzed before a plan is implemented?
__ Integrated: Is assessment integrated throughout the program and not viewed as an “add-on” (implemented only as a pre-post phenomenon)?
__ Focused: Is the assessment scope realistic, with two to three outcomes assessed per program per year?
__ Shared: Is assessment shared with others on campus through partnerships?
__ Supported: Is the senior leadership supportive of assessment efforts?
__ Resourced: Is there adequate time and funding for assessment efforts, and have administrators received sufficient training in assessment, with ongoing professional development?
__ Analyzed: Have the assessment tools, results, and process been analyzed and evaluated?
__ Communicated: Have the results been communicated to all stakeholders?
__ Used: Have the results been used for program improvement as well as for learner feedback?
__ Reviewed: Has the assessment process and strategy been reviewed on a regular basis and improved upon?

Developed by Darla Deardorff, 2008

Assessment should begin with a review of the institution’s mission statement and overall goals. What are the institution’s priorities, and what evidence can demonstrate their achievement? How can the institution best combine summative assessment (which focuses on learners’ development at a particular time) and formative assessment (a more holistic approach that uses multiple points of measurement to provide continuous feedback and improve educational opportunities) to meet its goals? By answering these and related questions, administrators can determine which assessment methods and tools to use.

The next step in developing an assessment protocol is to explore the institutional context and available resources, including others who are already engaged in assessment. This process should lead to the creation of a multiunit assessment team, as the undertaking is too complex for any one office to implement. The team should develop an assessment plan and review it regularly for refinement. Table 1 provides guidance in this process.

A good assessment plan should include multiple tools and methods integrated throughout the program. Within coursework, for example, assessments can evaluate both direct evidence of student learning (tests, papers, capstones, portfolios) and indirect evidence, meaning student perceptions of their learning (self-report instruments, focus groups, interviews). Questions to consider when selecting tools include: What does the tool measure, and how does it support stated goals? Is the tool valid, reliable, and based on a theoretical framework? What are the tool’s limitations and biases? Are the tool’s logistics manageable, and is the tool affordable?

Once the team has developed a strategy, assessment can begin. The assessment cycle consists of several steps: 1) Define outcomes (based on mission/goals) and establish measurable criteria 2) identify appropriate assessment methods 3) collect data 4) analyze data 5) use data: design and apply changes to the curricular and noncurricular program components 6) communicate results to all relevant stakeholders and 7) evaluate the assessment process and plan. (For further discussion of assessment in international education, see Deardorff 2007.)

Case Study: MSU

Michigan State University (MSU) is currently implementing a formative assessment project with promising results. The MSU study aims to determine the efficacy of using a mixed-methods approach to examine the outcomes of students’ international learning relevant to global and domestic issues, to investigate the influence of key antecedent factors, and to determine how findings can influence curricular and noncurricular enrichment decisions. To address these goals, MSU developed a pioneering conceptual model to assess students’ progress.

The qualitative (e-portfolio) phase of the project originated with five other institutions in a project funded by the Fund for Improvement of Postsecondary Education (FIPSE) and coordinated by the American Council on Education (ACE 2008). The quantitative phase (administration of the Beliefs, Events, and Values Inventory, or BEVI) originated with an international learning assessment project coordinated by the Forum on Education Abroad (2008). To our knowledge, MSU is the only university using a mixed-methods approach that combines the BEVI and e-portfolio.

Theoretical Background

To address the complexity of international learning, the MSU team analyzed different theoretical perspectives for insights into antecedent factors that influence student learning, different dimensions of international learning, and ways to measure these dimensions. The team drew from three theoretical frameworks (Equilintegration Theory, Attribution Theory, and Learning Theory) to structure a conceptual model.

Equilintegration (EI) Theory attempts to explain the processes by which beliefs, values, and worldviews are acquired and maintained, why students typically resist their alteration, and how and under what circumstances their modification occurs (Shealy 2004, forthcoming). EI Theory recognizes that values and beliefs are not easily modified: they represent the unique culmination of affective and attributional processes that arise from life experiences.

Attribution Theory (AT) focuses on three dimensions that influence an individual’s motivation: locus of control, stability, and controllability (Weiner 1974). Locus of control refers to the underlying causes of life events. An individual whose locus of control is internally oriented believes that controllable decisions and efforts guide behavior, while an external orientation suggests that behavior is guided by fate, luck, or other uncontrollable external factors (including race, gender, and socioeconomic level). Students are most motivated when they believe that success or failure results from their own (controllable) behavior rather than external (uncontrollable) circumstances.

Finally, Learning Theory (LT) explores how complex processes and environments affect international learning. According to Rogers (2003), learning can be examined as a product (change in behavior) and as a process (how and why behavior changes). Bloom (1956) classified learning products as cognitive (knowledge and intellectual skills), psychomotor (physical movement, coordination, and the use of motor skills), and affective (feelings, values, motivations, and attitudes). These three domains are also known as KSAs (knowledge, skills, and attitudes). Assessments can apply these theories to determine the impact of different learning processes in a range of learning environments—curricular, cocurricular, and extracurricular—all of which should be included in the assessment process (Rubin, Bommer, and Baldwin 2002; Mahoney, Cairns, and Farmer 2003).

Project Design

Guided by the missions and goals of their universities, the six teams involved in ACE’s early qualitative project developed nine common international learning outcomes (and associated performance indicators and scoring rubrics)—three outcomes each for the three domains of knowledge, skills, and attitudes (ACE 2008). (These KSAs are described in detail on the ACE Web site.) To address the complexity of learning outcomes and yield more accurate and comprehensive results, MSU chose to adopt a mixed-methods research design (Greene, Caracelli, and Graham 1989).

The quantitative instrument, the BEVI, is predicated on EI Theory and “designed to understand whether, how, and to what degree people are (or are likely to be) ‘open’ to various transformational experiences” (Shealy 2005, 99). Selected students take the pre-BEVI (as freshmen) and post-BEVI (ideally as seniors) to detect changes in international learning. The instrument contains three validity scales (to ensure that the respondent is answering in a consistent fashion across items) and ten “process scales” (assessing, for example, basic openness, receptivity to different cultures, tendency to stereotype, self/emotional awareness), as well as sixty-five demographic, situational, and background variables (Shealy 2005, forthcoming). The instrument is innovative, accessible (Web-based), affordable, and has demonstrated reliability and stability in previous applications. For more information on the BEVI, see www.forumea.org/research-bevi.htm.

The qualitative assessment incorporates AT and LT theories and relies on electronic portfolios to which students submit at least five work pieces over a period of time. Ideally, these artifacts represent a broad range of work from different class levels and international learning environments (curricular, cocurricular and extracurricular). Examples include course papers or other written work (in English or another language), photographs and digital images with commentary, course presentations, audio files containing music or recorded language skill demonstrations, and reflective essays. These artifacts provide direct evidence of changes in student attitudes and insight into values, affective development, and students’ potential for growth. Collecting them compels the student to select, interpret, and reflect on interconnected global experiences (Cambridge 2001, 15; Palomba and Banta 1999, 80-81). The e-portfolio provides an easily accessible mechanism to gather and represent international learning from a range of disciplines, learning environments, and class levels (Palomba and Banta 1999, 96, 26). For more information on the e-portfolio project, see www.acenet.edu.

Thus MSU’s conceptual model applies LT, AT, and EI Theory across a variety of processes and environments (see fig. 1). Preliminary data analysis suggests that this mixed-methods approach provides a more holistic and in-depth view of students’ development of multicultural competency. We believe that the project’s findings will inform MSU’s liberal learning outcomes, help coordinate effective classroom approaches with co- and extracurricular activities, and suggest other events and activities that might assist in internationalizing the student experience at MSU.

Figure 1: Conceptual Model

Figure 1: Conceptual Model

Developed by Pysarchik & Yun, 2008

 

Conclusion

The MSU project is illustrating that a well-implemented assessment protocol—linked to theory and using both qualitative and quantitative methods—is a powerful tool. While assessment requires resources and careful planning, it yields information that is highly beneficial to the institution and to the wider international education community. Its thoughtful implementation allows educators to help students develop the competencies that are critical to their success in our complex and interconnected world.

References

American Council on Education. 2008. Lessons learned in assessment.

Banta, T.W., J. P. Lund, K. E. Black, and F.W. Oblander. 1996. Assessment in practice: Putting principles to work on college campuses. San Francisco: Jossey-Bass.

Bloom, B. S. 1956. Taxonomy of educational objectives, Handbook I: The cognitive domain. New York: David McKay Co. Inc.

Cambridge, B. L., ed. 2001. Electronic portfolios as knowledge builders. Electronic Portfolios: Emerging practices in student, faculty, and institutional learning. Washington, DC: American Association for Higher Education, 1-11.

Deardorff, D.K. 2007. Principles of international education assessment. IIENetworker (Spring 2007): 51-52.

Forum on Education Abroad. 2008. www.forumea.org/research-bevi.htm

Greene, J. C., V. J. Caracelli, and W. F. Graham. 1989. Toward a conceptual framework for mixed-method evaluation design. Educational Evaluation and Policy Analysis 11 (3): 255-274.

Mahoney, J. L., B. D. Cairns, and T. W. Farmer. 2003. Promoting interpersonal competence and educational success through extracurricular activity participation. Journal of Educational Psychology 95 (2): 409-418.

Palomba, C. A. and T. W. Banta. 1999. Assessment essentials: Planning, implementing, and improving assessment in higher education. San Francisco: Jossey-Bass.

Rogers, A. 2003. What is the difference?A new critique of adult learning and teaching. Leicester: NIACE.

Rubin, R. S., W. H. Bommer, and T. T. Baldwin. 2002. Using extracurricular activity as an indicator of interpersonal skill: Prudent evaluation or recruiting malpractice? Human Resource Management 41 (4): 441-454.

Shealy, C. N. 2004. A model and method for “making” a combined-integrated psychologist: Equilintegration (EI) Theory, and the Beliefs, Events, and Values Inventory (BEVI). Journal of Clinical Psychology 60 (10): 1065-1090.

———. 2005. Justifying the justification hypothesis: Scientific-humanism, Equilintegration (EI) Theory, and the Beliefs, Events, and Values Inventory (BEVI). Journal of Clinical Psychology 61 (1): 81-106.

———. Forthcoming. Making sense of beliefs and values. New York: Springer Publishing.

Weiner, B. 1974. Achievement motivation and attribution theory. Morristown, NJ: General Learning Press.

Previous Issues