Assessment as a Strategy, Not a Stand-Alone Activity

When the Association of American Colleges and Universities (AAC&U) developed the VALUE rubrics ten years ago, the rubrics began as a set of shared expectations around key learning outcomes and essential skills associated with student success in school and life. These rubrics were created and agreed upon by educators in the field who described what to look for in student work as learners progress through their educational pathways to acquire progressively more sophisticated knowledge and competence as they approach attainment levels associated with a baccalaureate degree. Intended as metarubrics, VALUE rubrics use general descriptors to represent and capture fundamental dimensions of each learning outcome while encouraging students to demonstrate learning in different contexts, media, and forms appropriate to their circumstances.

Through the VALUE approach to assessment, institutions collect direct evidence of student learning using performance-based assessments. VALUE is exemplary in this regard because the direct evidence (e.g., final term papers) is closely linked to students’ learning experiences in college courses. This is in contrast to other direct evidence measures that are collected through standardized tests administered outside of the classroom and not aligned with the curriculum.

More than 70,000 individuals—affiliated with 5,895 organizations, including 2,188 colleges and universities—have downloaded the rubrics. In 2014, AAC&U was able to engage in a systematic scaling of the VALUE rubric approach in collaboration with the State Higher Education Executive Officers association’s Multi-State Collaborative (MSC), eventually involving thirteen states and ninety-two two- and four-year institutions. Most recently, in 2017, AAC&U announced the creation of the VALUE Institute.

Much More Than a Score

Paying attention to students’ development does not detract from their learning. In fact, promoting the highest levels of development among students seems to help them reach high academic goals (Ben-Avie et al. 2003). This is particularly the case with academic habits of mind. When faculty promote students’ academic habits of mind, students gain the competency to work autonomously, handle cognitive complexity, master processes of inquiry that are common to all academic disciplines, and advocate for themselves. In short, institutions can build multidimensional predictive models combining direct and indirect evidence to strengthen student learning success.

One of the best ways to illustrate the benefits of the VALUE approach compared to other assessment methods (like standardized testing) is to examine it in practice. One MSC state—Connecticut—illustrates the robust implications of the VALUE approach as a strategy for learning assessment rather than a one-off activity. The directors of assessment from the participating Connecticut institutions initially thought that the MSC was yet another stand-alone activity, not a strategy. Stand-alone activities tend to be externally mandated, and the results—even if they are widely shared on campus—are not effectively used to improve students’ learning and developmental outcomes. The results from these stand-alone activities are rarely incorporated into longitudinal cohort datasets. However, following students from new-student orientation until they graduate or transfer to other colleges provides information that is useful when identifying important metrics for judging the effects of higher education.

For example, at Southern Connecticut State University (SCSU), the VALUE rubric scores of student work artifacts were incorporated into longitudinal cohort datasets to observe students’ academic performance in relation to competencies (e.g., future orientation, interpersonal relationships, sense of belonging, self-regulation, academic habits of mind) from the developmental sciences. By incorporating scores of 444 work artifacts from SCSU students into long-term studies, VALUE and the MSC brought learning evidence into SCSU’s promotion of students’ success and development.

An important result from the study influenced the campus conversation about on-time graduation. Students who graduated in four years had significantly higher scores on quantitative literacy (QL) than those who graduated in five or six years. This finding attracted the interest of those involved in enrollment management and student success. Analyses determined that students’ higher QL scores were predicted by the experiences they had on campus instead of their past development, demographic characteristics (e.g., ethnicity or socioeconomic status), or precollege learning (e.g., SAT scores or high school GPA). As a result, VALUE results contributed to a shift in strategy at the university to promote student success.

The research results provided evidence that practices that higher education institutions control can promote students’ competencies in written communication, quantitative literacy, and critical thinking. Thus, the MSC VALUE approach became an antidote to metrics that had been proposed for judging the effectiveness of higher education that were outside the control of the university, because the characteristic or learning either occurred prior to college or was related to postgraduation employment. Consider, for example, the Wall Street Journal’s method of ranking colleges. As Melissa Korn explains this ranking method, “Outcome scores are derived from graduation rates and academic reputation, as well as measures of loan-repayment rates and graduate salaries” (Korn 2016).

At the institutional level, VALUE can effectively leverage assessment for improvement because participation requires, for instance, assessment leaders to actively engage faculty members and other educators in the data collection process. As a result, knowledge and understanding of assessment increase among educators. In addition, educators become more committed to assessment because VALUE invites them to attend training on how to use a rubric to score students’ work products. Through the use of rubrics, students’ reflective thought and insight are valued and, as a result, provide an incentive for faculty development in these areas.

At the same time, the MSC VALUE approach became an accreditation strategy when conversations about outcomes led to changes based on evidence instead of anecdotes or preconceptions. SCSU’s accrediting agency, the New England Association of Schools and Colleges (2017), commented on changes that were made at the university due to participation in the MSC:

We are pleased to learn that results gleaned from analyses of student work conducted as part of Southern Connecticut State University’s participation in the Multi-State Collaborative have been used to inform the restructuring of the University’s access programs, developmental math curriculum, liberal education program, and writing across the curriculum program.

Value PLUS Development Equals a Winning Model

By students’ senior year, differences in VALUE scores between students who were eligible for PELL grants and their peers were not found in written communication, quantitative literacy, and critical thinking. To improve students’ competencies in learning outcomes, faculty development events should focus on topics such as written communication, quantitative literacy, and critical thinking, because improving students’ competencies also requires faculty knowledge of how to apply the developmental sciences to classroom practices without compromising the objectives of a course.

At SCSU, assessment professionals built a predictive model using specific items on the Academic Habits of Mind and College Success Inventory (AHM-CS), designed by the Office of Assessment and Planning to measure the relationship between students’ learning and development. This self-assessment inventory predicted students’ overall VALUE scores in written communication, quantitative literacy, and critical thinking. For example, the item from the AHM-CS that asked students to indicate whether they take the initiative to talk with their professors when issues arise predicted students’ overall quantitative literacy scores. In other words, if we know the extent to which students take the initiative to talk with their professors, then we also know how they are likely to score on the quantitative literacy VALUE rubric.

On the critical thinking VALUE rubric, students are expected to demonstrate a high level of competency in evaluating issues, artifacts, and ideas before formulating a conclusion. Students are expected to analyze text and draw conclusions that either support or question the author’s viewpoints. Students who “strongly agreed” on the AHM-CS that their professors were teaching them how to express their positions during classroom discussions tended to have VALUE scores that effectively met the expectations at progressively higher levels within the critical thinking rubric.

The written communication VALUE rubric expects that students demonstrate ample consideration of the audience and purpose for the writing assignment. A moderate relationship was observed between the students’ “context of and purpose for writing” score on the rubric and the AHM-CS item that asked if the students were able to effectively read and comment on the work of fellow students. This indicates that the students who were required to review and critique the papers of their peers gained an awareness of how to write papers for two audiences: their professors and their peers.

The university’s assessment strategies can now be used to empirically evaluate the relationship between students’ learning and development. A student’s ability to demonstrate learning according to the rubrics is a function of (1) academic habits of mind such as the ability to work autonomously and handle cognitive complexity, (2) content knowledge, (3) interpersonal relationships, and (4) an orientation to the future by setting goals and taking immediate actions to achieve desired futures. Mastering these habits and competencies can change students’ developmental trajectory beyond what demographic characteristics and learning prior to college alone can predict. Thus, issues related to educational psychology are relevant for all students, and not only for students at risk of not thriving in college and in life. Using VALUE data, universities are able to show the impact of the education they provide as students veer from their predicted trajectories, making this “value added” one of the clearest metrics of the effectiveness of higher education.

Coda

For ten years, the VALUE rubrics have provided an approach to assessing AAC&U’s LEAP Essential Learning Outcomes associated with student success in society, work, and life. Employers continue to strongly echo educators in saying that graduates need to demonstrate competence in these outcomes, and they are increasingly finding value in these cross-cutting outcomes that is equal to—if not more important than—a graduate’s major field of study (Hart Research Associates 2018).

The MSC and the VALUE Institute have been able to systematically examine at scale the VALUE approach to learning assessment. The intriguing results have been gratifying for SCSU and include evidence that (1) students are learning essential outcomes; (2) engaging students in their learning (e.g., through high-impact practices) makes a positive difference in the quality of learning; (3) assessment results can engage educators and provide information useful for enhancing effectiveness in classroom practice; (4) educators and institutions can make a difference in closing equity gaps in learning quality and achievement; and (5) VALUE can lift up ways that educators can collaborate to achieve high-quality learning for all. In short, we now have evidence that higher education brings substantial value to individuals and to society.

 

References

Ben-Avie, Michael, Norris M. Haynes, Jayne White, Jacque Ensign, Trudy Raschkind Steinfeld, Loleta D. Sartin, and David A. Squires. 2003. “Youth Development and Student Learning in Math and Science.” In How Social and Emotional Development Add Up: Making Gains in Math and Science Education, edited by Norris M. Haynes, Michael Ben-Avie, and Jacque Ensign, 9−35. New York: Teachers College Press.

Hart Research Associates. 2018. Fulfilling the American Dream: Liberal Education and the Future of Work. Washington, DC: Association of American Colleges and Universities. https://www.aacu.org/sites/default/files/files/LEAP/2018EmployerResearchReport.pdf.

Korn, Melissa. 2016. “What’s the Payoff?” Wall Street Journal, September 29, 2016, R2.

New England Association of Schools and Colleges. 2017. Unpublished letter. February 2, 2017.


Michael Ben-Avie, Senior Director for Learning Assessment and Research, Quinnipiac University, formerly of Southern Connecticut State University; Kevin Kuna, Student, Southern Connecticut State University; and Terrel Rhodes, Vice President, Office of Quality, Curriculum, and Assessment and Executive Director of VALUE, AAC&U

 

Select any filter and click on Apply to see results