Peer Review

A Teaching Intervention that Increases Underserved College Students’ Success

The challenge to provide equitable opportunities for college students to succeed is a critical priority for the Association of American Colleges and Universities (AAC&U). In 2014, AAC&U partnered with the Transparency in Learning and Teaching in Higher Education (TILT Higher Ed) project, founded at the University of Illinois and now housed at the University of Nevada, Las Vegas, on an initiative that significantly increases underserved college students’ success. TG Philanthropy funded the Transparency and Problem-Centered Learning project (www.aacu.org/problemcenteredlearning), with Tia McNair, Ashley Finley, and Mary-Ann Winkelmes as the coinvestigators. In its first year, the endeavor has identified a simple, replicable teaching intervention that demonstrably enhances students’ success, especially that of first-generation, low-income, and underrepresented college students in multiple ways at statistically significant levels, with a medium to large magnitude of effect. These results offer implications for how faculty can help their institutions to right the inequities in college students’ educational experiences across the country.

The Project’s Problem and Research Question

While federal and state initiatives focused on tuition relief are providing greater access to higher education, they do not guarantee equity of educational experience. Black, Hispanic, Native American, and Pacific Islander students are about half as likely to complete a four-year college degree as their white and Asian classmates (US Department of Education 2014). Completion rates for low-income students lag far behind those of students whose family incomes are above the bottom quartile (Tough 2014). And first-generation college students are 51 percent less likely to graduate in four years than students whose parents completed college (Ishitani 2006).

Colleges and universities have of course made valuable efforts to address these skewed and inequitable outcomes, relying upon predictive analytics and resources including advising, scholarships, tutoring, and community-building programs. But there has been little systematic study of the role that faculty can play collectively in improving learning outcomes and success for underserved students. The Transparency and Problem-Centered Learning project aimed to complement existing student support efforts by training faculty and instructors to implement a teaching intervention that showed promise for increasing underserved students’ success, and to research the impact of the intervention on students’ learning experiences.

The project’s basic research question in the 2014–2015 academic year was: What is the effect when teachers provide two transparently designed, problem-based take-home assignments (compared to the unrevised, business-as-usual take-home assignments in the comparison group) on spring-term first-year college students’ learning experiences, especially underserved students’ experiences, as measured by

  • the amount of transparency students perceived in the course (measured by Transparency in Learning and Teaching in Higher Education Survey questions 36-44); (https://unlv.co1.qualtrics.com/jfe/form/SV_9G0YyMonDPOfrX7);
  • students’ self-ratings of three important predictors of success: academic confidence, sense of belonging, and improved mastery of skills that employers value (measured by Transparency Survey questions 4-6, 8-11, 22, 24-25, 32, 34-35);
  • direct assessment of students’ work as indicated by scored student work samples, selected randomly (addressed elsewhere in this special issue of Peer Review); and
  • short-term retention rates.

Rationale

These measures of success and the rationale for our intervention intentionally align with several important past studies. Multiple researchers have demonstrated that increases in college students’ academic confidence and sense of belonging are directly correlated with higher GPAs and persistence and retention rates, especially for underserved students. Furthermore, they have demonstrated that these increases could be achieved through teaching/learning interventions. For example, ethnically underrepresented (African American) first-year college students who completed an exercise that aimed to increase their feelings of social belonging earned higher GPAs in the subsequent three years, reduced the racial achievement gap, reduced their feelings of self-doubt, increased their confidence, and were more likely to be in the top 25 percent of their college class (Walton and Cohen 2011). In addition, struggling college students increased their test scores after endorsing the belief that intelligence is not fixed but rather malleable. One year later, these students were 80 percent less likely to drop out of college, and their GPAs continued increasing (Aronson et al. 2002). For both white and African American first-year college students, sense of belonging can indirectly increase students’ persistence behaviors (Hausmann et al. 2009).

Several past studies suggested that our intervention, in which teachers revised two take-home assignments in a term to make them more transparently designed (accessible) and problem centered (relevant) for students, might increase students’ academic confidence and sense of belonging, as well as their mastery of skills that employers value, across a group of geographically dispersed schools. A 2013 study identified transparency—engaging teachers and students in focusing together on how college students learn what they learn and why teachers structure learning experiences in particular ways—as a teaching method that showed promise for improving underserved students’ educational experiences in college (Winkelmes 2013). Another indicated that high-impact practices increased undeserved students’ engagement (Finley and McNair 2013). Underachieving students across a group of geographically diverse schools (high schools, in the study) experienced higher GPAs and increased pass rates after receiving two implementations of a mindset intervention that provided students with information about how they learn and the purposes for their academic work (Paunesku et al. 2015). Recent national surveys of employers identified the ability to apply skills to solving problems as one of the skills employers value most, and our project’s analyses included these problem-centered skills and others employers value (Transparency Survey questions 4-6, 8-11, 22, 24, 32) (Hart 2015, 2013; Winkelmes 2015).

Intervention

We selected an intentionally varied set of institutions to implement the project’s intervention, so that any school viewing the results would find a collaborator in our group with whom they could identify (see table 1).

Table 1. Participating Institutions

Institution Name Size Programs; Enrollment Carnegie Class; Setting Private Public
Community College of Philadelphia Large Two-year; exclusively undergrad, mixed part-time full-time Associates; Single campus,
Urban-serving, MSI
Public
California State University–Los Angeles Large Four-year, professional, Master’s; high
undergrad
Primarily non-
residential; MSI, HSI
Public
St. Edward’s University, Austin, TX Medium Four-year, professional, Master’s; very high undergrad full-time Primarily residential; MSI, HSI Private
Heritage University, Toppenish, WA Very Small Four-year professional;
high undergrad
Primarily non-residential; on Yakama Indian Reservation, MSI, Native American Serving Private
University of Houston–Downtown, TX Medium Four-year, professional, Master’s; very high undergrad, part-time Primarily non-
residential; MSI, HSI
Public
Queensborough Community College, Very Large Two-year; exclusively undergrad, mixed part-time full-time Associates; Multicampus, Urban-serving, MSI, HSI Public
Winston-Salem State University, NC Medium Four-year, professional Master’s; very high undergrad, full-time Primarily residential; MSI, HBCU Public

The experiment included 1,800 students and thirty-five faculty from these schools. As part of the project, faculty received training on how to make two take-home assignments in a course more transparently designed and more problem-centered for students. Each faculty member taught two class groups of the same course in the spring 2015 term; one group would receive the intervention of two revised assignments, and the other would receive unrevised versions of the two assignments. Most of the courses were introductory-level courses containing first-year students; twelve were intermediate-level courses. Class sizes ranged from nine to seventy-four students, with an average class enrollment of about twenty-nine students. Faculty who implemented the two revised assignments agreed to adopt the Transparent Assignment Template to frame conversations with students about the purposes, tasks, and criteria for each revised assignment, before students began working (fig. 1).

PR_WISP16__Winkelmes_Fig1_Web.jpg
 
 
 
 
 
 
 
 
 
 
(Click on figure to enlarge)
 

At the end of term, sixty-one of the seventy courses completed the experiment. However, many teachers struggled to keep the intervention cleanly out of their control courses after seeing students respond positively in their intervention courses. Others found it difficult to limit the intervention to only two assignments. All 1,800 students were invited to respond to questions about their learning experiences on the end-of-term Transparency in Learning and Teaching Survey online. Sixty-eight percent of students responded to the survey, with 1,174 students or 65.2 percent completing all the survey questions. Historically underserved students in this group exceeded the three-hundred-fifty-person sample size recommended by What Works Clearinghouse (WWC) standards (US Department of Education 2014). The survey was completed by 425 first-generation students, 402 non-white students, and 479 low-income students. In addition, 297 multiracial students completed the survey.

Results

The results of our project suggest that faculty can contribute to increasing all students’ success, especially that of underserved students, in their first year of college (when the greatest number of students drop out) (Head and Hosteller 2015). In courses where students perceived more transparency as a result of receiving the transparently designed, problem-centered take-home assignments, they experienced significantly greater learning benefits compared with their classmates who perceived less transparency around assignments in a course. Specifically, students who received more transparency reported gains in three areas that are important predictors of students’ success: academic confidence, sense of belonging, and mastery of the skills that employers value most when hiring. These are “substantively important” and statistically significant findings that satisfy WWC standards for baseline equivalence measures of 0.05 or below, sample sizes above three hundred fifty, and effect size differences above 0.25 (US Department of Education March 2014).

The discussion that follows includes data from all 1,174 students who completed the survey in all sixty-one courses that completed the experiment. In a constrained sample of thirty-nine courses where the intervention was implemented twice as planned, 262 students who received the intervention in eighteen courses experienced significantly increased academic confidence and sense of belonging (with a magnitude of ES=0.30 and ES=0.32 respectively) compared with 396 students in twenty-one control group courses who received the instructors’ unedited assignments. Instead of limiting our analysis to this subset, we discuss the full sample to offer a realistic indicator of what teachers and institutions can expect in practice when courses provide greater or lesser amounts of transparency for students around the purposes, tasks, and criteria for their academic work.

The benefits for all students in the full sample who received greater transparency were statistically significant (p<.05) and substantively important (fig. 2).

PR_WISP16__Winkelmes_Fig2_Web.jpg

 

 
 
 
 
 
 
 
 
 
 
(Click on figure to enlarge)

 

For first-generation, low-income, and underrepresented students, those benefits were larger. First-generation students and multi-racial students experienced medium-to-large effect size differences in the three domains that are critical predictors of students’ success: academic confidence, belongingness, and mastery of the skills that employers value (figs. 3 and 4).

 

PR_WISP16__Winkelmes_Fig3_Web.jpg

 

 
 
 
 
 
 
 
 
 
 
(Click on figure to enlarge)
 
 

PR_WISP16__Winkelmes_Fig4_Web.jpg

 
 
 
 
 
 
 
 
 
 
 
 
 
(Click on figure to enlarge)
 

A baseline equivalence test indicated that, prior to the intervention, groups who would receive more and less transparent instruction did not differ significantly (fig. 5).

PR_WISP16__Winkelmes_Fig5_Web.jpg
(Click on figure to enlarge)
 

The single largest underrepresented ethnicity group of students in our study was multiracial, with 237 students self-identifying in this category. Students who self-identified as belonging to a single underrepresented (non-white) ethnicity and students of low socioeconomic status (low-income, bottom income quartile) reported statistically significant, somewhat smaller benefits in the same three areas (figs. 6 and 7).

PR_WISP16__Winkelmes_Fig6_Web.jpg
(Click on figure to enlarge)
 
PR_WISP16__Winkelmes_Fig7_Web.jpg
(Click on figure to enlarge)
 

What was it about the intervention that underserved students noticed and appreciated? In the more transparent courses, first-generation, underrepresented, and low-income students indicated most often the following aspects:

  • In this course, I knew the purpose of each assignment.
  • Each assignment included a section that explained how the assignment was related to the objectives of the course.
  • In this course, I knew the steps required to complete my assignments.
  • Coursework and course activities benefited my learning.
  • In this course, I knew how my work would be evaluated.
  • My instructor provided students with annotated examples of past students’ work.
  • My instructor provided tools I could use to assess the quality of my and others’ work.

While all students in the aggregate in the more transparent courses reported an increase in their mastery of the skills that employers value most (Hart Associates 2015, 2013), these were the specific skills for which underserved students in our study noticed the greatest increases:

  • connecting information from a variety of sources;
  • learning on your own;
  • applying knowledge and skills to different contexts;
  • writing effectively;
  • judging the reliability of information from various sources;
  • considering opinions or points of view different from your own;
  • judging the strengths and weaknesses of ideas.

In science, technology, engineering, and mathematics courses that offered more transparency, there were small increases to students’ academic confidence and their sense of belonging, in comparison with the less transparent courses in these disciplines (fig. 8). In more transparent humanities, arts, and social science courses, students experienced medium effect size increases in the skills valued by employers, as well as small-to-medium effect size increases to their academic confidence and sense of belonging, in comparison with the courses that offered less transparency in these disciplines (fig. 9).

PR_WISP16__Winkelmes_Fig8_Web.jpg
(Click on figure to enlarge)
 
PR_WISP16__Winkelmes_Fig9_Web.jpg
(Click on figure to enlarge)
 

Students’ short-term retention rates in the more transparent courses were slightly higher than those of students in the less transparent courses. While 9.18 percent of students (65 of 708 students) withdrew before the end of the term from the twenty-four less transparent courses where we gathered withdrawal data, only 7.50 percent of students (44 of 586 students) withdrew from the twenty-one more transparent courses where we gathered withdrawal data. In a parallel study of 1,143 University of Nevada, Las Vegas undergraduates in more transparent introductory-level courses, 90.2 percent of students returned the subsequent academic year, in contrast to the average retention rate of 74.1 percent for first-time, full-time, first-year students.

Most of the faculty and instructors in our study now incorporate transparently designed, problem-centered assignments in all the courses they teach—not just the courses that our study included. We expect this to benefit long-term retention rates of students at these institutions.

Long-Term Implications

Our study identifies transparent teaching about problem-centered learning as an easily replicable teaching intervention that produces learning benefits already linked with students’ success. Providing greater transparency about academic work on two assignments resulted in significant benefits for first-generation, low-income, and underrepresented students, who experienced increases in areas that are established predictors of student success: their academic confidence, sense of belonging, and awareness of their improved mastery of the skills employers value most when hiring. In addition to students, faculty also experienced benefits. Faculty noticed increases in students’ motivation in class, higher-level class discussions with sharper focus, more on-time completion of assignments, and fewer disputes about grades (Winkelmes et al. 2015).

For institutions, training faculty to incorporate transparently designed, problem-centered assignments may help to provide more equitable educational experiences and increase retention and completion rates, especially for underserved students. Teams of faculty at an institution, like the seven teams in our project, can implement this intervention in order to complement existing student support services that aim to increase students’ success. As our project continues, we can expect increased GPAs, retention, and graduation rates of underserved students at the seven minority-serving institutions where the project is underway, and at more schools that join the initiative. The easily replicable and relatively small intervention of two transparently designed, problem-centered assignments in a term can support faculty and their institutions in righting current inequities in underserved college students’ educational experiences, and in providing more equitable opportunities for all students to succeed.

 

References

Aronson, Joshua, Carrie B. Fried, and Catherine Good. 2002. “Reducing the Effects of Stereotype Threat on African American College Students by Shaping Theories of Intelligence.” Journal of Experimental Social Psychology 38: 113–125.

Finley, Ashley, and Tia McNair. 2013. Assessing Underserved Students’ Engagement in High-Impact Practices. Washington, DC: Association of American Colleges and Universities.

Fry, Richard. 2014. “The Changing Profile of Student Borrowers: Appendix B Details on the Data Source.” Washington, DC: Pew Research Center.

Hart Research Associates. 2015. Falling Short? College Learning and Career Success. Washington, DC: Association of American Colleges and Universities.

——. 2013. It Takes More than a Major: Employer Priorities for College Learning and Student Success. Washington, DC: Association of American Colleges and Universities.

Hausmann, Leslie R. M., Feifei Ye, Janet Ward Schofield, and Rochelle L Woods. 2009. “Sense of Belonging and Persistence in White and African American First-Year Students.” Research in Higher Education 50 (7): 649–69.

Head, Alison, and Kirsten Hosteller. 2015. “Mary-Ann Winkelmes: Transparency in Teaching and Learning.” Project Information Literacy, Smart Talk Interview 25. Creative Commons License 3.0: Retrieved September 2, 2015, from http://projectinfolit.org/smart-talks/item/149-mary-ann-winkelmes-smart-talk.

Ishitani, Terry T. 2006. “Studying Attrition and Degree Completion Behavior among First-Generation College Students in the United States.” The Journal of Higher Education 77 (5): 877.

Paunesku, David, Gregory M. Walton, Carissa Romero, Eric N. Smith, David S. Yeager, and Carol S. Dweck. 2015. “Mind-Set Interventions Are a Scalable Treatment for Academic Underachievement.” Psychological Science 26: 784–93.

Tough, Paul. 2014. “Who Gets to Graduate?” The New York Times Magazine, May 15. New York: New York Times Company.

US Department of Education. 2014a. “Graduation rates of first-time, full-time bachelor’s degree-seeking students at 4-year postsecondary institutions, by race/ethnicity, time to completion, sex, and control of institution.” Digest of Education Statistics, Integrated Postsecondary Education Data System (IPEDS). Washington, DC: National Center for Education Statistics and Institute of Education Sciences. https://nces.ed.gov/programs/digest/d13/tables/dt13_326.10.asp.

——. 2014b. WWC Procedures and Standards Handbook, Version 3.0. Washington, DC: Institute of Educational Service.

Walton, Gregory M., and Geoffrey L. Cohen. 2011. “A Brief Social-Belonging Intervention Improves Academic and Health Outcomes of Minority Students.” Science 331 (6023): 1447-1451.

Winkelmes, Mary-Ann. 2009. Transparency in Teaching and Learning in Higher Education Project. Creative Commons License 3.0.

. 2013. “Transparency in Teaching: Faculty Share Data and Improve Students’ Learning.” Liberal Education 99 (2): 48–55.

. 2015. Transparency in Learning and Teaching in Higher Education Online Survey. Creative Commons License 3.0. Retrieved September 27, 2015 from https://unlv.co1.qualtrics.com/jfe/form/SV_9G0YyMonDPOfrX7.

Winkelmes, Mary-Ann, David E. Copeland, Ed Jorgensen, Alison Sloat, Anna Smedley, Peter Pizor, Katharine Johnson, and Sharon Jalene. 2015. “Benefits (Some Unexpected) of Transparently Designed Assignments.” National Teaching & Learning Forum 24 (4): 4-6.


Mary-Ann Winkelmes, coordinator, instructional development and research, office of the provost and associate graduate faculty, history department, University of Nevada, Las Vegas; senior fellow, AAC&U; Matthew Bernacki, assistant professor, educational psychology and higher education; Jeffrey Butler, visiting lecturer, department of economics; Michelle Zochowski, graduate student; Jennifer Golanics, graduate student; Kathryn Harriss Weavil, graduate student—all of University of Nevada, Las Vegas

Previous Issues