Peer Review

STEM/Non-STEM Differences in Engagement at US Institutions

A recent paper by one of us (Nelson Laird) and some colleagues brought some sobering news of differences between STEM (science, technology, engineering, and mathematics) and non-STEM undergraduates with regard to approaches to learning that promote more complex, deeper understanding. Using data from the National Survey of Student Engagement (NSSE) and the Faculty Survey of Student Engagement (FSSE), Nelson Laird and colleagues examined disciplinary differences in the extent to which students are exposed to educational environments that promote deep approaches to learning. These approaches to learning are important because “[s]tudents who use deep approaches to learning tend to perform better as well as retain, integrate, and transfer information at higher rates than students using surface approaches to student learning” (Nelson Laird, Shoup, Kuh, and Schwarz 2008, 470).

Nelson Laird and colleagues found—using models with extensive statistical controls—that, nationally, STEM faculty generally use pedagogies that encourage higher-order, integrative, and reflective learning significantly less than faculty in non-STEM fields and, not coincidently, STEM seniors experience “deep approaches to learning” less than seniors in non-STEM fields (for descriptions of the three measures, see Nelson Laird et al. 2008). The differences were small for Higher-Order Learning, the scale that is concerned with analysis, synthesis, and judgment regarding evidence—relatively good news—but quite large for the Integrative and Reflective Learning scales.

The study by Nelson Laird and colleagues is a part of a larger body of work about students engaging in educationally purposeful activities—those educational practices known to positively influence valued educational outcomes, activities such as active and collaborative learning and those that involve much student–faculty interaction, as noted in many of the articles in this issue of Peer Review. We know of the positive impact of pedagogies of engagement not only on general student learning, but also on STEM learning, from years of research.

It is discouraging that, nationally, faculty in STEM fields tend to have lower expectations for integrative and reflective learning relative to other faculty, and that results from seniors reflect those differences. The Integrative Learning scale assesses how often students use ideas from various sources and courses, include diverse perspectives in class discussions or writing assignments, and discuss ideas from readings or classes with faculty members and others outside of class. The Reflective Learning scale is a combination of responses to questions about trying out different perspectives and thinking about one’s own beliefs. The kinds of intellectual self-reflection skills these questions are about are surely as important in the STEM disciplines as they are in other disciplines, but we see that STEM majors have far fewer opportunities to develop these skills than students in other majors. Indeed, one might argue that it is especially in STEM that students should acquire these skills, given the way empirical evidence tends to be seen as harder in science than in other disciplines. Discovering a bad premise or assumption and being open to other interpretations are just as important in STEM disciplines as elsewhere.

These results caused us to want to look more closely at STEM/non-STEM differences and to determine whether there are circumstances where STEM seniors buck the general trends and are as engaged or more engaged than their non-STEM peers.

Looking Deeper

For our analyses we used responses to the 2008 NSSE survey from 614 institutions. From these institutions we selected seniors in all STEM fields categorized by NSSE: biological sciences, computer science, engineering, physical sciences, and mathematics (27,428 STEM seniors). Then, to keep the comparison group similar at each institution, we limited non-STEM majors to seniors in the arts and humanities and the social sciences, fields that are represented on the vast majority of college and university campuses (46,178 non-STEM seniors).

We used seniors’ responses to the three Deep Approaches to Learning scales and, to capture a broader picture of their engagement overall, their scores on NSSE’s Active and Collaborative Learning and Student–Faculty Interaction measures. Active and Collaborative Learning covers how often students ask questions in class, work with classmates in and outside of class, tutor or teach others, participate in community service, and discuss ideas from class with students, family members, coworkers, and others outside of class. Student–Faculty Interaction captures how often they discuss grades or assignments, discuss ideas from readings or classes outside of class, talk about career plans, and work on activities other than coursework with their faculty. It also includes how often students receive prompt feedback from faculty on their academic performance. In addition to broadening the focus, these extra measures tap aspects of engaged pedagogy much discussed and advocated among STEM reformers (Fairweather 2009).

The first thing to report is that the STEM/non-STEM differences Nelson-Laird et al. reported for the 2005 NSSE data were reaffirmed in the 2008 data—a small negative difference (i.e., the STEM average was below the non-STEM average) on High-Order Learning, but larger negative differences for Integrative and Reflective Learning. The difference was also small and negative for Student-Faculty Interaction, but small and positive (i.e., the STEM average was above the non-STEM average) for Active and Collaborative Learning. All differences were statistically significant.

Beyond the general effects, we were interested in what was happening at the institutional level. To examine this, we ran hierarchical linear models that produced adjusted means and estimated the STEM/non-STEM difference for each engagement measure at each institution. The models adjusted for the number of students at the institution (a data quality issue) and student characteristics (gender, race, first-generation college student status, living on campus, transfer status, foreign citizenship, full-/part-time status, Greek affiliation, and STEM/non-STEM major).

Our next step was to select the one hundred top scoring institutions (those with the highest adjusted means) for each of the five engagement measures and compare the composition of those groups of one hundred institutions to the composition of all 614 institutions (see table 1). What we see in table 1 is that Research Universities are underrepresented on all measures—indeed, no Research Universities make the top one hundred for Higher-Order Learning or Student–Faculty Interaction. Baccalaureate Arts and Sciences institutions, on the other hand, are overrepresented in the top one hundred on all measures. Baccalaureate Diverse institutions are underrepresented among the top one hundred institutions for the three Deep Approaches to Learning measures and overrepresented among the top one hundred for Student–Faculty Interaction. Private institutions are greatly overrepresented among the top one hundred institutions on all measures. With regard to selectivity, institutions ranked as Very Competitive or above on Barron’s selectivity rating are overrepresented on the three Deep Approaches to Learning measures, but they are slightly underrepresented on Active and Collaborative Learning and Student–Faculty Interaction.

PRSU11_Fig1.jpg

Institutions underrepresented in the top one hundred columns generally have higher than average proportions of STEM majors—a drag on their scores. This is particularly true for Research Universities where there is near parity between the proportions of STEM and non-STEM majors as we have defined them. An additional drag on Research Universities’ scores is their relatively high proportion of engineering majors among the STEM majors, as engineering seniors have been shown to score particularly low on deep approaches to learning and student–faculty interaction (NSSE 2003; Nelson Laird et al. 2008). These findings are also consistent with work that shows that institutions with a high percentage of departments offering both undergraduate and graduate education have lower levels of undergraduate student engagement (McCormick, Pike, Kuh, and Chen 2009).

Next we sorted the institutions by their STEM effect sizes to identify those institutions in the top one hundred on each variable that had minimal or nonexistent STEM/non-STEM differences (effect sizes greater than -0.1 and less than 0.1)—looking, in other words, for institutions where engagement was comparable for the two groups. Table 2 shows us that, with regard to Integrative and Reflective Learning, STEM seniors scored more than 0.3 standard deviations lower than non-STEM seniors in 99 percent and 98 percent of all 614 institutions in our sample, respectively. Those proportions are only slightly smaller for the top one hundred institutions. There are essentially no institutions where there is not a STEM/non-STEM difference on these scales!

PRSU11_Fig2.jpg

For Higher-Order Learning and Student–Faculty Interaction, most institutions in the total sample and the top one hundred have small STEM/non-STEM differences. For Active and Collaborative Learning more than half have small STEM differences, but the distributions favor STEM disciplines: that is, on this measure, STEM seniors score higher than non-STEM seniors at more institutions than the reverse. Interestingly, the top one hundred institutions were more likely to have negative STEM differences for Higher-Order Learning and less likely to have negative STEM differences for Student–Faculty Interaction. A look at table 2 with an eye toward getting an institution into the top one hundred reveals at least three possibilities: (1) pull an institution’s overall average up by improving non-STEM student engagement (see the greater proportion of top one hundred institutions with negative effect sizes for Higher-Order Learning), (2) pull the overall average up by improving STEM student engagement (see the greater proportion of top one hundred institutions with positive effect sizes for Active and Collaborative Learning), and (3) increase student engagement as needed to equalize STEM/non-STEM differences (see the greater proportion of top one-hundred institutions with trivial differences for Student–Faculty Interaction).

We posit that the last option should be institutions’ preferred method and that it should cut across lots of measures of student engagement. So our final task was to determine whether any institutions were in the top one hundred across multiple measures and had small STEM/non-STEM differences. Since no institutions had small differences for integrative and reflective learning, we focused on the other three measures. Only ten institutions were in the top one hundred for Higher-Order Learning, Active and Collaborative Learning, and Student–Faculty Interaction and also had small STEM/non-STEM differences across all three measures. These Engaging Ten are all private, less selective (Barron’s selectivity rating of Competitive Plus or below), and Master’s (6) or Baccalaureate Arts and Sciences institutions (4).

Encouraging Results—Lessons Moving Forward

While our analyses point to some discouraging results, there are also several encouraging trends for STEM education reformers. First of all, we are excited that large fractions of top one hundred institutions on three measures—higher-order learning, active and collaborative learning, and student-faculty interaction—have minimal STEM/non-STEM differences. This indicates that at many high-performing institutions, students across fields are experiencing the benefits of an engaging educational experience. This is a signal that institutions can foster what might be called a single “culture of engagement” on campus, one that diminishes the impact of those disciplinary teachings and traditions brought in by faculty that actually hamper undergraduate engagement and learning (e.g., a preference for passive teaching methods).

Our results also suggest that the use of active and collaborative learning practices by STEM seniors may be the crowning achievement of the STEM reform movement to date. At 95 percent of the top one hundred institutions on this measure and 91 percent of all institutions, STEM senior averages are greater than non-STEM senior averages. It appears that efforts to increase the use of active and collaborative pedagogies throughout STEM fields have taken root. To further document this result, we plan analyses that will map the trend in the use of active and collaborative learning in STEM fields over the past decade.

Another encouraging result is that, with the exception of the Research Universities, all institutional types (public and private, more and less selective) are represented in the profiles of high-performing institutions. In addition, the institutions that are most engaging across multiple measures and have small STEM/non-STEM effects are private, but not elite, supporting the notion that making an institutional commitment to get undergraduate education right—across STEM and non-STEM fields—may, in the end, be more important than selectivity and the financial resources that typically accompany it.

Indeed, evidence suggests that educational experiences focused on engagement and deeper learning, as opposed to passive teaching methods and “weeding students out,” achieves better outcomes and does not necessarily cost more (DeHaan 2005; Fairweather 2009). Even if such educational experiences do cost more than passive and weeding-out methods on the front end, better learning outcomes, improved satisfaction and retention, and shorter time to degree completion should offset any additional up-front investment through lower long-term costs to institutions and families and greater achievement of educational missions. Spelling out the specific costs and benefits is another area where additional work can fruitfully be carried out.

Alongside these encouraging results, however, is news that STEM seniors lag well behind non-STEM seniors in integrative and reflective learning at nearly all institutions. These approaches to learning, we and others (AAC&U 2007; National Research Council 2003) argue, are central to good scientific thought and critical components of what makes for a well-prepared college graduate. If active and collaborative learning was the area receiving most attention in the past few decades, it seems time to turn reformers’ energies toward integrative and reflective activities and move our institutions toward cultures of engagement that span a spectrum of sound educational practices. Doing so may be the next step in what DeHaan refers to as the “revolution” in undergraduate STEM education.

 

References

Association of American Colleges and Universities. 2007. College Learning for the New Global Century: A Report from the National Leadership Council for Liberal Education and America’s Promise. Washington, DC: Association of American Colleges and Universities.

DeHaan, R. L. 2005. “The Impending Revolution in Undergraduate Science Education.” Journal of Science Education and Technology 14(2): 253–269.

Fairweather, J. 2009. Linking Evidence and Promising Practices in Science, Technology, Engineering, and Mathematics (STEM) Undergraduate Education: A Status Report for the National Academies National Research Council Board of Science Education. East Lansing, MI: Center for Higher and Adult Education, Michigan State University.

Martin, E. 1991. “The Egg and the Sperm: How Science Has Constructed a Romance Based on Stereotypical Male-Female Roles.” Signs 16: 485–501.

McCormick, A. C., G. R. Pike, G. D. Kuh, and D. P. Chen. 2009. “Comparing the Utility of the 2000 and 2005 Carnegie Classification Systems in Research on Students’ College Experiences and Outcomes.” Research in Higher Education 50:144–167.

National Research Council. 2003. BIO2010: Transforming Undergraduate Education for Future Research Biologists. Washington, DC: National Academy Press.

National Survey of Student Engagement. 2003. Converting Data into Action: Expanding the Boundaries of Institutional Improvement. Bloomington, IN: Indiana University Center for Postsecondary Research.

Nelson Laird, T. F., R. Shoup, G. D. Kuh, and M. J. Schwarz. 2008. “The Effects of Discipline on Deep Approaches to Student Learning and College Outcomes.” Research in Higher Education 49: 469–494.


Thomas F. Nelson Laird is an associate professor; Alexander C. McCormick is the NSSE director and associate professor—both of Indiana University, Bloomington; Daniel F. Sullivan is the president emeritus; Christine Zimmerman is the director of institutional research—both of St. Lawrence University.

Previous Issues