Dive Deep

When I first transitioned to my current role running the Office of Institutional Research after more than a decade as a faculty member at California State University, Northridge, I attended all the major higher education conferences, eager to find my place and learn about the big-picture issues affecting colleges and universities. I was intrigued by sessions on student success data, especially those focusing on equity across demographic characteristics such as race, gender, and socioeconomic status. The California State University chancellor had declared in 2016 that we would close all equity gaps in graduation rates by 2025, and this felt like a personal directive as I began my new role.

After a few conferences, though, I became disappointed by the lack of data-informed solutions offered to eliminate the disparities in educational achievements. Presenters would spread the message to disaggregate the data—to break out student outcomes by key characteristics such as race and gender—to reveal equity gaps. But they never proposed ways to use these same data to identify ways to close those gaps. Disaggregating is certainly a critical first step toward equity, but it should be viewed as just that: a first step. It only reveals the symptoms; it does not diagnose the underlying causes. Disaggregated graduation rate data may tell us that Black students have lower graduation rates compared to their White peers. However, the why is not evident.

I therefore propose “deep data” as a critical next step. Simply disaggregating outcomes data, such as graduation rates or test scores, results in a relatively shallow understanding: for example, disaggregation of data at Northridge reveals that 80 percent of our Latinx first-time college students stay into their second year. But deep data work—diving into particular situations or individual characteristics—allows us to go beneath this shallow surface layer: for example, among Latinx first-time college students at our university, a specific demographic (male-identifying, first-generation college students) with certain admissions characteristics (high school GPA below 3.00) are retained at the lowest rates (63 percent), unless they experience a particular kind of cocurricular programming (such as a peer mentorship program).

The goal in deep data work is to understand what underpins the inequities revealed when student success data such as retention and graduation rates are disaggregated to see how those outcomes came about and, ultimately, to identify strategies to address the inequities.

Deep data analysis requires collaboration. A community-based approach that has been used for decades that is well suited to deep data work is action research, defined by Ernest Stringer in 2007 as “a collaborative approach to inquiry or investigation that provides people with the means to take systematic action to resolve specific problems.” Action research is different from traditional academic research: One of the first steps in action research is to establish jointly agreed-upon goals for the work (such as identifying curricular barriers to degree completion). It’s also important that no particular type of expertise is held above others (so contributions of students to the research is equally valued to that of tenured faculty).

Deep data work involves a wide array of both qualitative and quantitative information such as institutional data, including course grades and student demographics, as well as classroom assessments and student experiences gathered in surveys and focus groups. Analysis of such a wide swath of data requires the collaboration and engagement of key stakeholders across the university, including students and faculty of all ranks. Deep data collaborators also include those who may not always be top of mind: for example, the director of the math tutoring center, academic advising staff, coordinators of programming such as new student orientation, and other staff who interface with students directly. Having a broad range of collaborators helps to identify the relevant data to collect and examine—an academic advisor, for instance, might suggest exploring reasons students choose particular majors. Collaborators also provide multiple perspectives on how to interpret the findings—the advisor might help interpret misconceptions that first-generation college students have about some majors that result in misalignment between students’ expectations and faculty goals, while students might weigh in on when and where key information about majors would be most effectively shared.

And, while I recognize that I bring a healthy amount of bias to this point, I cannot emphasize enough the importance of engaging your institutional research/effectiveness (IR/IE) colleagues as equal partners in deep data work. Not only do IR/IE staff manage and report on the academic data for the institution—including course data, student demographic data, student success data, and even data of which you may be unaware, such as where students enroll if they leave the institution—they can also provide insights from their vast experience as data scientists. For example, they can reveal misinterpretations of the data due to confounding variables. And as the data mount up, IR/IE staff can connect the patchwork of information from different databases via student identifiers—as a data hub to ensure that the intersections among the data are not lost—and keep the data secure.

So what does working with deep data using an action research approach look like in practice? It is a circular process: in other words, the questions you ask help with understanding the pieces of data being studied but also feed further inquiry. For example, excavating inequitable student outcomes such as low retention rates among minoritized students leads us not only to find ways to address racial inequities but also to identify unforeseen paths of inquiry such as academic pathways in which students get stuck or campus-wide programs that may particularly benefit students of color.

Take an example from Northridge. Over the course of three years, the data literacy program Data Champions guided more than eighty faculty and staff to examine data on issues they believed could help explain different student outcomes, including the reasons some courses had persistent high failure rates and how the career paths of students who changed majors diverged.

In one point of entry, the process began with ethnic studies departments analyzing a data set resulting from an observation that in a large proportion of courses in their disciplines, students of color outperformed their White peers (an outcome sometimes referred to as a “reverse equity gap”). Data Champions examined the characteristics of these courses and found that when the race of the instructors was the same as that of a large proportion of their students, students experiencing the race match had an increased likelihood of earning a high grade. This is particularly revealing on a campus like Northridge, where more than 70 percent of our students come from racially minoritized groups. But the inquiry didn’t stop there. Data Champions dived deeper into the whys by reviewing syllabi, observing courses, and interviewing students. One additional why turned out to be that students in these courses felt that their own identities were represented in course materials, such as reading materials that used examples from a wide variety of cultures. This in turn fueled questions about courses in other disciplines with similar outcomes. A next step involves conducting interviews with instructors with an eye toward creating racial equity–focused programming and identifying best pedagogical practices for faculty development.

This is just one example. So much more can be done. By working with deep data, we can move beyond simple disaggregation and employ this powerful tool to provide a more equitable and inclusive experience for students at our institutions—and leverage the well of expertise that already exists on our campuses.

Illustration by Jon Han