AAC&U News | July/August 02

Making Sure Science Skills Don't Rust
Penn Tests New Science Literacy Assessment Tool and Measures the Impact of a New General Education Program

Although calls for improved assessment mechanisms and greater accountability within higher education are nothing new, the media have recently turned the public's attention to assessment debates in higher education. According to a recent article in The New York Times, "[I]t is no longer enough for a university to say that it accepts only students with high SAT scores; those scores say nothing about what students learn once they're in."

Many colleges and universities, of course, have been quietly retooling their curricula and finding better ways to assess what students are actually learning in college. The University of Pennsylvania (Penn) is testing the effectiveness of their curriculum changes in a variety of ways. One way is to test the level of students' science literacy.

"We wanted to test science literacy coming in and going out," says Ingrid Waldron, professor of biology and member of the Pilot Curriculum Evaluation Committee at Penn. At a school where many students arrive with AP credits through which they "test out" of core science courses, the survey is a way to measure the science knowledge and understanding the students bring with them, and what happens to that core of science literacy during their college careers. Adds Waldron: "A lot of what's out there assessment-wise is high school level; we were trying to get at reasoning ability and statistical concepts—in addition to basic knowledge."

With the new science literacy survey, Penn also seeks to compare the science literacy of students in a new pilot curriculum and those enrolled in their more traditional curriculum. Two hundred students were randomly selected to participate in the new pilot program for the College of Arts and Sciences instituted in 2000. Students enrolled in the pilot program work closely with an advisor to design a course of study that is developed from a more compact but more carefully focused general education experience. In place of traditional introductory courses in a discipline that often serve also as general education courses, pilot students take newly created, cross-disciplinary general education courses, such as "Quantum Theory and Its Philosophical Implications," taught by faculty from both the chemistry and philosophy departments.

Penn's science survey, administered during orientation, is brief, requiring about twenty minutes to complete twenty-three multiple-choice questions to test basic science skills. The test also includes some general information questions in order to categorize the respondents. The results are compared for students in three groups: those who are part of the new "Pilot Curriculum," students who applied for the pilot program but were randomly selected to be in the standard curriculum, and all other students. When the test was first administered to all freshmen in September 2001, the school received usable surveys from 83 percent of the students. On average, the students gave correct answers for 55 percent of the science literacy items on which they were tested. The creators of the survey look forward to testing the same class during their junior year spring semester in order to compare the progress of the students in the pilot curriculum with the other students.

So now that we know how literate the students are, how successful is the survey itself? "So far, so good," is Kent Peterman's analysis of the survey's success. Peterman serves as the director for Academic Affairs of the College of Arts and Sciences and as member of Penn's Committee on Undergraduate Education as well as a member of the Pilot Curriculum Committee. Coming up with the best college-level questions was not easy. According to Peterman, "We asked items of appropriate difficulty. They didn't all ace it and they didn't all bomb out, either." The hope is that the survey will be "sensitive enough to detect small differences" when the tool is administered to the same group of students in their junior year. In addition, Penn found that scores on the science survey correlated with the total number of AP and college-level science and math courses that incoming students took in high school. In short, the more such classes students took, the higher their scores.

"For science majors, we don't expect to find a big difference between the levels of science literacy for students in the pilot curriculum versus our traditional curriculum," according to Waldron. It is the non-science majors they are watching to see if the two curricula promote different levels of science literacy. "We are particularly interested to see if the non-science majors graduate with increased science literacy."

A description of the Science Literacy Survey and some findings from its initial administration are available at www.sas.upenn.edu/faculty/Teaching_Resources/sci_literacy_survey.html. Copies of the Survey itself are available by contacting Dr. Kent Peterman, College of Arts and Sciences, 120 Logan Hall, University of Pennsylvania, Philadelphia, PA 19104-6304 (peterman@sas.upenn.edu) (215-898 -7867).

For more information on Penn's Pilot Curriculum, visit www.college.upenn.edu/pilot_curriculum/.

For information about AAC&U initiative, Science Education for New Civic Engagements and Responsibilities (SENCER) including model general education courses for the project, visit www.aacu.org/sencer/index.cfm.