Liberal Education

Accountability "Light": Our Version Is Going the Way of the Dollar vs. the Euro

It was July 14, 2006, when the Spellings Commission on the Future of Higher Education posted the second draft of its final report on the Web for all to see. It was the first of the commission’s drafts in which the alleged illiteracy of U.S. college graduates was placed in bold space along with all the other flagellations of higher education in which the commission delighted. The text of the Web posting, though, was an unhappy validation of the complaint: the cases of violation of subject-verb agreement leaped off the electronic pages, along with paragraphs that can be kindly described as strings of non-sequiturs. The commission’s draft would not be awarded a C- in English Composition on a generous grading day. Yet no one noticed the irony. After all, this was a distinguished panel. Higher education had not seen a federal undertaking such as this since 1984, and it was getting visible attention in the general press for topics other than college costs.

But it would not have been hard to notice, in both the Web-posted drafts and the final report of the commission, that the only references to higher education outside the United States recited ill-informed data putatively demonstrating how far we had “fallen behind” other nations.1 Nobody on the commission or its staff had bothered to examine the ways other countries are dealing with issues we also face—access, degree completion, and, most of all, accountability. In its insular and chauvinistic way, this major pronouncement on, and recommendation of remedies for, the state of U.S. higher education pretended, without looking, that the world is learning more, and we are not—a conclusion designed more to fit our appetite for bad news and self-degradation than to enlighten.

Now, if the rest of the world is learning more, would that be surprising? And in a global economy, would it really be so disappointing? Culture and language ensure that the world is not flat, but in terms of knowledge it is, and the world’s knowledge content, like the oceans, is rising. To the extent that the world learns more, we all benefit, and we cannot pretend that the United States holds a monopoly on knowledge. More seriously, because credentials awarded are used as a proxy for learning in international—as well as domestic— comparisons, nobody knows for sure whether students in other countries are, in fact, learning more. But nowhere in recent years have academic leaders, faculty, and students themselves wrestled more with the knots of credentials and learning than in the old nations of Europe. Through the largest reconstruction of higher education ever undertaken anywhere—known as the Bologna Process—they have come up with a convincing and credible scaffolding of accountability. When one considers what the Europeans have done—and where they have more to do—one realizes that we in the United States have a long way to go in understanding what “accountability” means.

The Bologna Process

The Bologna Process is named for the city where, in 1999, the education ministers of twenty-nine countries first reached a meeting of the minds on the future of higher education in Europe. It is an inevitable outgrowth of the process of European integration that began in 1950. By the 1990s, and certainly after the Maastricht Treaty of 1992, Europe began to resemble a quasi-federal arrangement. Although there were few economic borders, the common European workforce was, ironically, stuck within political borders because individual countries did not yet fully recognize—or even understand—their neighbors’ education credentials. In order to allow for the recognition of higher education credentials across borders and, thus, to provide mobility for the advanced knowledge workforce, some convergence of educational practices and standards was needed.

Bologna became a force for converting education systems into similar forms and processes. Its discourse is a type of technology transfer that brings nations from different platforms of educational development to the point of embracing compatible paradigms. In other words, they wind up singing in the same key—though not the same tune. With its simultaneous respect for distinctive national traditions and institutional autonomy, the Bologna Process became doubly attractive as the only game in town.

In 1999 and subsequently—forty-six countries are now involved—the ministers agreed to an action agenda for dissolving educational borders in the same way that economic borders had been dissolved. The agreement was an inevitable consequence of reforms that had been stirring across European education during the 1990s. The feeling was that, having lost their way and their world leadership, ancient systems of higher education needed a kick-start. In 1998, the ministers of the four largest countries in Europe—France, Germany, Italy, and the United Kingdom—met at the Sorbonne in Paris and planted the seeds from which the European Higher Education Area has grown. The Bologna ministers optimistically set 2010 as the date by which the transformations they imagined were to be realized. But inevitable inertia and resistance at the institutional level, new provisions, and additional partners have rendered the 2010 completion marker a mirage; 2020 is more likely.

The Bologna Process is a huge undertaking. Its action lines include revising degree cycles, establishing degree qualifications frameworks, overhauling the credit system, increasing access via more flexible paths into and through higher education, validating student attainment, establishing a quality assurance system, and increasing the attractiveness of the European Higher Education Area for foreign students. Some four thousand institutions, which enroll sixteen million students, are now participating. Some of what has been accomplished has already been adapted in Latin America, North Africa, and Australia. Indeed, the Bologna model is well on the way to becoming the dominant global paradigm for higher education by roughly 2025. We cannot afford to ignore it.

Real accountability

The Spellings Commission report and the response to it from within the U.S. higher education community give the impression that accountability is a very simple matter. In order for colleges and universities to be “accountable” to those who subsidize them or pay their tuition and fees, they need only to make public their graduation rates, demographic mix, job placement rates, and some data from the National Survey of Student Engagement on what their students like about opportunities and services. They may perhaps also throw in a test or two to show that a sample of their students know how to write or solve a problem or that a random set of one hundred student volunteers produced an X-standard deviation improvement between entrance and graduation on a test of “critical thinking” (which improvement, of course, is attributed entirely to the institution2). But this is show, not substance. Wish lists for student learning, flowery goals statements, and agendas of commitment for the twenty-first century are sometimes offered in parallel rubrics and posed as accountability statements. But none of these efforts explains what credentials represent or what students must do to earn them. There are no public reference points or performance criteria. Students—our constituents who ought to be our partners as well—played no role in fashioning these efforts. And given how divorced these various approaches are from the actual work of most faculty and their students, their likely impact on student learning is limited indeed.

So what are the Europeans doing differently? They have essentially created an accountability loop that is executed by national systems and institutions. Although still a work in progress, the European accountability loop begins with degree qualifications frameworks and circles back with a document that serves as a public warranty, attesting to the institution’s judgment and the student’s achievement. In between are credential qualifications frameworks at the disciplinary level and a student-centered credit system to which a growing number of institutions have added levels of challenge. Countries outside of Europe have already adapted some of these elements for use within their national systems or institutions, and it is worthwhile to consider whether some elements might usefully be adapted to higher education in the United States.

Qualifications frameworks

Within the context of the Bologna Process, a qualifications framework is a statement of the learning outcomes and competencies that a student must demonstrate in order to be awarded a degree at a specific level. It is not simply a statement of objectives or goals; it is not a wish list. Rather, it is a set of performance criteria. When an institution of higher education is governed by a qualifications framework, it must “demonstrate” that its students have “demonstrated”—all of its students, not merely a hundred volunteers who take a standardized test. While a qualifications framework does not dictate how that demonstration takes place or the nature or form of assessments employed, it does provide learning outcome constructs within which the demonstration is conducted.

A second key characteristic of a qualifications framework is that it clearly indicates the criteria for each level of credential offered. The language of the framework accomplishes this through a “ratcheting up” of learning benchmarks—that is, by increasing the level of challenge at each rung on the degree ladder. This “ratchet principle” pervades all of the content challenge and performance statements of Bologna, from individual courses to degrees, and has penetrated the credit system as well. This principle embodies content and performance standards, and it functions as an engine of accountability.

There are three levels and types of qualifications frameworks under Bologna: transnational, national, and disciplinary/field. The trans­national Framework of Qualifications of the European Higher Education Area provides a wide-angle set of generic markers that clearly distinguish among the “short-cycle” degree (the equivalent of our associate’s degree), the bachelor’s degree, and the master’s degree. The distinctions are drawn not in terms of a number of credits or a minimum grade point average, but rather in terms of student learning outcomes across five bands of requirements: knowledge and understanding; application of knowledge and understanding; fluency in the use of increasingly complex data and information; breadth, depth, and range of audiences for communication; and degree of autonomy gained for subsequent learning. The language used to describe these requirements “ratchets up” the level of challenge as one moves up the degree ladder. This framework has been accepted by all forty-six countries participating in the Bologna Process.

One might expect each national higher education system to develop its own compatible version of the Qualifications Framework for the European Higher Education Area—adding detail, taking into account the peculiar varieties of its own institutions as well as their historical missions and commitments, and perhaps including “intermediate” qualifications between the three degrees. In practice, however, that’s not how the national qualifications frameworks have developed. Creating consensus on a national framework is a time-consuming challenge, and to date, only seven of the forty-six participating countries have completed the task. Among them are five highly distinct models that illustrate the way the Bologna countries can achieve “convergence” with variation. The Irish and Scottish framework is comprehensive, from kindergarten through the doctoral level, while the German framework offers a more parsimonious articulation of how university students must demonstrate knowledge through what are called instrumental competencies, systemic competencies, and communicative competencies. The Dutch framework references labor market positions and tasks, and the Swedish framework reaches into nineteen specific applied disciplines that lead to licensure occupations (e.g., nursing). The French framework requires every degree program in the country to undergo a central registry review, and the program dossier is made public via the Internet.

Tuning and benchmarking

The disciplinary qualifications frameworks are developed through what is called the “tuning” process. “Tuning” is a methodology for producing “reference points” that faculty can use in developing statements of learning outcomes, levels of learning, and desired competencies in the disciplines. The purpose is to ensure that these statements are both transparent and comparable. This does not mean that the content is standardized, however. The criterion-referenced competency statements that result from the tuning process are not straightjackets. To reprise the music metaphor: everybody winds up with the same music staves, range of time signatures, tempo commands, and system of notation. Then, all programs in the same discipline sing in the same key—engineering in A-minor, business in B-flat—but they don’t necessarily sing the same tune. The first phase of the tuning project involved nine fields in 138 institutions across sixteen countries. In the second phase, sixteen other fields joined the model. Tuning is the largest Bologna “export” to date: twelve disciplines in 182 universities in eighteen Latin American countries have adopted the methodology.

However attractive “tuning” may be, it is not always executed to match its purpose. Faculty members have difficulty writing criterion-referenced learning outcome statements, and that’s something the Europeans—and we in the United States—need to work on. Discipline-based benchmarking, a strong suit of the United Kingdom’s Quality Assurance Agency (QAA), is an especially prominent alternative. Like tuning, benchmarking provides reference points and boundaries for designing, modifying, and evaluating the presentation of a discipline within an institution or a group of similar institutions. The QAA benchmarking statements for degrees in accounting, for example, clearly describe the subject-specific knowledge, skills, and cognitive operations that the graduate will have “demonstrated” at a level crossing a threshold standard. While institutions choose their own forms of assessment and set their own thresholds, a student who does not “demonstrate” does not earn
a degree.

So what can we learn from the European experience with qualifications frameworks, tuning, and benchmarking? Accountability begins with the establishment of public definitions of degrees and criterion-referenced statements of performance. Once such definitions are established, the credential-awarding institution can say, “this is what this student did, and this is what the degree represents.”

The importance of discipline-specific content has been a consistent theme of the Bologna Process. After all, students go to college to earn a degree in anthropology, mechanical engineering, or nursing; they go to community colleges to earn a degree in medical technology or commercial art. When you ask them what they are studying, they will answer by telling you about their discipline or field. And, of course, faculties the world over are also organized by discipline or field. When our students enter a major—and when they graduate—they deserve to know what they are in for, and, later, what they have accomplished. Students are not likely to get this clarity from the institutional graduation rates, indicators of student engagement, “value-added” metrics of “critical thinking,” and generalized curricular goals one finds in so many “accountability” pronouncements in the United States. Our students deserve better.

Euro-Credits

While the European Credit Transfer System (ECTS) is built on a fundamentally different assumption from that used for U.S. higher education credits, if it plays out in its ideal form, it will go beyond functioning as a measure of accumulation and become another accountability tool.

In the United States, credit assignments are based on faculty contact hours, with the assumption that, in relation to each faculty contact hour, the student engages in other types of learning activities. In contrast, ECTS uses the student as the primary reference point, asks how many hours the average student must spend to accomplish the various tasks in a course module, and converts the total to credits. The conversion formula is based on the length of the academic year, the total number of hours available for study during that year, and a reference marker of sixty credits per academic year. The conversion formula ranges from twenty-five to thirty workload hours per credit.

The virtues and drawbacks of such a system are obvious. When faculty really have to think about what they are asking students to do, there is strong potential for curricular change. Redundancies, gaps, and opportunities for cyber-efficiencies can be identified, and appropriate changes in coverage and delivery can be made. The Bologna literature describes such reforms in fields ranging from music to chemistry. The obvious drawback can be simply phrased: when you have a formula, it becomes the default. Water finds the easiest way to flow downhill, and both academic administrators and faculty become much too mechanical and downright sloppy in assigning credits to course modules.

How does one connect student workload to learning outcomes, to the principle of the ratcheting up of challenge and, hence, to qualifications frameworks and the structure of accountability? We don’t worry about such things in the United States. We grant three credits for Introduction to Sports and three credits for Econometrics or assign nine credits for a Recreation Practicum and four credits for Neuropsychology—and brush such obvious dissonances under the rug. Nobody—not the Spellings Commission or the various “accountability”-oriented responses of the U.S. higher education community—has evidenced any desire to acknowledge, let alone repair, these obvious holes in the academic integrity of our credit system. In the United States, credits live in the office of the vice president for finance, not the vice president for academic affairs; the student is incidental to it all. If we care about accountability for student learning, and if we are serious about matching practice to rhetoric, then we may need a redesign. If ECTS plays out in ideal form even among a plurality of Bologna participants, it will show us the way.

Given different modes of student work in the disciplines, our European colleagues have gone about the task of linking workload to learning outcomes with alternative proxies. In the most intriguing of these approaches, that taken by the United Kingdom and Scotland, credits are placed within degrees of challenge. That link—between the measure of estimated student time-on-tasks and the level of demand inherent in those tasks—creates a “credit level” defined as “an indicator of the relative demand, complexity and depth of learning and of learner autonomy.”3 There are nine such credit levels, and each carries a generic description that is independent of discipline but applicable to all disciplines. Every course is tagged with a credit level, and the number of credits awarded is treated as a separate issue. Once levels such as these are established, degree qualifications can be set in terms of minimums—40 percent of credits at level six, for example, and 65 percent of credits at levels five and six. If all colleges and universities in the United States added such indicators of the challenge of content to credits, quarrels about credit transfer would diminish considerably.

Diploma Supplements

After qualification frameworks, tuning, and an ideal credit system marked with parallel structures of challenge, what is left in this very different scaffolding of accountability? What evidence of learning and attainment does the graduate carry forward into the world, and how is that evidence communicated for all to see? The piece of paper we call a diploma does not say much, and our European colleagues have added something very important to it.

As the final element in the Bologna accountability loop, the Diploma Supplement serves as a kind of warranty, providing evidence of the graduate’s learning and attainment. In addition to including a transcript and indicating superior performance (i.e., honors), the Diploma Supplement is intended to convey information about

  • the national system in which the degree was awarded and the position of the degree within that system’s credentialing hierarchy;
  • the status, type, and accreditation of the institution awarding the degree;
  • the purpose and function of the credential;
  • the student’s major field of study as well as the duration and the entry requirements of the program in which the credential was granted;
  • discipline-level qualifications and degree requirements, including internships, theses, and final projects;
  • modes of study, including enrollment intensity and distance learning.4

Even if they included only the information listed above, Diploma Supplements would go some way toward providing public assurance about higher education credentials. Some serious revisions are needed, however. The absence of discipline-level qualifications statements, for example, is a very unhappy omission in light of the qualifications frameworks core of Bologna. And the intended warrantee, the student, plays too minor a role in the current version of the Diploma Supplement for the loop to close.

In addition to providing a public warranty for our students, the creation of a U.S. version of the Diploma Supplement would put institutions of higher education on the public record in terms of their standards for degree qualifications, and it would hold them to consistency. Setting aside the transcript as a separate document (as it should be), what should our version of the Diploma Supplement contain in addition to the bullets above covering the institution, the degree and its function, modes of study, and “compressed signals” of superior academic performance? I advocate the following:

  • Not-so-standard boilerplate indicating all other institutions attended by the student from which credits were accepted as well as the percentage of the student’s credits that were earned at the institution awarding the degree
  • A tuning-type qualifications statement for the student’s major and information about any required internships, theses, final projects, portfolios, or comprehensive examinations
  • The title and a short description of the student’s thesis or final degree-qualifying project, if applicable
  • Any external certification examinations passed or licenses granted to the student (although the institution is not the awarding body in these cases, the institution certifies that it has recognized and recorded them)
  • A maximum of two noteworthy and documented services performed by the student for the institution or its surrounding community
  • Student research, creative, or service participation, if applicable (field, title of project, and faculty sponsor)
  • Documented proficiency in languages other than English, including method of documentation

To be effective and credible, these markers on a Diploma Supplement should be limited, based on unobtrusive institutional records of the student’s activities, concentrated on achievements related to the degree awarded, and verifiable and validated by whoever signs the document on behalf of the institution. Otherwise, they are properly part of a resume.

Such a Diploma Supplement would close the accountability loop. To repeat: it is both a public warranty and a private assurance of the meaning of the degree, of the standards for awarding it, and of what the student did to earn it. Thus the loop returns to the public definition of degrees and public criterion-referenced statements of performance, with real students carrying the evidence of having met both. This would be a far more convincing enactment of accountability than what we have seen in the United States over the past two years.

The accountability framework that has been developed through the Bologna Process is worth learning from and thinking deeply about, and I urge academics in the United States to take it very seriously—before the “dollar” of their enterprise loses further value, no matter how many degrees they award.

 

Notes

  1. Comparisons among national systems rely principally on the methodology of the Organization for Economic Co-operation and Development (OECD) and its highly visible vehicle, Education at a Glance. Yet OECD itself has for some time been unhappy with its higher education indicators and, in the recently released 2008 edition of Education at a Glance, has made the first of a series of changes in data reporting: an indicator based on cohort graduation rates for first-time, full-time students in bachelor’s degree-granting institutions has been added. OECD’s traditional indicator for postsecondary attainment relies on population ratios—i.e., from census data or labor force surveys in each country, the number of people holding postsecondary degrees in relation to the total working-age population. Variations on this indicator also use population ratios for specific age brackets (e.g., twenty-five to thirty-four). The problems with this approach are numerous, beginning with different census methodologies and different definitions of degrees. This article is not the occasion for elaboration. For an overview, I recommend Jane Wellman’s Apples and Oranges in the Flat World: A Layperson’s Guide to International Comparisons of Postsecondary Education (American Council on Education, 2007). I also recommend reading the technical notes in Education at a Glance—particularly for the new indicator—as the notes demonstrate national variances in definitions, inclusions, and exclusions.
  2. Not considered are a host of potential intervening variables such as whether the student volunteers were transfers-in or attended other schools as undergraduates, whether they were parents and/or veterans, whether they held jobs that provided learning challenges, whether they practiced debate in Bible or Talmud study in religious institutions, and whether their significant others spoke languages other than English.
  3. Credit and HE Qualifications (Joint Credit Bodies for England, Wales, and Northern Ireland, 2001). For text of a sample of four of these credit levels, see The Bologna Club: What U.S. Higher Education Can Learn from a Decade of European Reconstruction (Institute for Higher Education Policy, 2008), 60.
  4. The official “Outline Structure for the Diploma Supplement” is available online.

Clifford Adelman has been senior associate at the Institute for Higher Education Policy since the fall of 2006. Prior to that, he served for twenty-seven years as senior research analyst at the U.S. Department of Education, and ten years in both teaching and academic administration.


To respond to this article, e-mail liberaled@aacu.org, with the author’s name on the subject line.

Previous Issues