Wineburg questions new study that sounds the alarm over student ignorance of history

Condemned to Repeat It

A new study sounds the alarm over undergraduates' ignorance of American history. Is it a crisis or a case of crying wolf?

November 24, 2006
Chronicle of Higher Education
By John Gravois

The study's findings were irresistibly damning, even if they did sound vaguely familiar: American colleges, supposedly the nation's hothouses for enlightened citizenship, in fact produce civic nincompoops.

That was the word in late September from the Intercollegiate Studies Institute, a Delaware think tank, when it released its study, "The Coming Crisis in Citizenship: Higher Education's Failure to Teach America's History and Institutions."

The think tank had hired researchers at the University of Connecticut to administer a 60-question, multiple-choice test to 14,000 freshmen and seniors at 50 different institutions. The average score among seniors on the test of American history, politics, and economics was 53.2 percent — a big, fat F, the institute suggested. What's more, seniors only scored 1.5 percentage points higher on average than freshmen. And at some institutions, including such power-houses as Brown and the Johns Hopkins Universities, seniors actually did worse than freshmen. "Negative learning," declared the institute, coining an ominous phrase. The press took notice.

"The national stockpile of dummies appears to be in no danger of running dry," wrote a columnist for The Boston Globe, one of several indignant pronouncements that appeared in editorial pages across the nation that week. Some news outlets printed the study's main data table, a list of the 50 universities ranked according to how much better (or worse) their seniors did on the test than their freshmen — a measure of "value added," in the think tank's terms. The story was not quite man bites dog, but it was almost as surprising: It was University of Mobile (ranked eighth out of 50) bites Yale (ranked 44th).

In an age obsessed with outcomes and assessment, the study was a broadside attack on higher education's curricular sovereignty. By going directly to the media with a multiple-choice test of what one of the think tank's leaders called "apple-pie American history" and a set of rankings, the study hurdled over a slow, sometimes-reluctant academic discussion about how best to characterize and measure historical knowledge.

But it also hurdled the kind of review process that would have led to publication in a major history or social-science journal, opting instead for a release at the National Press Club, managed by a New York public-relations firm whose clients have included Anheuser-Busch and American Express.

And once the news cycle had passed, some in academe began scratching their heads about the study's methodology and findings. Other professors and administrators wondered whether the Intercollegiate Studies Institute, an intellectually conservative organization that has long advocated more required courses and traditional curricula, was an objective source for research that seemed packaged to confirm the institute's own deeply held positions.

Regardless of any of these concerns, for Sam Wineburg, the whole episode was an all-too-familiar spectacle of hand-wringing over civic decline. A professor of education who specializes in history at Stanford University (ranked 31st by the study), Mr. Wineburg says there is one piece of history that everyone seems to forget: For the past century, tests that show young people to be "appallingly ignorant" have come around at least once a generation. When the alarm has been sounded so many times, does it begin to ring hollow?

Flunk by Any Other Name

Some of the news outlets that covered the study printed a few sample questions that the Intercollegiate Studies Institute had provided from its 60-question test, along with the average scores for those questions (see box). Most commentators regarded the sample questions as testing only the most basic civic knowledge.

The problem, according to many in aca-deme, is that the think tank has not released any more than that first handful of sample items from the test, making it hard to judge what the test's results really mean. The institute says it is still using the questions in a continuing round of testing, and so does not want to release them yet.

One finding from the study that did not make it into news reports was that only two students out of 14,000 got perfect scores on the test. "When only two out of 14,000 people get the exam 100 percent correct, it sounds like the test is designed to show what people don't know," says Rebecca F. Goldin, an associate professor of mathematics at George Mason University and the director of research at a media-watchdog group affiliated with her university. With the caveat that she has not been able to see the rest of the test questions, she says that to conclude that the average score of 53.2 percent is a "flunking" grade is just to trust the institute's opinion of what every student should know.

P. Michael Ratliff, a senior vice president of the institute, says it asked a group of faculty advisers, who are professors of history, economics, and political science, to "prepare a test that they thought students who had completed an intro course at the freshman or sophomore level would get right." He added: "We understood as well that it would make it more pointed and newsworthy to characterize these results as an F or an A."

As for criticism that the study was not peer-reviewed, Mr. Ratliff says it was vetted by that same group of advisers and that the institute might try to publish journal articles later that analyze the results. "We wanted it to be newsworthy," he says. "One of our objectives all along has been to contribute to public accountability of higher education."

The institute admittedly has a conservative intellectual position, and has long provided financial support to conservative campus publications. It has also published its own college guide for several years, called All-American Colleges: Top Schools for Conservatives, Old-Fashioned Liberals, and People of Faith.

However, says Mr. Ratliff, the institute does not devote any of its activities to partisan politics. And he asserts that when the institute commissioned the study, it "had no idea what the results were going to be."

The Mighty Brought Low?

The institute's ranking scheme naturally caused some consternation among those institutions that ranked low.

James J. O'Donnell, provost of Georgetown University (ranked 43rd), is not accustomed to seeing his institution fare so poorly in college rankings. So he was alarmed when the Intercollegiate Studies Institute singled out Georgetown in several of its press releases as a exemplar of "negative learning."

He was even more alarmed when he went to the study's data tables and found that by another means of interpreting the data generated by the study, Georgetown sits in the top 10.

Looking just at the average scores of seniors on the test, Harvard University ranks first, Princeton University ranks second, and Williams College ties with Grove City College for third. Georgetown students had the sixth highest average score. By this measure, elite schools come out looking pretty good.

The study's published rankings draw out drastically different results. Colorado State University, for instance, comes in second, heads and tails above more commonly prestigious institutions. Why? Because freshmen there earned a grade of 40.6 percent on the test and seniors earned a grade of 51.5. (That's 10.9 percentage points of "value added.") Put another way, seniors at Colorado State gave an average of 6.54 more correct answers on the test than did freshmen. But for all that improvement, the scores of Colorado State seniors were still below the overall average of 53.2 percent. (The first-place institution, Rhodes College, had an average score of 50.6 percent among freshmen and 62.2 percent among seniors.)

Harvard, meanwhile, despite having the highest senior scores, ranked a middling 25th place because seniors there only gave 1.14 more correct answers than freshmen did. And Georgetown landed way down in 43rd place because seniors gave about 0.72 fewer correct answers than freshmen. That the scores of freshmen at both those schools started out high was immaterial in the ranking scheme.

Gary Scott, a senior research fellow at the institute, says this approach was no accident. The study, he says, was not trying to determine "who can find smart people, but who can create smart people in this domain."

Mr. O'Donnell, however, says the study's ranking system simply rewarded schools that focus on what he calls "remedial history," rather than on developing the skills of close reading and critical thinking that Georgetown stresses.

Mr. Ratliff, the senior vice president of the institute, says he agrees that developing skills of analysis and synthesis should be the main focus of a college education. But knowing the basics, he says, should be a prerequisite.

A Peculiar Neurosis

In March 2004, Mr. Wineburg, the professor at Stanford, published an article in The Journal of American History titled "Crazy About History." The article begins by discussing a study that was published in the Journal of Educational Psychology in May 1917, the year the United States entered World War I and, according to Mr. Wineburg, "history's apex as a subject in the school curriculum."

In the 1917 study, two researchers compiled a test of names, dates, and events from American history, the kinds of items, writes Mr. Wineburg, "that history teachers said every student should know." The test was then administered to 1,500 elementary, high school, and college students at Texas institutions. The college students scored 49 percent on a test of what the researchers called "the simplest and most obvious facts of American history."

Then in 1943, Mr. Wineburg writes, a similar test of basic historical facts administered to 7,000 college freshmen yielded similarly terrible results. Only 6 percent of the 1943 subjects could identify the original 13 colonies, he writes. The greatest generation? Apparently not so great when it came to history tests. The New York Times editorial page harrumphed about the "appallingly ignorant youth" of the day.

In 1976 the Times and the Educational Testing Service collaborated on another test of history staples, this one multiple choice, that went out to 2,000 freshmen at 194 colleges. Average score: a promptly bemoaned 50 percent.

Mr. Wineburg's article then goes on to describe how, when the National Assessment of Education Progress was administered in 1987, 1994, and 2001, low scores in American history prompted the same rounds of editorial jeremiads about national decline.

"If anything," Mr. Wineburg sums up, "test results across the last century point to a peculiar American neurosis: each generation's obsession with testing its young only to discover — and rediscover — their 'shameful' ignorance."

Another lesson altogether, one covering the nature of historical knowledge, can be drawn from this look back, Mr. Wineburg says. "A liberally educated person is not someone who carries a repository of decontextualized information," he says. "The human mind needs pattern and form. We can ensure a very short half-life of factual material if it doesn't find form in a larger embracing historical narrative."

To demonstrate this point, Mr. Wineburg refers to a small study he once carried out. He gathered a handful of academic historians from various fields and a handful of high-performing high-school students. He asked both groups a set of straightforward identification questions ("What was Fort Ticonderoga?" "What were the Townshend Acts?"). The high school students often scored better than the professional historians on these. However, when he asked the two groups to make sense of a set of historical documents about the Battle of Lexington, the historians, not surprisingly, fared significantly better.

For all the fault he finds with the recent study by the Intercollegiate Studies Institute, Mr. Wineburg happens to agree with one of the organization's main prescriptions for higher education. He thinks there should be more required college history courses. "I think the facilities of historical thinking and reasoning, including the mastery of core content, are the prerequisite skills to effective citizenship," he says.

Mr. Scott, from the Intercollegiate Studies Institute, says that, in at least one significant way, his organization's research differs from the long line of studies Mr. Wineburg pillories in his article. "Our study is more positive in the sense that it identifies who alleviates ignorance; it doesn't point fingers at ignorance," Mr. Scott says. "The question is: Who lights the candle? Who enlightens students and who doesn't?"

Mr. Wineburg, however, says he would rather see an assessment, from a more impartial researcher, that tries more to measure students' historical reasoning skills. Based on what he has seen of the think tank's test, he says, how much value to ascribe to its measure of "value added" is an open question at best.