Carnoy says public academic disagreements over methods of data analysis not unusual

October 14, 2009
The Stanford Daily
By Melanie Scheible

Hoxby, CREDO spar over charter schools study

As the adage goes, it’s not what you do. It’s how you do it.

That’s the lesson playing out between Stanford economics Prof. Caroline Hoxby and researchers at the Center for Research on Education Outcomes, who disagree over whether the center’s study on charter schools really contained the statistical error that Hoxby alleged in a memorandum last week.

The center’s study, published this summer, compared the performance of students at charter schools to the performance of students in traditional public schools by measuring a number of achievement indicators. It featured profiles of hypothetical public school students created by averaging scores across demographics.

These hypothetical students’ scores were compared directly against those of real charter school students’ of the same demographics.

But the study, Hoxby said, has a problem.

In her memo last Thursday, “A Serious Statistical Mistake in the CREDO study of Charter School,” later amended, along with a new title: “A Statistical Mistake in the CREDO Study of Charter Schools,” she contended that because the public school students’ profiles are simulated through averages, there is a smaller margin of error for measuring their achievement.

Hoxby released the memo through the National Bureau of Economics Research Web site, critiquing CREDO’s method of data analysis. The memo includes what she calls a “boiler-plate” proof.

According to Hoxby, the math in her memo is easily recognizable by any economist. “This is not an original mistake,” she said. Her memo, she claimed, uses the proof that would be taught in a first-year graduate-level economics class to explain why this mistake was made.

Education Prof. Martin Carnoy, who is not associated with CREDO or NBRE, said public academic disagreements like this are not unusual. Furthermore, he said, they are not unusual for Hoxby.

“Hoxby’s pretty aggressive on this stuff,” Carnoy said.

Hoxby’s criticism sparked a rebuttal from CREDO Director and Hoover Fellow Margaret Raymond, who said CREDO was obligated to point out that Hoxby was wrong in her assessment. The methods for analysis had been vetted through a pilot study that revealed the methods would work, Raymond said.

“The pilot gave us the confidence that our approach was solid,” Raymond said. “It was peer reviewed by four [other institutions], and none of Hoxby’s views emerged.”

While Raymond said CREDO had spoken on the matter, the question remains as to whether Hoxby’s memo seeks to serve her own research results.

Raymond, Carnoy and charter school literature all indicate that there are a wide range of perspectives on the efficacy of charter schools. Some schools are effective in their geographical area, some in their academic expertise; some are not successful at all. Also highly variable are the conclusions of policymakers and researchers.

Hoxby’s work on charter schools consistently points toward a positive assessment of charter school achievement.

While Hoxby insists that the memo regarding CREDO’s study is completely unrelated to her own research, the conclusions of her research cast a much more positive light on charter schools than the CREDO study’s conclusions.

“I was being asked about the CREDO study often,” Hoxby said, “but it was hard to explain the problem with the statistics in words to every person who asked.” That, she said, is why she wrote the memo for all of the colleagues and education researchers to read as an answer to their questions.

Carnoy commented that “maybe she did find something wrong [with the study], but she obviously did not like the message.”