What Constitutes Scientific Evidence?

Since its beginning, the field of education has been plagued by skepticism as to whether its research can be considered “scientific.”This view has been held not only by academicians in other fields, but also by some educators themselves.The concern about the usefulness of educational research is again being raised in the context of the passage of NCLB, new DOE policies, and increasing pressure on schools to use “scientifically- based research” to bring about results.

In a December 2003 IES report titled, Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide, the agency unequivocally spells out what it considers scientifically- based research. It lists three categories of research quality: strong, possible, and no evidence of effectiveness. IES identifies randomized controlled trials as the only research design that provides strong evidence of effectiveness.

IES’ new What Works Clearinghouse—an education version of Consumer Reports packaged into an online database—is tied to this rating system. It lists new research studies and rates them according to the IES system. Local and state administrators and other decision makers are encouraged to use the clearinghouse ratings to guide their decisions regarding policies, programs, practices, and materials. However, the site contains a dearth of information because few existing studies meet the new IES methodology requirements.

Putting the Cart Before the Horse

Not surprisingly, these policy changes at DOE have created a spectrum of responses among educators. In 2001–2002, SUSE faculty members Richard Shavelson and Denis Phillips served on a National Research Council (NRC) committee, comprised of members from education and other fields. Its charge was to investigate the most effective scientific methods of educational research and offer a plan for a federaleducational research agency to support high-quality scientific work. (Readers are encouraged to read the full report, Scientific Research in Education, available at the National Academies Press website, as well as an informative theme issue on this topic in Educational Researcher 31(8) (November 2002).

According to Shavelson, who chaired the NRC committee, perhaps the most important conclusion of the report was that no research method is in itself scientific or unscientific.“How one applies the method to a particular problem determines the quality of the research.The research question should always drive the design of the study, and not the other way around,” Shavelson said.

PHOTO: Erin Marie Furtak, a SUSE PhD candidate in CTE-Science Education, videotapes a middle school science classroom and SUSE student teacher as part of the Romance Project (described in caption on p.1). She is particularly studying the effects of embedded formative assessment, a type of assessment that occurs while learning is still in progress so that the teacher can gather information to move the students toward the learning goals.

A good deal of the controversy surrounding the DOE definition of scientifically-based research rests here, because the method promoted by the DOE (i.e., randomized controlled trials) is driving the research questions and is at odds with the NRC report. According to Phillips, most educational researchers have been highly critical of DOE for labeling most qualitative research as unscientific —including ethnographies and case studies—and for defining science in terms of randomized experiments.“ Based on this narrow definition,” Phillips said,“ eminent researchers such as Newton, Galileo, Einstein, and Darwin would never have qualified for federal funding.”

Combining Quantitative and Qualitative Methods

At the same time, both Shavelson and Phillips voiced support for randomized controlled studies, but not as a substitute for other methods. “ These are very appropriate when you want to determine if an educational program produces a discernible effect,” explained Phillips.“However, this doesn’t tell you why or how the program worked or didn’t work.You need finer-grained qualitative studies to determine that,” he said.“ Together the two very different types of scientific research can provide powerful, complementary information.”

Critics of qualitative research point to shoddy research designs that result in weak evidence. Phillips counters that one doesn’t throw out the methods just because they’re not properly used. “ When ethnographies and case studies are rigorously designed,” he said, “ they produce highly scientific, reliable data.”

Other Perspectives on the Controversy

Gerald Sroufe, director of governmental relations for the American Educational Research Association (AERA) concurs. “IES talks about the gold standard of randomized trials with a missionary zeal that overrides everything else.Well, there’s a gold standard for every type of research, not just for randomized trials. Before you can formulate an experimental hypothesis for a randomized trial you have to do a lot of other research using ethnographies, case studies, observations, or pre-post studies.This is the crux of the disagreement between IES and others.”

Even so, an increasing number of new research projects being funded by DOE are now going to scholars outside the traditional circles of AERA because they are using the randomized trial design. However, Paul Goren (PhD, ’91), vice-president of the Spencer Foundation—the only U.S. foundation focused exclusively on educational research—indicated there are no plans to shift its funding philosophy.“ Our approach is to fund the best possible research out there. Although the federal government is looking at one methodological approach, we pay attention to all methods that will produce high-quality research and improve education.”The foundation has an endowment of approximately $400 million. It makes annual grant payments between $15–20 million in support of education research.

A key player that appears to have been almost entirely left out of the IES plan to foster scientifically based research is the 24,000-member AERA. The NRC report specifically refers to the need for “the community of education researchers” to collaborate with the federal educational research agency in its work. However, current IES policy documents do not even mention the NRC report, much less reflect its recommendations. Sroufe says he hopes this will change, and that there will be greater opportunities for IES and AERA to work more closely together in the future.

What does this debate mean for research training at Stanford’s School of Education? According to the School’s dean, Deborah Stipek, rigor has and will continue to be the primary concern—whatever the research method used.“Students need to be well informed about these debates and prepared to defend the method they choose,” she said,“but we will not discontinue training in potentially useful methods because the federal government isn’t currently funding studies that use them.”Two strong methodologists, Tony Bryk and Sean Reardon, who joined the SUSE faculty this fall, will help the School maintain its high standards for rigorous research training.