Science is more attitude than method

DAVID C. BERLINER Regents’ Professor of Education

Arizona State University

SUSE PhD ’68

berliner@asu.edu

Pharmaceutical research is the model we have in mind when we promote randomized trials as the standard for educational research. But the usefulness of findings from randomized studies for educators are not as clear as advocates of this approach would have us believe.

For example, pharmaceutical giant GlaxoSmithKlein admits that most of its drugs don’t work on most of its patients. Demonstrating an effect in a group taking a drug doesn’t mean that all or even most members of the group show the effect. Said their vice-president,“The vast majority of drugs—more than 90 percent—only work in 30 or 50 percent of the people.” A significant effect in a randomized trial does not tell us much about the pervasiveness of the effect.

So randomized trials can provide teachers good ideas for their classrooms. But individual differences among their students and variations in the environments in which they work will moderate the success rate of treatments supported in randomized studies. Most teachers’ decisions will still be based on past experience, conversations with other professionals, and concerns about nurturing the uniqueness of individual children. Randomized trials will not eliminate most teachers’ uncertainty.

Further, promoters of randomized studies imply that knowledge obtained in other ways is not credible. But anthropology in the social sciences and astronomy in the physical sciences succeed without much reliance on randomized studies. And causality can also be determined without randomization, as when we build a credible theory around correlational data, as in the case of cigarette smoking and disease. Science is more of an attitude than it is a particular method. Reliable knowledge isn’t generated only from studies using random assignments.

Let’s implement what works

CHARLES R. HOKANSON, JR. Chief of Staff and Senior Counsel, Office of the General Counsel, U.S. Department of Education* (current)

Professional Staff, U.S. House of Reps. Committee on Education & the Workforce

Research Fellow, Manhattan Institute

Stanford BA ’93, MA ’93

charles_hokanson@ stanfordalumni.org

Our work as education practitioners should be informed and guided by scientifically valid knowledge about “what works.” Every day I hear about new educational interventions that claim to raise scores, keep students in school, or achieve other coveted results. Many of these reading programs, new educational technologies, and reform models purport to be supported by evidence, but upon closer inspection, not much is there, except maybe poorly designed studies and policy papers that might be best described as advocacy.

At the federal level, randomized controlled trials are clearly the future of education research. The No Child Left Behind Act in 2001 focuses a great deal of attention on the use of “scientifically- based research.” Increasingly, federal K-12 grant programs are requiring a demonstrated evidence base showing grantees’ effectiveness in improving educational outcomes.The D.C. School Choice Incentive Act, passed by Congress in early 2004, establishes a new five-year school choice program for low-income Washington, D.C. residents that will be rigorously evaluated through a randomized controlled trial comparing outcomes of eligible applicants (students and parents) who were assigned by lottery to either receive or not receive a scholarship.

Well-designed and properly implemented random assignment trials allow researchers to evaluate with a high degree of confidence whether the educational intervention itself—and not other factors—causes the outcomes. We also know that “pre-post” and “comparison group” study designs can produce inaccurate estimates of effect. After decades of basically static achievement results by our nation’s students, education practitioners should be discerning and seek out interventions that have a solid, scientifically validated evidence base. This is a key aspect as we focus our efforts in improving children’s education and making sure that no child is left behind.

*Mr. Hokanson writes in his private capacity. His opinion does not necessarily reflect the opinion of the U.S. Department of Education.

New innovations in education research design are preferable

JOHN B. WILLETT

Charles William Eliot Professor

Harvard University Graduate School of Education

SUSE, PhD ’85

John_Willett@harvard.edu

I can understand how recent emphasis on “scientific research” in federal policy-making has created renewed attention to random assignment, but I think this attention reflects a narrow view of research design.The debate really needs to be about causal attribution — how you design educational research to make credible statements about whether one thing causes another, and how to obtain unbiased estimates of that effect.

In designing such research, you must ensure that exogenous (i.e., independent) variation is present in the levels of hypothesized “cause.” Classical experiments meet this criterion because participants are randomized to treatment levels by the investigator, but such designs are limited and may not fit the practical realities of education.

We need to take advantage of newer innovations in design and analysis that provide, or identify, exogenous variation in levels of cause. We haven’t really capitalized on the many natural experiments that arise constantly in education because of disruptions in policy over time, or because of geographical differences. We haven’t made full use of the regression discontinuity approach, which provides unbiased causal estimation and can mitigate moral issues implicit in random assignment. And, most importantly, we’ve ignored the instrumental variables approach, in which specially selected predictors (“instruments”) permit you to tease out whatever part of the treatment variation is truly exogenous, even if participants have self-selected into levels of cause, attrition has occurred, or randomization has failed.

The consequences of selection bias, attrition, participant choice and migration, missing data, the impact of the multilevel nature of education, and so on, won’t be resolved by a blind reliance on classical experiments.They’ll be solved by the careful implementation of innovative designs and analytic procedures sensitive to the complexities of the educational context.

* In every issue, the Educator poses a question about a timely topic. Selected members of the community (alumni, faculty and students) are invited to respond. If you have a suggestion for a future Forum Question, or would like to be a respondent for a particular topic, please contact the editor at suse.alumni@stanford.edu