Better data: a modest proposal for education reform

One of the tragedies of education reform is that the evidence on so many important questions is so thin. Are private schools more effective on average than public schools? It's impossible to say: most commonly cited studies derive their results from ghastly regressions that attempt to isolate the effect of schooling by tossing in countless demographic "controls." The unobservable qualities of students—motivation, parental engagement, and so on—are ignored, despite their presumably pivotal role in driving students' school choices.

Randomization provides a far more reliable source of statistical evidence, but for obvious reasons it's not common in education (with a few notable exceptions). Yet there is one relatively frequent situation where randomization is the standard accepted practice: a lottery for students applying to an oversubscribed charter school. And in a recent working paper, Angrist et al. use the lottery for admission to a KIPP charter school in Lynn, Massachusetts to identify the school's impact on students.

The conclusion? While "KIPP Lynn applicants have (pre-lottery) math and reading scores that are 0.39 and 0.44 standard deviations below the state average and somewhat below the LPS average," the authors estimate that:

Our results show overall reading score gains of about 0.12 standard deviations for each year a student spends at KIPP, with significantly larger gains for special education and LEP students of about 0.3-0.4 standard deviations. Students attending KIPP gain an average of 0.35 standard deviations per year in math; these effects are slightly larger for LEP and special education students.

In other words, a single year of KIPP education can almost completely close the math achievement gap between Lynn students—primarily low-income minorities—and students in the highest-scoring state in America.

This isn't conclusive, of course: although randomization is the best way of identifying causal effects, there are still a few possible weaknesses in the data, and we only have results for one school. By any standard, however, this is tremendously encouraging news. If we could replicate this result—or even a result half as strong—in KIPP charter schools across the country, the impact on education policy would be profound.

This research should be easy. Anywhere in America where a lottery is held to determine entrance to a charter school, records should be kept and linked with students' performance. If education researchers across the country could access a dataset with large-scale randomized variation in school assignment, our understanding of what really works (and what doesn't) in charter schools would be vastly improved. I can't think of any reform that offers such overwhelming potential benefit at such low cost.