Randomization provides a far more reliable source of statistical evidence, but for obvious reasons it's not common in education (with a few notable exceptions). Yet there is one relatively frequent situation where randomization is the standard accepted practice: a lottery for students applying to an oversubscribed charter school. And in a recent working paper, Angrist et al. use the lottery for admission to a KIPP charter school in Lynn, Massachusetts to identify the school's impact on students.
The conclusion? While "KIPP Lynn applicants have (pre-lottery) math and reading scores that are 0.39 and 0.44 standard deviations below the state average and somewhat below the LPS average," the authors estimate that:
Our results show overall reading score gains of about 0.12 standard deviations for each year a student spends at KIPP, with significantly larger gains for special education and LEP students of about 0.3-0.4 standard deviations. Students attending KIPP gain an average of 0.35 standard deviations per year in math; these effects are slightly larger for LEP and special education students.In other words, a single year of KIPP education can almost completely close the math achievement gap between Lynn students—primarily low-income minorities—and students in the highest-scoring state in America.
This isn't conclusive, of course: although randomization is the best way of identifying causal effects, there are still a few possible weaknesses in the data, and we only have results for one school. By any standard, however, this is tremendously encouraging news. If we could replicate this result—or even a result half as strong—in KIPP charter schools across the country, the impact on education policy would be profound.
This research should be easy. Anywhere in America where a lottery is held to determine entrance to a charter school, records should be kept and linked with students' performance. If education researchers across the country could access a dataset with large-scale randomized variation in school assignment, our understanding of what really works (and what doesn't) in charter schools would be vastly improved. I can't think of any reform that offers such overwhelming potential benefit at such low cost.