Sunday, June 21, 2009

Too much emphasis on averages

The Quick and the Ed links to a study by the American Institutes for Research that translates state-by-state math scores in the US on the NAEP into scores on the internationally benchmarked TIMSS test. The moral? Our best-performing states, like Massachusetts and Minnesota, have test scores right up there with the world's best, and indeed better than all nations outside of East Asia. Take a look:

International Grades for Countries in 2007 Mathematics, Grade 8
  1. Taiwan: 598
  2. South Korea: 597
  3. Singapore: 593
  4. Hong Kong: 572
  5. Japan: 570
  6. Massachusetts: 547
  7. Minnesota: 532
  8. North Dakota: 530
  9. Vermont: 529
  10. 13 other American states...
  11. Hungary: 517
  12. England: 513
This analysis isn't perfect, of course. We don't have actual state-by-state TIMSS scores, and we can't accurately predict what they might be. But the study makes the comparison in the most sensible way possible given our limited data, using equivalent national samples on the NAEP and TIMSS. A state whose NAEP score is 0.2 standard deviations above the national student mean is mapped to a TIMSS score 0.2 standard deviations above the national mean. I suspect that this is quite accurate as a reflection of the performance of America's states on an international basis.

At this point, however, I think that the obvious question is missed: why do we always look at averages, anyway? They provide a useful summary statistic, but they don't tell us much about what's going on in specific parts of the distribution. Take, for instance, this list of the cutoff PSAT scores for "National Merit" status, which are set at the 99th percentile in every state:

1. District of Columbia: 221
1. Massachusetts: 221
3. Maryland: 220
4. New Jersey: 219
5. Hawaii: 218
5. Connecticut: 218
5. Virginia: 218...

47. West Virginia: 202
47. South Dakota: 202
49. Arkansas: 201
49. Wyoming: 201
51. Mississippi: 199

The test is scored out of 240, and it's clear that the differences between the states at the top and bottom are enormous. A student at the 99th percentile in Mississippi would probably be at the 96th percentile (if that) in Massachusetts, implying that the number of students at the "top end" differs by a factor of four between states. These gaps are larger than anything you'd suspect by looking at averages, and although there are similarities between the two state rankings, there are also a lot of differences. (Most notably the District of Columbia, an extreme underperformer in average performance that rockets to first place when you look at only the top slice of students.)

You can plausibly argue that this is more the result of economics and demographics than real differences in the quality of schools. And certainly a high 99th percentile doesn't excuse the District of Columbia's school system, which does an abysmal job for so many of its students. But I think there's a strong case that the performance of the top 1% is a lot more important than average performance for long-term economic growth. The defining technological innovations of our time come from people at the extreme right end of the ability distribution, not the "average." And since it's clear that the two metrics can diverge so markedly, we should be careful to examine how our policies affect the best students, rather than just the average ones.

No comments: