Friday, August 20, 2010

The era of recognized genius

When Vinay Deolalikar (falsely, it now appears) announced that he had proven that P does not equal NP—thereby purportedly solving the most important open problem in computer science—I was intrigued but skeptical. I was intrigued because Deolalikar's background is far from that of the typical crank who announces a "proof" of some longstanding conjecture: he has a master's degree from IIT Bombay and a Ph.D. from USC, along with a few publications in good math journals. His manuscript was typeset in LaTeX (an easy way to filter out most completely disreputable attempts), long, and clearly fluent in the relevant technical vocabulary.

But I was also skeptical. Why? First, most attempts at proving such an important problem turn out to be flawed, even if they initially seem credible. Second, I've noticed that in recent years, virtually all successful efforts at solving key mathematical puzzles have come from people whose exceptional brilliance was already recognized via standard channels.

Consider perhaps the most impressive mathematical feat of our era, Grigori Perelman's proof of the Poincare Conjecture. A great deal of media attention focused on Perelman's apparent eccentricity: he lived with his mother, declined the Fields Medal, and now even seems to be declining the million-dollar Millennium Prize. But while he fits the "lone genius" cliche, he was hardly an unrecognized lone genius. As a high school student, he received a perfect score at the International Mathematical Olympiad, an incredibly difficult feat accessible to only an elite few. He went on to receive a Ph.D. from what is now St. Petersburg State University—one of the top institutions in Russia—and held positions at elite American math departments like SUNY Stony Brook, NYU's Courant Institute and UC Berkeley. Apparently he was offered professorships at Stanford and Princeton after leaving UC Berkeley, but turned them down in favor of returning to the Steklov Institute in Russia.

In other words, Perelman made his way to the very top of the mathematics profession long before he vanquished the Poincare conjecture.

Or consider Andrew Wiles, whose proof of Fermat's Last Theorem in the 1990s was easily the most widely recognized mathematical accomplishment of the decade. When he proved the theorem, he was a professor at Princeton University, which is as good a gig in mathematics as one can imagine. He was an undergraduate at Oxford, obtained his Ph.D. at Cambridge, and held a professorship at Oxford in between stints at Princeton.

Compare these examples to some of the widely publicized false proofs from recent years. In 2006 we saw a flawed attempt at solving the Navier-Stokes Equations (another of the Clay Mathematics Institute's "millenium problems") from Penelope Smith of Lehigh University. Smith was hardly a crank (which is why her attempt received so much attention) but her professional background didn't come close to Wiles or Perelman. She was an associate professor, never promoted to full professor despite almost three decades since her Ph.D., at a department in the lower half of the National Research Council rankings.

I don't mean to be elitist—I could never be a math professor anywhere. And my evidence is admittedly anecdotal. But I do see a compelling pattern here: more than ever before, the most compelling advances in mathematics come from people who were already at the top of their profession. The era of Swiss patent clerks making major contributions is over.

Is this because our current set of institutions is better at identifying talent early on? Because math at the research frontier has become so ungodly complicated that one needs to be plugged into the research elite to understand it? Or, on a related note, because ever more arcane mathematics requires more time to understand deeply, driving up the traditionally low median age of mathematical accomplishment and offering more time for our institutions to recognize geniuses before they make their greatest advances? All of the above?

I can't say.


Chris D said...

I think you've got it pretty well in general: disciplines have become so specialized that it takes many years to attempt a big contribution. That said, I'd prefer numbers over anecdotes.

You might consider saying "flawed" or "incorrect" rather than "false" to describe failed proofs; to me, the latter implies purposeful deception (e.g. "false prophet"). If you chose that word intentionally, I'd be interested in how you hear it differently than I do.

With Respect to X said...

For mathematics, I'd say its definitely all of the above, the only question is the relative importance of each effect.

Matt Rognlie said...

Chris D:

I agree -- saying that the failed proofs were "false" gives the wrong shade of meaning, even if it's technically an accurate description. I wasn't thinking very carefully when I chose it to describe efforts like Deolalikar's.

I wish I had numbers instead of anecdotes too, but unfortunately the quantity of relevant events is so small (and concepts like prestige in the profession so hard to define quantitatively) that I'm at a loss.

With Respect to X:


Justin said...

Two things:

1) Further evidence that Deolalikar isn't a crank is that he only announced the proof after someone he contacted emailed others saying "this proof looks promising" one of whom then blogged about it. Cranks do send their manuscripts to real researches, but they typically also self-publish early and often.

2) While it's agreed that Deolalikar's proof is flawed, I believe the discussion is still ongoing about whether his strategy has any fundamental holes in it. If the strategy is good, that's still an enormous accomplishment. It's worth noting that Perelman was following a well-established strategy for tackling the Poincaré conjecture, it was just ungodly difficult to achieve.

I may be wrong, but I think prior to Deolalikar, there were not really any strategies for solving P ≠ NP. If you want to look at this issue, you might look at, Timothy Gowers (who is mostly linking, rather than doing first order discussion), or Terence Tao, who has summarized a lot of the discussion.

Abhishek said...

Hey Matt,

First, your FB auto-publicizing works, here I am reading some of this very intriguing stuff after midnight on Sunday!

On this post however, I'm afraid I'm going to vehemently albeit unscientifically disagree with you :) In my witness-box of anecdotal reference I will summon collaborative efforts like the Netflix prize winning team (a clean directed problem a bit unlike some of economics' long standing mysteries) and InnoCentive, another heralded "platform" to match inventors to problems. I'm sure the existence of an entire centre here at mit filled with folks who find this interesting will drive home my point :) (

I think an interesting dimension in which what you're saying is true holds is the level of "subjectivity" of the problem at hand. When its unclear as to what the problem is, and how comprehensively a particular solution solves it the community seeks clues in status and position rather than the work itself obscuring its fair review (and I'm only holding economics to be partially culpable of this transgression). From my limited understanding some problems in math might have a more "clear cut" solution, but given that no one knew if Perelman was right for a couple of years and it seems like the jury will be out for quite a while on this Deolalikar chap means perhaps that I'm mistaken. But the trend if anything is for these things to matter less and less -- it'd be interesting if someone could tease out how these effects have changed over time and over fields.