Skill and Random Chance

An interesting article appeared recently highlighting the work of Wharton Professor Cade Massey who has analyzed whether any of the NFL teams did a better job of using its draft picks. His conclusion, after crunching the numbers? There are no differences in teams’ ability to draft.

In his article, Mike Jensen writes:

There are clearly “huge differences” in outcomes, Massey was quick to point out. “Some teams have great years, other teams have bad years – and it matters,” Massey said. “But those differences aren’t persistent year-to-year, which tells me that they are chance driven. Something between 95 and 100 percent – I’m not exaggerating – of team differences in the draft is driven by chance.”

There is skill involved in selecting players, the professor said. “It’s just that teams are equally skilled, in a very uncertain environment,” Massey said.

That eye-opening conclusion probably won’t be received with a lot of “darn rights” within the NFL. But Massey, who previously worked at Duke and Yale, and coauthors NFL power rankings that were published in the Wall Street Journal, has punctured NFL balloons before.

An earlier paper he co wrote, titled “Loser’s Curse: Overconfidence vs. Market Efficiency in the NFL Draft,” basically flipped established beliefs about the worth of NFL draft choices. It got plenty of attention across sports. On a basic level, this all makes sense. How much credit do the Patriots deserve for drafting Tom Brady in the sixth round? If they’d known Brady was going to be half as good as he turned out to be, they obviously wouldn’t have let everyone else have a crack at him for five-plus rounds.

But Massey’s work goes deeper than that, into the real value of picks. For years, NFL teams based the worth of draft choices on something that came to be known as The Chart, and gave a value to each pick. Except, according to research data collected by Massey and coauthor Richard Thaler, it was all wrong. Top choices were overvalued once you factored in their salaries.

“The genesis of that paper is the ‘99 draft,” Massey said. The consensus was that Kentucky quarterback Tim Couch was the top pick, as he turned out to be. Of course, Couch didn’t turn out to be the top QB. That was the year Donovan McNabb was drafted.

“We thought when you looked at ‘83, the lesson should have been, you don’t know which of these quarterbacks are going to be the good ones,” Massey said. “(Dan) Marino was the sixth (QB) taken. (John) Elway was first, but there’s (Tony) Eason and (Todd) Blackledge in between.”

For the football-history impaired, Marino was a bit better than Blackledge or Eason. But Massey’s point isn’t about scouting ability. (Alleged character issues were the reason for Marino’s fall in the draft.) It’s about overconfidence. “They’re too sure that they can predict the future,” he said.

That’s Massey’s real interest. The NFL draft happens to provide the data. “This isn’t about the NFL draft,” he said.

His field, Massey explained, has shown very clearly that people don’t appreciate how much uncertainty there is in life. And now he’s trying to come up with strategies that “are optimal when the environment is more uncertain.”

Massey is offering ideas that aren’t wedded to sports alone.

“You should play lottery environments very differently than math-problem environments,” Massey said. “With math problems, there’s a right answer. Work hard enough and you’ll get the right answer. If you can’t get it, find the smartest guy in the room and he’ll have the right answer.”

And lotteries?

“You don’t spend “any time trying to get the right number because it doesn’t make any difference,” he said. “You try to get as many draws as possible or spend as little money as possible.”

One of Massey’s central tenets: People tend to think of life as more math problem than lottery. There’s some mix, obviously. But they’ve got the ratio wrong. Given the strong element of chance, he said, you don’t throw up your hands and give up.

The prescriptions are very pedestrian, he added.

“As many draws as possible. Minimize cost. Value the process,” he said.

That last one is interesting. If there is more uncertainty, Massey believes, the process should actually be valued more.