Which Picks Did NFL Mock Drafts Get Most Wrong?

With the first round of the NFL draft complete, it appears that the wisdom of the crowds wasn’t particularly wise. The first three picks went relatively as expected, but the draft went off script with the Oakland Raiders’ pick at No. 4 overall: defensive end Clelin Ferrell of Clemson — a player who mock drafters believed would go somewhere in the middle of the first round. The Raiders’ pick was the first of many that defied expectations and left amateur GMs scratching their heads.

In the case of the New York Giants, some fans were banging their heads against the wall and collapsing in tears. New York, which passed on many quarterbacks a year ago to take running back Saquon Barkley, took Duke QB Daniel Jones at No. 6. Jones averaged a 20.4 pick in mock drafts taken in the last 30 days before the draft but came off the board an eyebrow-raising 14.4 picks earlier. The Giants seemed to be trying to get ahead of a quarterback run that didn’t exist: Ohio State’s Dwayne Haskins lasted until Washington took him at No. 15 (6.2 picks later than expected), and no subsequent QBs were taken on Thursday night.

But the New York football Giants, armed with three picks in the first round alone, weren’t finished reaching. Using the 17th overall pick they acquired when they dealt Odell Beckham Jr. to the Browns, the Giants selected DT Dexter Lawrence of Clemson, 10.5 picks earlier than expected. The Giants were able to capture some surplus value with their third and final pick of the first round, however: Georgia CB Deandre Baker lasted 3.2 picks longer than expected and should help fill the void in the Giants secondary that was left when Eli Apple was traded to New Orleans last October for picks in the fourth and seventh rounds.

The NFL draft has been full of surprises

The first round of the 2019 NFL draft by each player’s pick and his average draft position (ADP) in mock drafts since March 26, 2019

team player Position pick ADP diff
Arizona Kyler Murray QB 1 1.8 -0.8
San Francisco Nick Bosa DE 2 2.1 -0.1
N.Y. Jets Quinnen Williams DT 3 3.7 -0.7
Oakland Clelin Ferrell DE 4 19.0 -15.0
Tampa Bay Devin White LB 5 7.0 -2.0
N.Y. Giants Daniel Jones QB 6 20.4 -14.4
Jacksonville Josh Allen LB 7 3.7 +3.3
Detroit TJ Hockenson TE 8 13.0 -5.0
Buffalo Ed Oliver DT 9 9.3 -0.3
Pittsburgh Devin Bush LB 10 15.5 -5.5
Cincinnati Jonah Williams OT 11 13.3 -2.3
Green Bay Rashan Gary DE 12 11.2 +0.8
Miami Christian Wilkins DT 13 19.0 -6.0
Atlanta Chris Lindstrom G 14 29.3 -15.3
Washington Dwayne Haskins QB 15 8.8 +6.2
Carolina Brian Burns LB 16 16.0 +0.0
N.Y. Giants Dexter Lawrence DT 17 27.5 -10.5
Minnesota Garrett Bradbury C 18 25.7 -7.7
Tennessee Jeffery Simmons DT 19 29.5 -10.5
Denver Noah Fant TE 20 22.9 -2.9
Green Bay Darnell Savage S 21 54.7 -33.7
Philadelphia Andre Dillard OT 22 17.6 +4.4
Houston Tytus Howard OT 23 60.7 -37.7
Oakland Josh Jacobs RB 24 27.2 -3.2
Baltimore Marquise Brown WR 25 25.4 -0.4
Washington Montez Sweat DE 26 10.6 +15.4
Oakland Johnathan Abram S 27 33.6 -6.6
L.A. Chargers Jerry Tillery DT 28 31.6 -3.6
Seattle L.J. Collier DE 29 62.9 -33.9
N.Y. Giants Deandre Baker CB 30 26.8 +3.2
Atlanta Kaleb McGary OT 31 43.3 -12.3
New England N’Keal Harry WR 32 29.3 +2.7

Sources: NFL, Ben Robinson

The selections of Lawrence and Ferrell were part of a larger trend: NFL GMs appear to have been particularly enamored with Clemson players. Three Tiger defensive standouts from the national championship team were selected in the first round, and they went 10.5 slots earlier on average than mock drafts predicted.

A dominant theme of the night, as expected, was NFL teams trying to find the next star pass rusher. But it was a pass rusher who had the biggest slide down the board among the first-round selections. Washington appears to have gotten a substantial value when it selected Mississippi State DE Montez Sweat 26th overall. In a draft class stacked with edge rushing talent, Sweat came off the board 15.4 picks later than expected.3

When we look at all 32 first-round picks, the correlation between what mock drafters expected and what actually occurred was about the same in 2019 as it was in 2018. In 2019, the average draft position in mock drafts explained 48 percent of variance, down slightly from 49 percent of variance explained in 2018. This year’s first round skewed toward reaches, with six teams trading up on draft day to get their guys. Overall, players came off the board six picks earlier than expected; last year, that difference was five spots.

As a result, Day 2 of the draft should be one in which savvy teams can find more value than they may have initially anticipated. That could even drive more pick swapping, as teams look to swoop in and grab coveted players like mock draft darling D.K. Metcalf on the cheap.


From ABC News:
Biggest picks from the 1st round of NFL Draft

The NFL Is Drafting Quarterbacks All Wrong

No position in professional sports is more important or more misunderstood than the quarterback. NFL scouts, coaches and general managers — the world’s foremost experts on football player evaluation — have been notoriously terrible at separating good QB prospects from the bad through the years. No franchise or GM has shown the ability to beat the draft over time, and economists Cade Massey and Richard Thaler have convincingly shown that the league’s lack of consistent draft success is likely due to overconfidence rather than an efficient market. Throw in the fact that young QBs are sometimes placed in schemes that fail to take advantage of their skills,1 that red flags regarding character go unidentified or ignored2 and that prospects often lack stable coaching environments, and there is no shortage of explanations for the recurring evaluation failures.

All of this uncertainty makes the NFL draft extremely exciting: You never know for certain who will be good and who will be an absolute bust. Last year, much of the pre-draft speculation surrounded where current Buffalo Bills starting QB Josh Allen — who is tall and can hit an upright from his knees from 50 yards away — would be selected. This year, when Oklahoma’s Kyler Murray decided to forgo a career in baseball for a chance to become a top pick in the 2019 NFL draft, his measurables captured attention in a different way. Murray, listed at 5-foot-10 and 194 pounds, is 7 inches shorter and more than 40 pounds lighter than Allen, and he’s the the smallest top QB prospect in recent memory. While some scouts and NFL decision makers think Murray’s odds for NFL success are long — or have him off their draft boards entirely because of his lack of size — there is strong evidence in the form of metrics and models that he is actually a good bet to succeed.

Like the rest of the league, practitioners of analytics have a pretty poor track record at predicting QB success. It wasn’t just Browns fans who were high on Johnny Manziel — many predictive performance metrics liked him as well. If some of the world’s best football talent evaluators are convinced that Murray’s height is at least a minor red flag, how can we be confident that a 5-foot-10 college QB will be productive in the NFL? When it comes to the draft, deep humility is warranted. Still, there are solid reasons to be excited about Murray.

Completion percentage is the performance measurable that best translates from college to the NFL. The metric’s shortcomings — players can pad their completion percentage with short, safe passes, for instance — are well-known. But even in its raw form, it’s a useful predictive tool.

Completion percentage translates from college to the NFL

Share of NFL quarterback performance predicted by college performance in seven measures, 2011-18

share predicted
Completion percentage 17.9%
Average depth of target 16.7
ESPN’s Total QBR 12.1
Yards per game 9.2
Touchdown rate 8.5
Yards per attempt 7.0
Adjusted yards per attempt 4.2

For players who attempted at least 100 passes in the NFL.

“Share predicted” here refers to the amount of variance in the dependent variable explained by the independent variable in a bivariate regression.

Source: ESPN Stats & information group

Its kissing cousin in the pantheon of stats that translate from college to the pros is average depth of target: Passers who throw short (or deep) in college tend to continue that pattern in the NFL. These two metrics can be combined3 to create an expected completion percentage, which helps correct the deficiencies in raw completion percentage. If you give more credit to players who routinely complete deeper passes — and dock passers who dump off and check down more frequently — you can get a clearer picture of a player’s true accuracy and decision-making.

Another important adjustment is to account for the level of competition a player faced. ESPN’s Total Quarterback Rating does this, and we’re doing it, too. For instance, passes in the Big Ten are completed at a lower rate than in the Big 12 and the Pac-12. We should boost players from conferences where it is tougher to complete a pass and ding players whose numbers are generated in conferences where passing is easier.

When we make these adjustments, and then subtract expected completion percentage from a QB’s actual completion percentage, we get a new metric: completion percentage over expected, or CPOE. An example: In 2011 at Wisconsin, Russell Wilson had a raw completion percentage of 73 percent. We would expect an average QB in the same conference who attempted the same number of passes at the same depths that Wilson attempted to have a completion percentage of just 57 percent. So Wilson posted an incredible CPOE of +16 percentage points in his last year of college. CPOE translates slightly less to the NFL than either raw completion percentage or average depth of target,4 but it does a substantially better job of predicting on-field production. In stat nerd parlance, we’ve traded a little stability for increased relevance.

CPOE best predicts yards per attempt in the NFL

Share of an NFL quarterback’s yards per attempt predicted by college performance measures, 2011-18

share predicted
Completion percentage over expected 15.5%
Completion percentage 13.5
ESPN’s Total QBR 13.2
Yards per attempt 7.0
Average depth of target 0.0

For players who attempted at least 100 passes in the NFL.

“Share predicted” here refers to the amount of variance in the dependent variable explained by the independent variable in a bivariate regression.

Source: ESPN Stats & Information group

The test of a good metric is that it is stable over time (for example from college to the NFL) and that it correlates with something important or valuable. Completion percentage over expected is slightly more stable than other advanced metrics like QBR. CPOE is also the best predictor of NFL yards per attempt. Since yards per attempt correlates well with NFL wins, and winning is both important and valuable, we’ve found a solid metric. It should help us identify NFL prospects likely to be good — so long as they are drafted and see enough playing time to accumulate 100 or more passing attempts.5

But before we stuff the metric into a model and start ranking this year’s quarterback prospects, it’s worth asking why CPOE in college might be a good measure of QB skill. One possible explanation is that it’s measuring not just accuracy but also the signal from other qualities that are crucial to pro success. The ability to consistently find the open receiver and complete a pass to him requires a quarterback first to read a defense and then to throw on time and on target. Throwing with anticipation and football IQ are both crucial to playing in the NFL at a high level, and they are likely both a part of the success signal in the metric.

CPOE is also probably capturing the ability to execute a system efficiently. A quarterback who understands how each piece of the offense complements the others and constrains the opposing defense is a huge asset for his team. The term “system QB” has a negative connotation in player evaluation circles that is probably unwarranted. If a quarterback is operating at a high level, he is inseparable from the system he’s being asked to run. It’s also likely the case that the mental and physical abilities to run any system efficiently are traits that translate — even if only imperfectly — to the pro game.

CPOE also measures accuracy, of course — which many believe is the most important trait a QB can posses. Some coaches believe accuracy is an innate skill and not something that can be taught once a player has reached college. Others believe that mechanical flaws can be corrected if other traits like arm strength are present. The Bills clearly hold this view or they wouldn’t have drafted Allen, a player with an incredibly live arm but who had a college completion rate 9.2 percentage points below expected. But regardless of whether accuracy can be taught at the NFL level, all evaluators acknowledge its importance.

With all this in mind, I built a simple logistic regression model that attempts to identify players who will go on to establish a career mark of at least 7.1 yards per attempt in the NFL — the league average from 2009 to 2018. The model took into account CPOE and six other metrics, all calculated for the player’s college career.6 There are 49 quarterbacks who have entered the NFL since 2012 who have also attempted at least 100 passes — except for small-school QBs for whom advanced college data wasn’t available. I randomly split those players into two sets and used one set to build the model and the second set to test to see if the model is any better than random chance at identifying which prospects will go on to play productive NFL football. Though the model is relatively simple — and it would be wonderful if the sample size were larger — the results are promising. The model correctly identified many players who went on to have NFL success and many more who didn’t. The best estimate for its generalized accuracy is that it will correctly identify a QB prospect as a hit or a bust in around 74 percent of cases.7 The table below shows the results of the model, labeled Predict, and includes players’ college stats.

Results from the quarterback prospect model

A random sample of the 49 quarterbacks who were drafted since 2012* by model probability, along with college stats including completion percentage over expected (CPOE)

College stats
name CPOE YPA Avg. depth of target Total QBR Predicted prob.† Career NFL YPA
Russell Wilson +16 10.3 10.4 94 >99% 7.9
Johnny Manziel +9 9.1 8.8 89 99 6.5
Jameis Winston +8 9.4 9.6 83 98 7.6
Kellen Moore +10 8.7 7.8 86 97 7.5
Deshaun Watson +5 8.4 8.8 86 93 8.3
Sam Darnold +5 8.5 9.5 80 77 6.9
Matt Barkley +4 8.2 8.1 77 73 7.4
Jared Goff +1 7.8 9.0 74 61 7.7
Kevin Hogan +4 8.5 9.3 80 37 6.1
Marcus Mariota +4 9.3 8.2 90 33 7.2
Kirk Cousins +4 7.9 8.5 58 29 7.6
Paxton Lynch +2 7.4 7.9 59 14 6.2
Geno Smith +3 8.2 7.3 74 5 6.8
Nathan Peterman +1 7.9 8.9 71 4 4.3
Zach Mettenberger +4 8.8 10.5 71 4 6.8
Trevor Siemian 0 6.4 8.2 53 3 6.8
Matt McGloin -2 7.2 8.5 60 2 6.7
Blake Bortles +4 8.5 7.5 78 1 6.7
Lamar Jackson 0 8.3 11.0 82 0 7.1

*And have attempted at least 100 passes in the NFL.

†Probability the player will meet or exceed a career yards per passing attempt average of 7.1.

Source: ESPN stats & information group

Humility is warranted at this moment, so let’s point and laugh at the failures first. After all, all models are universally wrong, but some can be useful. This one was wrong about Johnny Football, as it practically guaranteed Manziel to be an above-average NFL quarterback. What it didn’t know about was Johnny’s love of all-night parties and other off-field shenanigans. Kellen Moore, a lefty passer who had a decorated career at Boise State, is another hiccup for the model. Moore is an interesting case of a player who just barely reached the 100 passing attempt threshold and eclipsed 7.1 yards per attempt for his NFL career but still bounced around the league and never found success or even a starting job. So the model predicted his statistical success in yards per attempt but not his actual success on the field. The problem here is that our success metric — career yards per attempt over 7.1 — doesn’t perfectly discriminate between good and bad NFL QBs. Much like human evaluators, models can sometimes be right for the wrong reasons, and Moore is a prime example.8

The model was also suspiciously bad at predicting Lamar Jackson, ranking him as an almost sure bust as a passer. Jackson’s career yards per attempt — most of those attempts coming in just seven games — is right at the 7.1 threshold, and while he is no one’s idea of Drew Brees, a success probability of zero seems an overly harsh assessment for a player that has clear talent — especially running the ball — and has already helped his team to the playoffs.

Still, the good outweighs the bad. The only other false negatives in the bunch are Kirk Cousins and Marcus Mariota, both of whom have career yards per attempt figures above 7.1. Meanwhile the low probabilities assigned to passers like Nate Peterman, Zach Mettenberger, Paxton Lynch, Geno Smith and Blake Bortles all appear reasonably prescient.

Looking forward and applying the model to the current draft class, we find a few surprises. Kyler Murray sits comfortably at the top with a 97 percent probability of being an above-average pro quarterback. Murray’s physical and statistical production comps with Russell Wilson are especially striking. Wilson and Murray had roughly the same yards per attempt in college, identical average depth of target and similar Total QBR.9 Both are also under 6 feet tall and played baseball at a high level. As far as comps go for short QBs, you really can’t do any better.

Murray isn’t just a scrambler who excels working outside of the pocket and on broken plays, either. According to the ESPN Stats & Information Group, 91 percent of Murray’s 377 pass attempts in 2018 came inside the pocket, and 81.6 percent of those throws were on target and catchable. Murray faced five or more defensive backs on 82 percent of his passing attempts and threw a catchable pass 78.8 percent of the time. Against nickel and dime packages, he was even better when blitzed, with 79.1 percent of his passes charted as catchable when the defense brought pressure. And Murray didn’t just check down to the outlet receiver when the other team sent heat. Kyler pushed the ball downfield at depths of 20 yards or greater 21 percent of the time vs. a blitzing defender.

Meanwhile the other consensus first-round talent, Ohio State’s Dwayne Haskins, is viewed as much less of a sure thing by the model. While his CPOE is identical to Murray’s and his QBR is similar, the model rates his yards per attempt and low average depth of target as red flags that drag down his probability of success. Nickel is the base defense in the NFL, so a quarterback’s performance against it is important, and Haskins faced five or more defensive backs far less often than Murray, dropping back against nickel or dime on just 63 percent of his pass attempts. And when Haskins was blitzed out of those looks, he was not as adept at delivering on-target passes, with 76.4 percent charted as catchable despite only 6.6 percent traveling 20 yards or more in the air.

Kyler Murray’s accuracy and rushing put him atop his class

College quarterbacks invited to the 2019 NFL combine by their career statistics and predicted probability of success*

College stats
Player CPOE YPA Avg. depth of target Total QBR Predicted Prob.†
Kyler Murray +9% 10.4 10.4 92 97%
Will Grier +6 9.0 10.2 78 90
Ryan Finley +4 7.6 8.5 76 78
Jordan Ta’amu +6 9.5 10.1 72 72
Dwayne Haskins +9 9.1 7.8 87 63
Brett Rypien +5 8.4 9.9 67 39
Jake Browning +3 8.3 8.8 73 38
Clayton Thorson 0 6.3 7.9 61 29
Trace McSorley +3 8.1 9.7 73 22
Daniel Jones -2 6.4 7.7 62 17
Gardner Minshew +2 7.1 6.8 70 4
Jarrett Stidham +3 8.5 8.3 69 3
Kyle Shurmur -3 7.0 9.0 59 1
Drew Lock -1 7.9 10.3 66 <1
Tyree Jackson -2 7.3 10.4 59 <1
Nick Fitzgerald -4 6.6 10.2 72 <1

*Excluding Easton Stick because of lack of data

†Probability the player will meet or exceed a career average of 7.1 yards per passing attempt

Source: ESPN Stats & Information Group

Other surprises from the consensus top-four prospects are the rankings of Duke’s Daniel Jones and Missouri’s Drew Lock — both of whom completed fewer passes than we would expect, and both of whom were assigned a low probability of NFL success. Teams should probably be very wary of both players. Since 2011, college QB prospects with completion percentages under expected — a list that includes Brock Osweiler, Trevor Siemian, Mike Glennon, Matt McGloin and Jacoby Brissett — have all failed to post career yards per attempt above the league average of 7.1. Meanwhile West Virginia’s Will Grier — a player few experts have mocked to go in the first round — looks to be the second-best QB prospect of the class. With his excellent college production and nearly prototypical size at 6-foot-3 and 217 pounds, Grier is a player whose stock could rise with a good performance on and off the field at the combine.

There are many weeks of interviews, testing and evaluation left to come for each of these prospects, and analytics are just one piece of the process. Models are certainly not a player’s destiny. Murray might end up profiling as a selfish diva who can’t play well with others. Lock could somehow morph into Patrick Mahomes. But ultimately the model and the metrics agree with Arizona Cardinals coach Kliff Kingsbury’s assessment that Murray is worthy of the top overall pick in the draft. Ship him off to a team with an early pick and a creative play-caller, and enjoy the air raid fever dream that follows.