The NFL Is Drafting Quarterbacks All Wrong

No position in professional sports is more important or more misunderstood than the quarterback. NFL scouts, coaches and general managers — the world’s foremost experts on football player evaluation — have been notoriously terrible at separating good QB prospects from the bad through the years. No franchise or GM has shown the ability to beat the draft over time, and economists Cade Massey and Richard Thaler have convincingly shown that the league’s lack of consistent draft success is likely due to overconfidence rather than an efficient market. Throw in the fact that young QBs are sometimes placed in schemes that fail to take advantage of their skills,1 that red flags regarding character go unidentified or ignored2 and that prospects often lack stable coaching environments, and there is no shortage of explanations for the recurring evaluation failures.

All of this uncertainty makes the NFL draft extremely exciting: You never know for certain who will be good and who will be an absolute bust. Last year, much of the pre-draft speculation surrounded where current Buffalo Bills starting QB Josh Allen — who is tall and can hit an upright from his knees from 50 yards away — would be selected. This year, when Oklahoma’s Kyler Murray decided to forgo a career in baseball for a chance to become a top pick in the 2019 NFL draft, his measurables captured attention in a different way. Murray, listed at 5-foot-10 and 194 pounds, is 7 inches shorter and more than 40 pounds lighter than Allen, and he’s the the smallest top QB prospect in recent memory. While some scouts and NFL decision makers think Murray’s odds for NFL success are long — or have him off their draft boards entirely because of his lack of size — there is strong evidence in the form of metrics and models that he is actually a good bet to succeed.

Like the rest of the league, practitioners of analytics have a pretty poor track record at predicting QB success. It wasn’t just Browns fans who were high on Johnny Manziel — many predictive performance metrics liked him as well. If some of the world’s best football talent evaluators are convinced that Murray’s height is at least a minor red flag, how can we be confident that a 5-foot-10 college QB will be productive in the NFL? When it comes to the draft, deep humility is warranted. Still, there are solid reasons to be excited about Murray.

Completion percentage is the performance measurable that best translates from college to the NFL. The metric’s shortcomings — players can pad their completion percentage with short, safe passes, for instance — are well-known. But even in its raw form, it’s a useful predictive tool.

Completion percentage translates from college to the NFL

Share of NFL quarterback performance predicted by college performance in seven measures, 2011-18

share predicted
Completion percentage 17.9%
Average depth of target 16.7
ESPN’s Total QBR 12.1
Yards per game 9.2
Touchdown rate 8.5
Yards per attempt 7.0
Adjusted yards per attempt 4.2

For players who attempted at least 100 passes in the NFL.

“Share predicted” here refers to the amount of variance in the dependent variable explained by the independent variable in a bivariate regression.

Source: ESPN Stats & information group

Its kissing cousin in the pantheon of stats that translate from college to the pros is average depth of target: Passers who throw short (or deep) in college tend to continue that pattern in the NFL. These two metrics can be combined3 to create an expected completion percentage, which helps correct the deficiencies in raw completion percentage. If you give more credit to players who routinely complete deeper passes — and dock passers who dump off and check down more frequently — you can get a clearer picture of a player’s true accuracy and decision-making.

Another important adjustment is to account for the level of competition a player faced. ESPN’s Total Quarterback Rating does this, and we’re doing it, too. For instance, passes in the Big Ten are completed at a lower rate than in the Big 12 and the Pac-12. We should boost players from conferences where it is tougher to complete a pass and ding players whose numbers are generated in conferences where passing is easier.

When we make these adjustments, and then subtract expected completion percentage from a QB’s actual completion percentage, we get a new metric: completion percentage over expected, or CPOE. An example: In 2011 at Wisconsin, Russell Wilson had a raw completion percentage of 73 percent. We would expect an average QB in the same conference who attempted the same number of passes at the same depths that Wilson attempted to have a completion percentage of just 57 percent. So Wilson posted an incredible CPOE of +16 percentage points in his last year of college. CPOE translates slightly less to the NFL than either raw completion percentage or average depth of target,4 but it does a substantially better job of predicting on-field production. In stat nerd parlance, we’ve traded a little stability for increased relevance.

CPOE best predicts yards per attempt in the NFL

Share of an NFL quarterback’s yards per attempt predicted by college performance measures, 2011-18

share predicted
Completion percentage over expected 15.5%
Completion percentage 13.5
ESPN’s Total QBR 13.2
Yards per attempt 7.0
Average depth of target 0.0

For players who attempted at least 100 passes in the NFL.

“Share predicted” here refers to the amount of variance in the dependent variable explained by the independent variable in a bivariate regression.

Source: ESPN Stats & Information group

The test of a good metric is that it is stable over time (for example from college to the NFL) and that it correlates with something important or valuable. Completion percentage over expected is slightly more stable than other advanced metrics like QBR. CPOE is also the best predictor of NFL yards per attempt. Since yards per attempt correlates well with NFL wins, and winning is both important and valuable, we’ve found a solid metric. It should help us identify NFL prospects likely to be good — so long as they are drafted and see enough playing time to accumulate 100 or more passing attempts.5

But before we stuff the metric into a model and start ranking this year’s quarterback prospects, it’s worth asking why CPOE in college might be a good measure of QB skill. One possible explanation is that it’s measuring not just accuracy but also the signal from other qualities that are crucial to pro success. The ability to consistently find the open receiver and complete a pass to him requires a quarterback first to read a defense and then to throw on time and on target. Throwing with anticipation and football IQ are both crucial to playing in the NFL at a high level, and they are likely both a part of the success signal in the metric.

CPOE is also probably capturing the ability to execute a system efficiently. A quarterback who understands how each piece of the offense complements the others and constrains the opposing defense is a huge asset for his team. The term “system QB” has a negative connotation in player evaluation circles that is probably unwarranted. If a quarterback is operating at a high level, he is inseparable from the system he’s being asked to run. It’s also likely the case that the mental and physical abilities to run any system efficiently are traits that translate — even if only imperfectly — to the pro game.

CPOE also measures accuracy, of course — which many believe is the most important trait a QB can posses. Some coaches believe accuracy is an innate skill and not something that can be taught once a player has reached college. Others believe that mechanical flaws can be corrected if other traits like arm strength are present. The Bills clearly hold this view or they wouldn’t have drafted Allen, a player with an incredibly live arm but who had a college completion rate 9.2 percentage points below expected. But regardless of whether accuracy can be taught at the NFL level, all evaluators acknowledge its importance.

With all this in mind, I built a simple logistic regression model that attempts to identify players who will go on to establish a career mark of at least 7.1 yards per attempt in the NFL — the league average from 2009 to 2018. The model took into account CPOE and six other metrics, all calculated for the player’s college career.6 There are 49 quarterbacks who have entered the NFL since 2012 who have also attempted at least 100 passes — except for small-school QBs for whom advanced college data wasn’t available. I randomly split those players into two sets and used one set to build the model and the second set to test to see if the model is any better than random chance at identifying which prospects will go on to play productive NFL football. Though the model is relatively simple — and it would be wonderful if the sample size were larger — the results are promising. The model correctly identified many players who went on to have NFL success and many more who didn’t. The best estimate for its generalized accuracy is that it will correctly identify a QB prospect as a hit or a bust in around 74 percent of cases.7 The table below shows the results of the model, labeled Predict, and includes players’ college stats.

Results from the quarterback prospect model

A random sample of the 49 quarterbacks who were drafted since 2012* by model probability, along with college stats including completion percentage over expected (CPOE)

College stats
name CPOE YPA Avg. depth of target Total QBR Predicted prob.† Career NFL YPA
Russell Wilson +16 10.3 10.4 94 >99% 7.9
Johnny Manziel +9 9.1 8.8 89 99 6.5
Jameis Winston +8 9.4 9.6 83 98 7.6
Kellen Moore +10 8.7 7.8 86 97 7.5
Deshaun Watson +5 8.4 8.8 86 93 8.3
Sam Darnold +5 8.5 9.5 80 77 6.9
Matt Barkley +4 8.2 8.1 77 73 7.4
Jared Goff +1 7.8 9.0 74 61 7.7
Kevin Hogan +4 8.5 9.3 80 37 6.1
Marcus Mariota +4 9.3 8.2 90 33 7.2
Kirk Cousins +4 7.9 8.5 58 29 7.6
Paxton Lynch +2 7.4 7.9 59 14 6.2
Geno Smith +3 8.2 7.3 74 5 6.8
Nathan Peterman +1 7.9 8.9 71 4 4.3
Zach Mettenberger +4 8.8 10.5 71 4 6.8
Trevor Siemian 0 6.4 8.2 53 3 6.8
Matt McGloin -2 7.2 8.5 60 2 6.7
Blake Bortles +4 8.5 7.5 78 1 6.7
Lamar Jackson 0 8.3 11.0 82 0 7.1

*And have attempted at least 100 passes in the NFL.

†Probability the player will meet or exceed a career yards per passing attempt average of 7.1.

Source: ESPN stats & information group

Humility is warranted at this moment, so let’s point and laugh at the failures first. After all, all models are universally wrong, but some can be useful. This one was wrong about Johnny Football, as it practically guaranteed Manziel to be an above-average NFL quarterback. What it didn’t know about was Johnny’s love of all-night parties and other off-field shenanigans. Kellen Moore, a lefty passer who had a decorated career at Boise State, is another hiccup for the model. Moore is an interesting case of a player who just barely reached the 100 passing attempt threshold and eclipsed 7.1 yards per attempt for his NFL career but still bounced around the league and never found success or even a starting job. So the model predicted his statistical success in yards per attempt but not his actual success on the field. The problem here is that our success metric — career yards per attempt over 7.1 — doesn’t perfectly discriminate between good and bad NFL QBs. Much like human evaluators, models can sometimes be right for the wrong reasons, and Moore is a prime example.8

The model was also suspiciously bad at predicting Lamar Jackson, ranking him as an almost sure bust as a passer. Jackson’s career yards per attempt — most of those attempts coming in just seven games — is right at the 7.1 threshold, and while he is no one’s idea of Drew Brees, a success probability of zero seems an overly harsh assessment for a player that has clear talent — especially running the ball — and has already helped his team to the playoffs.

Still, the good outweighs the bad. The only other false negatives in the bunch are Kirk Cousins and Marcus Mariota, both of whom have career yards per attempt figures above 7.1. Meanwhile the low probabilities assigned to passers like Nate Peterman, Zach Mettenberger, Paxton Lynch, Geno Smith and Blake Bortles all appear reasonably prescient.

Looking forward and applying the model to the current draft class, we find a few surprises. Kyler Murray sits comfortably at the top with a 97 percent probability of being an above-average pro quarterback. Murray’s physical and statistical production comps with Russell Wilson are especially striking. Wilson and Murray had roughly the same yards per attempt in college, identical average depth of target and similar Total QBR.9 Both are also under 6 feet tall and played baseball at a high level. As far as comps go for short QBs, you really can’t do any better.

Murray isn’t just a scrambler who excels working outside of the pocket and on broken plays, either. According to the ESPN Stats & Information Group, 91 percent of Murray’s 377 pass attempts in 2018 came inside the pocket, and 81.6 percent of those throws were on target and catchable. Murray faced five or more defensive backs on 82 percent of his passing attempts and threw a catchable pass 78.8 percent of the time. Against nickel and dime packages, he was even better when blitzed, with 79.1 percent of his passes charted as catchable when the defense brought pressure. And Murray didn’t just check down to the outlet receiver when the other team sent heat. Kyler pushed the ball downfield at depths of 20 yards or greater 21 percent of the time vs. a blitzing defender.

Meanwhile the other consensus first-round talent, Ohio State’s Dwayne Haskins, is viewed as much less of a sure thing by the model. While his CPOE is identical to Murray’s and his QBR is similar, the model rates his yards per attempt and low average depth of target as red flags that drag down his probability of success. Nickel is the base defense in the NFL, so a quarterback’s performance against it is important, and Haskins faced five or more defensive backs far less often than Murray, dropping back against nickel or dime on just 63 percent of his pass attempts. And when Haskins was blitzed out of those looks, he was not as adept at delivering on-target passes, with 76.4 percent charted as catchable despite only 6.6 percent traveling 20 yards or more in the air.

Kyler Murray’s accuracy and rushing put him atop his class

College quarterbacks invited to the 2019 NFL combine by their career statistics and predicted probability of success*

College stats
Player CPOE YPA Avg. depth of target Total QBR Predicted Prob.†
Kyler Murray +9% 10.4 10.4 92 97%
Will Grier +6 9.0 10.2 78 90
Ryan Finley +4 7.6 8.5 76 78
Jordan Ta’amu +6 9.5 10.1 72 72
Dwayne Haskins +9 9.1 7.8 87 63
Brett Rypien +5 8.4 9.9 67 39
Jake Browning +3 8.3 8.8 73 38
Clayton Thorson 0 6.3 7.9 61 29
Trace McSorley +3 8.1 9.7 73 22
Daniel Jones -2 6.4 7.7 62 17
Gardner Minshew +2 7.1 6.8 70 4
Jarrett Stidham +3 8.5 8.3 69 3
Kyle Shurmur -3 7.0 9.0 59 1
Drew Lock -1 7.9 10.3 66 <1
Tyree Jackson -2 7.3 10.4 59 <1
Nick Fitzgerald -4 6.6 10.2 72 <1

*Excluding Easton Stick because of lack of data

†Probability the player will meet or exceed a career average of 7.1 yards per passing attempt

Source: ESPN Stats & Information Group

Other surprises from the consensus top-four prospects are the rankings of Duke’s Daniel Jones and Missouri’s Drew Lock — both of whom completed fewer passes than we would expect, and both of whom were assigned a low probability of NFL success. Teams should probably be very wary of both players. Since 2011, college QB prospects with completion percentages under expected — a list that includes Brock Osweiler, Trevor Siemian, Mike Glennon, Matt McGloin and Jacoby Brissett — have all failed to post career yards per attempt above the league average of 7.1. Meanwhile West Virginia’s Will Grier — a player few experts have mocked to go in the first round — looks to be the second-best QB prospect of the class. With his excellent college production and nearly prototypical size at 6-foot-3 and 217 pounds, Grier is a player whose stock could rise with a good performance on and off the field at the combine.

There are many weeks of interviews, testing and evaluation left to come for each of these prospects, and analytics are just one piece of the process. Models are certainly not a player’s destiny. Murray might end up profiling as a selfish diva who can’t play well with others. Lock could somehow morph into Patrick Mahomes. But ultimately the model and the metrics agree with Arizona Cardinals coach Kliff Kingsbury’s assessment that Murray is worthy of the top overall pick in the draft. Ship him off to a team with an early pick and a creative play-caller, and enjoy the air raid fever dream that follows.

Can Oklahoma Survive The Opening Half To Compete With Alabama?

One year ago, the Oklahoma Sooners fielded the worst defense to ever qualify for the College Football Playoff. Under first-year head coach Lincoln Riley, and behind Heisman Trophy-winning quarterback Baker Mayfield, Oklahoma took a 17-point lead on Georgia, the eventual national runner-up, before losing in the Rose Bowl semifinal. Twelve months later, Riley has another Heisman-winning quarterback in Kyler Murray and has piloted another one-dimensional Sooners team to a playoff berth.

The reward — a date with Alabama, college football’s lead power broker — seems more like a punishment. The Tide, long a defensive force under coach Nick Saban, now boast what’s likely the best offense in program history. Las Vegas oddsmakers cared not for Oklahoma’s three-game winning streak against the Tide and opened with Alabama as two-touchdown favorites. According to FiveThirtyEight’s college football prediction model, Alabama has a 41 percent probability of winning the national title. Oklahoma faces much taller odds, with an 11 percent probability of winning it all.

Here’s what to look for the when the two programs meet in the Orange Bowl semifinal Saturday at 8 p.m. Eastern.

Will Tua Tagovailoa or Kyler Murray win the QB showdown?

Seldom do a Heisman winner and his runner-up meet after the winner is crowned. Even given that rarity, this may be the best postseason clash of college quarterbacks we’ve ever seen. Both are coming off of historic regular seasons, with each in line to trump the record for Total Quarterback Rating, which ESPN has tracked since 2004 and is measured on a scale of 0 to 100.

But it’s not just the quarterbacks. In terms of offensive efficiency, this is the best matchup since the playoff began in 2014. Oddsmakers have taken notice, setting a points over/under total that’s unprecedented in the playoff era.

It seemed logical that the departure of Mayfield, the No. 1 pick in the 2018 NFL draft, would abate Oklahoma’s offensive horsepower. That 2017 team had the most efficient offense ever tracked, according to the ESPN Stats & Information Group.1 But in Murray’s first full season as a starting college quarterback, the Sooners’ offense has actually improved. “Kyler Murray has accomplished more in one season and had more impact on the Sooners’ tradition in one season than any other player in our history,” former Oklahoma coach Barry Switzer told The Athletic. “He’s broke all the damn records.”

The Sooners have gotten better under Murray

Oklahoma’s offensive production in 2018, with Kyler Murray at quarterback, vs. 2017, with Baker Mayfield at quarterback

Metric 2018 2017
Offensive points per game 47.0 43.6
Yards per play 8.8 8.3
Percentage of first downs or TDs per play 41.0% 37.6
Percentage of first downs or TDs per pass attempt 51.1% 49.4
Percentage of plays for zero or negative yards 25.6% 26.6

Source: ESPN Stats & Information Group

As if spring-loaded, Murray’s legs have minced opposing defenses. On a 75-yard touchdown run against Kansas, a broadcaster declared, “You’re not going to catch him,” before Murray had passed the 40-yard line. The junior is the country’s pre-eminent dual-threat wizard, whose 892 rushing yards place him seventh among all Football Bowl Subdivision quarterbacks this year.

At the same time, Tagovailoa has been the figurehead of the Tide’s offensive ascension since replacing Jalen Hurts in last season’s national championship game. Saban has been in Tuscaloosa since 2007, and this year’s offense has been his best in terms of, well, everything.

This is Saban’s most dominant Alabama offense

Alabama’s offense by season under head coach Nick Saban

Per play Per Game
Season Yards Yards Passing
Yards
1st Downs Offensive Points
2018 7.92 527.6 325.5 24.6 43.9
2017 6.59 444.1 193.4 22.2 36.1
2016 6.47 455.3 210.3 21.0 31.9
2015 5.89 427.1 227.1 21.9 30.1
2014 6.66 484.5 277.9 24.3 36.3
2013 7.15 454.1 248.5 23.2 34.2
2012 6.95 445.5 218.0 21.6 37.2
2011 6.46 429.6 215.2 21.6 32.1
2010 6.96 444.1 261.2 22.1 33.4
2009 5.96 403.0 187.9 20.6 30.1
2008 5.52 355.8 171.1 18.8 25.6
2007 5.05 373.8 224.5 22.6 26.4

Source: ESPN Stats & Information Group

Both Murray and Tagovailoa are having little difficulty stretching the field. If they keep up this pace during the playoffs, each would rank in the top three among all QBs since 2004 in single-season passing yards per attempt, with Murray’s current 11.92 mark in line to set the all-time record.

They won’t, however, be competing against equally proficient defensive units. Murray will be staring down a top-flight fortress that spent the past few months razing offensive lines and leveling quarterbacks. The Crimson Tide rank second in defensive efficiency, behind Clemson, and they lead the country in adjusted defensive quarterback rating, which accounts for the strength of the opposing quarterback. The Tide ranks among the 15 best teams in opponent completion percentage (51.8 percent) and yards allowed per pass attempt (5.86).

Tagovailoa will have the luxury of playing against a defense that might seem as though it’s providing Alabama an escort to the end zone. Oklahoma ranks 92nd in defensive efficiency, unseating last year’s squad as the new worst defense to make it to the playoff; the Sooners have allowed 56 touchdowns this year, 13 more than Alabama has allowed since the beginning of the 2017 season. Uninspired performances led to the midseason firing of defensive coordinator Mike Stoops. But the unit’s play hasn’t improved. After giving up 39 first downs to Oklahoma State, the most allowed by any FBS team this season, interim defensive coordinator Ruffin McNeill found room to praise his team for making “critical stops.”

Pass defense, in particular, has been ghastly for the Sooners. No FBS team allows more passing yards per game (291.4), and only five allow more completions (22.3). The Sooners love to give up the long play, having allowed 56 passing plays of 20-plus yards, the fourth-most by any team.

It’s unlikely that the turnover margin will favor the Sooners — even though Tagovailoa threw as many interceptions in his last outing as he did the rest of the season total. Oklahoma has generated 11 takeaways all season, the fewest of any qualifying team in the playoff era. Alabama has forced 11 since the beginning of October — and 21 in total.

On the opening drive of the SEC championship game, Tagovailoa suffered a high ankle injury, which required surgery the following day. The sophomore has said that he’ll be unencumbered come game time, and given how little he rushes — generating just the 89th-most rushing yards among QBs — he won’t need much mobility. Even a pocket-locked Tagovailoa can still shred an opposing defense.

Can Oklahoma survive Alabama’s first-half avalanche to win the second half?

Players come and go, and each season carries idiosyncrasies, but the narrative arcs of the Sooners’ two previous playoff appearances were seemingly penned by the same author. In both, a lead evaporated and a dominant first half gave way to a second-half dud.

In those appearances — in the 2015-16 Orange Bowl and the 2017-18 Rose Bowl — the Sooners outscored opponents 48 to 33 in the first half and were pummeled 49 to 14 in the second.2 Those losses, Riley said,3 could be traced to physical opponents and conservative play-calling.

Rewriting the script, then, will be paramount this time around for Oklahoma.

Offensively, the Sooners seem to have the first half covered. No team puts up more first-half yardage than Oklahoma, which averages 313.4 yards a game. The team racks up 9.5 yards per first-half play — nearly a first down on each play. Riley’s offense is outscoring opponents by an average of 11.3 points in first halves this season, the eighth-best mark in the country.

Of course, what separates Alabama from Oklahoma is its defense. The Tide allow 7.9 points per game in first halves, 12th-fewest in the nation and 7.9 fewer points than the Sooners. Alabama doesn’t just shut the door on its opponents in the opening 30 minutes — it packs their bags, shuttles them to the airport and ushers them through TSA. Remember when the Tide turned Tiger Stadium into a morgue by the third quarter of November’s top-five showdown with LSU? Or when the Tide took a trip to Oxford, watched Ole Miss score on its opening play and then blitzed the Rebels with 62 unanswered points, including 49 in the first half?4

LSU and Ole Miss aren’t alone. Saban’s squad is outscoring opponents 388 to 103 in first halves this season. On average, the Crimson Tide enter the locker room at halftime with a 21.9-point lead, the second-biggest margin by any team since at least 2004, the first year for which data is available.5 Ninety-three teams have scored fewer total touchdowns than Alabama has scored in first halves. Only four teams in the past 15 seasons have scored more first-half touchdowns than the Tide’s 53.

These lopsided first halves mean that the Tide hardly ever fall behind on the scoreboard. The average college football team this season played 178.6 first-half offensive snaps when trailing. Alabama played 18 — 32 fewer than the next-closest team.

Bama rarely plays from behind

Total first-half snaps when trailing for this year’s playoff participants

Team Offensive Snaps Defensive Snaps
Alabama 18 12
Clemson 63 24
Notre Dame 69 40
Oklahoma 128 54
National average 177 129

Source: ESPN Stats & Information Group

Coming into the SEC title game, teams had run a combined 388 first-half plays against the Tide defense. They didn’t have the lead on any of them. Georgia finally broke through in the SEC championship; the Bulldogs ran 43 first-half offensive plays against the Tide and led for 12 of them.

The nightmare doesn’t end for Alabama opponents in second halves: Then they’re outscored by a touchdown and a half, on average. In second halves, teams score 0.92 touchdowns per game against the Tide, tied for the sixth-fewest.

Conversely, Oklahoma’s dominance tapers off considerably in the final two quarters, when it outscores opponents by only 5.2 points, which ranks 24th nationally.

Little if any of that decline is attributable to the offense, which roars from start to finish. But defensively, the bottom seems to fall out for the Sooners after halftime, as they allow an average of 2.23 touchdowns per game in second halves, the most by any team in the Big 12 and tied for the 20th-most nationally. Riley’s defense has allowed 29 second-half touchdowns, the most by an Oklahoma defense since at least 2004.

However, should Oklahoma keep the game close down the stretch, it has a peerless crunch-time quarterback in Murray. The Sooners have played seven games decided by 14 or fewer points, while Alabama has played only one. In the fourth quarter, when the scoring margin is within 14 points, Murray has a nation-leading quarterback rating of 99 — that’s on a 1-to-100 scale, mind you.

There’s an argument to be made that Oklahoma is better equipped — certainly more experienced — to handle high-leverage situations.6 But Alabama has been so dominant that it simply hasn’t mattered.