Teams Are Excelling When Their QBs Leave The Pocket. Can That Continue?

In Kansas City’s Oct. 6 game against Indianapolis, Chiefs QB Patrick Mahomes faced a third and 18 from the Colts 27-yard line. Perhaps a bit skittish after an 8-yard sack on the previous play, Mahomes vacated the pocket early despite good protection from his line. Retreating backward, Mahomes faked left, then spun around and sprinted to the right sideline. He turned upfield at the boundary, and before crossing the line of scrimmage, he threw the ball approximately 35 yards across his body to wide receiver Byron Pringle in the end zone.

Mahomes is perhaps best known for his rocket arm and improbable no-look passes, but his ability to salvage plays like this — scrambling outside the pocket when the intended play design fails — is also off the charts. Mahomes has quickly become exhibit A when coaches talk about the value of a QB who can create out of structure. Throws outside the pocket have been on the rise across the league in recent years, including plays that are explicitly designed to put the QB on the move. Chiefs head coach Andy Reid has even said that scheming movement for his quarterback is what makes his offense go.

This season, it seems to be paying dividends for teams that are proficient at it. With some notable exceptions, NFL teams that are successful on dropbacks outside the pocket have tended to win more games.

The Chiefs, 49ers and Patriots are all likely playoff teams that have taken different approaches to moving their quarterbacks.

The Chiefs have eight wins and rank fifth in the NFL in expected points added per play on dropbacks outside the pocket. I looked at all 60 plays that Kansas City has run outside, and I determined that half were improvised — plays made when the original call in the huddle broke down. On those improvised plays, Mahomes and the Chiefs averaged 0.39 EPA per play1 — 0.27 EPA per play better than the schemed plays designed by Reid.

San Francisco’s similar success to Kansas City belies a profound difference in approach. The 49ers are tied for the league lead in wins with 10 and are first in the NFL in EPA per play on outside dropbacks (0.51). But the Niners have run the second-fewest outside dropbacks in the league (28), and while the Chiefs have been reliant on the improvisational brilliance of their superstar QB, San Francisco has leaned heavily on head coach Kyle Shanahan. Nearly two-thirds of the 49ers’ plays outside the pocket in 2019 have been schemed, and Shanahan’s offense has been brilliant on those 18 occasions, earning 1.2 EPA per play. But when quarterback Jimmy Garoppolo has been asked to improvise outside the pocket, he’s been a liability, accruing -0.43 EPA per play.

Similarly, the Patriots seldom ask Tom Brady to leave the pocket. The Patriots rank just above the 49ers in outside dropbacks on the year with 32, third-fewest in the league. Perhaps one reason is that when the 42-year-old Brady does move outside, it’s generally been a disaster: 62.5 percent of Brady’s plays outside the pocket have been improvised, and New England has an EPA per play of -0.70 on those scrambles. That’s far worse than the team’s work on schemed plays, which have generated 0.0 EPA per play.

Which approach is most likely to succeed? To answer that, we need to know if we should expect outside-the-pocket performance to continue. And if it does, which flavor of outside-the-pocket performance is most likely to persist: the improvisational approach exemplified by Mahomes and Kansas City, or the scheme-based approach of Shanahan and San Francisco?

To try to answer these questions, I tested the year-to-year stability of EPA per play both inside and outside the pocket.2 The results were somewhat surprising. It turns out that a quarterback’s performance inside the pocket has a 96 percent chance of being more stable than their performance outside the pocket. That is, a QB’s performance inside the pocket is a better predictor of future success (and future struggles) than the plays he runs outside of structure. This aligns generally with the work of Eric Eager at Pro Football Focus, who found that QB play from a clean pocket is a more consistent measurement of quarterback performance, relative to QB play under pressure. Moreover, the evidence suggests that performance on schemed plays outside the pocket — specifically play-action — is almost completely unstable, further driving home that we can’t infer much about a quarterback’s success on outside the pocket.

None of this is to say that outside-the-pocket performance doesn’t matter. It’s simply unpredictable. Fumbles have an enormous impact on the outcome of a game, but they’re basically random events. Just because a team has had good or bad fumble luck throughout a season doesn’t mean future fumbles are more or less likely to occur.

We can use this information to set reasonable expectations for the rest of the season and into the postseason. Kansas City relies on Mahomes’s out-of-pocket exploits to a large degree: They’ve run nearly as many plays outside as San Francisco and New England have combined. And while outside-the-pocket performance is generally unstable across years, the type of plays Mahomes makes out of structure tend to be the most steady. Mahomes is also impressive throwing from the pocket. Among healthy, qualifying QBs, Mahomes ranks second in in-the-pocket QBR, behind only Lamar Jackson. It’s reasonable to conclude that his performance has a good chance to continue.

Mahomes has performed well in and out of the pocket

Quarterback rating in plays inside and outside the pocket for QBs this season with a minimum of 100 passing attempts, through Week 13

QBR
player Team games out of pocket in pocket TOTAL
Matthew Stafford* DET 8 15.1 82.7 68.5
Lamar Jackson BAL 12 72.1 76.9 81.6
Patrick Mahomes KC 10 75.2 74.6 76.6
Drew Brees NO 7 8.6 72.8 61.0
Dak Prescott DAL 12 81.1 70.8 72.9
Deshaun Watson HOU 12 84.7 68.0 71.9
Derek Carr OAK 12 55.3 67.7 56.3
Kirk Cousins MIN 12 66.5 66.7 60.5
Kyler Murray ARI 12 23.7 64.0 59.6
Jimmy Garoppolo SF 12 87.5 63.8 58.4
Russell Wilson SEA 12 92.8 63.4 72.1
Jacoby Brissett IND 11 18.4 59.5 48.2
Teddy Bridgewater NO 7 18.0 58.2 47.8
Matt Ryan ATL 11 28.3 58.0 54.5
Carson Wentz PHI 12 94.4 57.9 64.3
Ryan Fitzpatrick MIA 11 58.5 57.7 64.1
Aaron Rodgers GB 12 66.1 55.5 55.9
Philip Rivers LAC 12 12.0 55.1 46.2
Tom Brady NE 12 12.1 54.4 52.5
Josh Allen BUF 12 18.1 54.2 47.3
Jared Goff LA 12 46.0 50.3 42.1
Ryan Tannehill TEN 8 18.3 49.8 52.9
Sam Darnold NYJ 9 36.2 48.7 43.3
Joe Flacco* DEN 8 35.7 48.0 48.3
Case Keenum WSH 8 72.0 47.1 47.2
Baker Mayfield CLE 12 30.8 46.9 51.8
Daniel Jones* NYG 11 28.4 46.7 51.4
Jameis Winston TB 12 92.1 46.0 52.0
Mitchell Trubisky CHI 11 39.0 44.1 38.8
Andy Dalton CIN 9 7.3 42.7 39.5
Kyle Allen CAR 10 43.6 41.4 38.3
Nick Foles JAC 4 5.9 40.0 34.3
Gardner Minshew JAC 10 66.6 38.1 43.3
Marcus Mariota TEN 6 16.7 33.1 33.0
Mason Rudolph PIT 9 65.2 32.7 32.9
Jeff Driskel DET 3 45.6 30.8 48.8
Josh Rosen MIA 6 13.3 21.5 19.7
Dwayne Haskins WSH 6 10.2 13.0 14.2

*Quarterbacks sidelined with injuries coming into Week 14.

Source: ESPN Stats & Information Group

The performances of Garoppolo and Brady outside the pocket, however, take on less importance. Garoppolo likely won’t be as good in the future on outside plays — since most of them were schemed play-action — and there’s a chance that Brady won’t be as bad moving forward. We should probably temper our expectations on that count, though, since it’s unlikely that Brady will grow new legs at age 42. Perhaps the poor out-of-pocket numbers are here to stay for New England. More concerning is Brady’s very average in-the-pocket QBR of 54.4 through 13 weeks. If that number doesn’t improve, the Patriots’ path through the postseason could end up being one of the toughest, despite their gaudy regular-season win total.

Check out our latest NFL predictions.

Daniel Jones Hasn’t Figured Out Zone Coverage Yet

In Week 7 against the Arizona Cardinals, New York Giants quarterback Daniel Jones dropped back to pass on third and 13 from his own 35-yard line. Faced with a four-man rush and seven defenders in coverage, Jones waited patiently — perhaps too patiently — for his receivers to complete their routes past the first-down marker. Feeling pressure on his left, Jones stepped up in the pocket and fired the ball downfield — right into the arms of Cardinals linebacker Jordan Hicks. Hicks was playing zone coverage in the hook/curl area of the field, and he was in the perfect position to step in front of intended receiver Golden Tate.

While it might be tempting to dismiss the mistakes of a rookie QB as growing pains, the interception wasn’t an isolated case of a young quarterback making a questionable decision. It turns out that zone coverage has been a problem for Jones since he took over the Giants starting job in Week 3. Six of Jones’s eight interceptions on the year have come against zone, and he’s averaging just 5.6 yards per attempt against the coverage — worst in the league. Jones has completed passes 1.7 percentage points under what we would expect of a league-average QB against zone, according to NFL Next Gen Stats, and his QBR is an anemic 27.5. More surprising, given generational talent Saquon Barkley at running back, is Jones’s QBR of 24.9 on play-action passes against zone, which places him 30th out of 32 qualifying quarterbacks.

Jones’s poor performance on play-action is a little strange because we might expect that teams would play more Cover 1 (a form of man coverage with a single high safety) against the Giants with Barkley in the backfield, a defensive formation that allows them to bring a safety down from deep coverage to help out against the run. Through Week 12, however, that hasn’t been the case. New York opponents are playing 50.4 percent man coverage vs. 49.1 percent zone, which is almost exactly league average. The team would likely benefit from opponents playing more man coverage to match up against Barkley, as Jones is completing passes 1.9 percentage points over expected against man, with a more respectable QBR of 71.5.

Interestingly, Jones’s coverage splits are the inverse of league trends. At nearly every depth of target from zero to 30 yards, quarterbacks’ completion percentages are higher against zone coverage than against man in 2019.

The gap between zone and man is particularly pronounced on throws of 10 air yards or less. This makes intuitive sense: Defenses playing zone are typically happy to allow the short completion and rally for a tackle. But the success of man versus zone on deeper passes is more of a surprise. Zone seeks to take away the deep ball in favor of short, manageable gains. But in 2019, passers are completing a higher percentage of attempts against zone of up to at least 28 yards.

Meanwhile, play-action across the league has been quite successful against man coverage, which might come as a surprise to some NFL coaches.

Man is still the more effective coverage, but the gap between play-action and other passes is wider against man coverage than zone, making it a preferred tactic on intermediate throws.

Yet these leaguewide base rates don’t fit Jones’s statistics. Jones is an enigma — bad at things that most QBs excel at, like completing passes on play-action, yet good at some aspects of the game that many quarterbacks find extremely challenging. Against disguised coverages, for instance — coverages that start out looking like man or zone but then switch mid-play — Jones has the highest completion percentage over expected in the league4 at 10.8 percent. It’s pure zone coverage that’s his kryptonite.

Jones’s struggles against zone coverage likely explain at least some of the Giants’ disappointing year. He’ll need to show progress before next season if New York has any hope of competing with the Dallas Cowboys and Philadelphia Eagles in the NFC East. But there are some signs that he may be improving: Jones’s lone TD against zone coverage came in last week’s Week 12 loss to Chicago. A strong finish to the season might be enough for the Giants to find a reason for optimism heading into 2020.

Check out our latest NFL predictions.

How Do You Find Good NFL Defenders? By Measuring What’s Not There.

The New England Patriots’ Stephon Gilmore is widely considered one of the best cornerbacks in the NFL. Through Week 11, he had given up a 47.1 passer rating when targeted this year — to put that into perspective, a 39.6 passer rating is awarded to a quarterback who spikes the ball on every play. (Through Week 11, no qualifying starting QB in 2019 had a passer rating lower than 70.) Because of his prowess, offensive coordinators and QBs don’t even look his way: Of the 326 passing attempts against the Patriots this year, only 64 (19.6 percent) came against Gilmore.

When it comes to measuring this aspect of defensive performance, we’re only beginning to scratch the surface. That’s because individual defense is inherently difficult to assess: A player’s defensive impact may be more significant in the absence of activity, but we can only count things that do happen. To measure a defensive player, we’re always chasing ghosts, trying to count things that don’t.

But thanks to the NFL’s Next Gen Stats tracking data — and the impressive work of researchers such as Baltimore Ravens personnel analyst Sarah Mallepalle2 — we can visualize that absence of activity.

A useful example involves where on the field opponents choose to attack a defense. Every week, coaches huddle around game film, analyzing defensive schemes and players, planning how to exploit what they see. The results of that game-planning show up in the tracking data. Here are charts that show where every team’s pass defense was the most and least vulnerable in the first eight weeks of the season:3

The charts shown are scaled from blue to white and white to red, with red representing the locations where offenses attempted the most passes against the defenses, blue representing the fewest pass attempts and white representing the average number of passes. In theory, the blue areas are places where the absence of activity reveals defensive strength.

In practice, it’s more complicated to solve the problem than simply plotting the distribution of all passing attempts or completions. Some offenses pass in unique ways because of their schemes and the skill sets of their quarterbacks and receivers. Take the Seattle Seahawks, for example:

Seattle QB Russell Wilson’s pass attempts are skewed to the left.4 Why? This is likely because of the inability of D.K. Metcalf, the Seahawks’ rookie wide receiver and 2019 second-round pick, to run a full tree of routes. During the season’s first two weeks, he was targeted only while lining up on the left side of the field. Through the first half of the season, Metcalf had been targeted 45 times, and only 10 were on the right side of the field.

Given that the Seattle passing attack is skewed, we don’t want to penalize any defense’s right side5 just because those defenders had to match up against Metcalf. So to normalize for an opposing offense’s usual tendencies, I compared an offense’s pass distribution against the defense in question with that offense’s distributions against the other defenses it has faced during the 2019 season. We can consider the difference to be the defense’s relative effect on a typical offense’s gameplan.

Let’s take a deeper look at the Buffalo Bills.

The Bills defense has been outstanding, giving up 304.1 total yards and 197.8 passing yards per game through Week 11 — third best in the league in both categories. But we can see a clear disparity in how opposing offenses are attacking starting cornerbacks Tre’Davious White and Levi Wallace. White’s reputation and production seem to have discouraged offenses from throwing in his direction. So far this season, White has given up a 58.9 passer rating on 66 targets, compared with a 101.2 passer rating on 79 targets for Wallace.

The beauty of the Bills’ defense is that their star cornerbacks don’t “shadow” or follow specific opposing WRs wherever they line up. Instead, they mainly stay on one side of the field, no matter who lines up there. Through Week 8, White had lined up on the left side of the field for 97 percent of his snaps, and Wallace lined up on the right side of the field for 96 percent of his snaps.

Our chart reflects that, as well as which cornerback scares off opposing QBs and where on the field that is. Opposing offenses target the left side of the field significantly more against Buffalo (to challenge Wallace, the weaker corner) than when they play against other defenses.

This approach to analyzing defense assumes that teams react rationally to defensive weaknesses they see on tape. That assumption may not always hold true, but taking note of changing tendencies is still one of the best ways to look for those hidden absences of activity that are key to identifying good individual defense.

We still don’t know if these splits have any predictive power. But this is the next logical step in understanding passing: analyzing the horizontal level. At a minimum, these new visualizations provide an interesting new insight into how offenses change in order to attack defenses.

Check out our latest NFL predictions.