PDA

View Full Version : The NFL Passer Rating System, and Improving It for Rushing



Joel
12-11-2011, 10:55 AM
Not sure where to put this, but this makes more sense than any other option.

Part I: The PRS and Its Flaws

25(20Comp+Yds+80TD-100Int)+12.5/6
6Att

That's the critter, easily understood and far less mysteriously complicated than the NFLs "official" calculation, but results are identical for a "qualifying" numbers of throws (224, or 14/game.) Quarterbacks get:
20 points per completion, 1 point per yard, 80 points per TD and -100 points per interception. Divide by attempts, multiply by 25/6 and add 12.5/6.

If you're still here, congrats: You can now figure a passer rating, probably in your head (but maybe you already could.) A convenient example is Tim Tebows* 10/15 for 202 yds, 2 TDs and 0 Ints last week:

25(20*10+202+80*2-0)+12.5/6, or 25*562/90+12.5/6, which equals 14050/90+12.5/6
6*15

10 cancels out of the top and bottom (it's always wise to cancel when mentally figuring PRs) leaving 1405/9+12.5/6=158.1944....

Yet the NFL lists Tebows PR against Minnesota as 149.3; why? Because the Leagues official calculation sets upper and lower bounds for each category, bounding PR between 0 and 158.33... so it says ALL QBs have between:
0-77.5% completions, 0-12.5 YPA, 0-11.875% TDs and 0-9.5% Ints

In the case of Denver@Minnesota, Tebows 13.466... YPA and 13.33...% TDs greatly exceed the limits, so the League drops the excess. There are three equally negative consequences; the NFL PRS says:

1) A TD on every pass is no better than a TD every eighth pass,
2) An Int on every pass is no worse than an Int every tenth pass and
3) 50 YPA is no better than 12.5 YPA

Anyone agree? Didn't think so.

The upper and lower bounds are the first problem with the PRS, and probably the smallest; they can practically never affect season and career stats (but can and do skew game and partial season stats as demonstrated.) It's hard to appreciate without calculating a game rating around 400 only to hear announcers declare it a "perfect" 158.3, or calculating a rating in negative triple digits only to hear announcers mercifully declare it "zero." That's why Tebow had a 149.3 rating last week instead of 158.2, and why Craig Mortons rating for SB XII is 0 instead of -76.

Additionally, casual examination of the formula reveals what the NFLs official process obscures: According to the PRS, 5 completions=1 TD pass! 5*20=1*20+80, right? Perhaps worse, completing EVERY pass but LOSING yardage gets stellar ratings! (5*20-25)/5>(2*20+25)/5, right? Of course, the second guy's half a football field closer to the goal line, but don't let that distract you from the first guys better RATING. ;) People point to the West Coast offenses higher ratings as vindication of it, but the logic is rather circular: The PRS was practically DESIGNED for West Coast QBs; it rates them best because that's its job, not because they're the best. Some people even take THAT to vindicate the PRS BECAUSE it places "the best" (i.e. WC) QBs on top, leaving those of us with a basic grasp of logic pounding our skulls on our monitors (and wishing we could reach theirs.)

This has always been the thing I hate most about the NFL PRS: If Fran Tarkenton goes 6/20 for 200 yds, 2 TDs and 0 Ints while Joe Montana goes 15/20 for 100 yds, 1 TD and 0 Ints, they have the same rating. Tarkenton moving the Vikings twice as far and scoring twice as many points only (BARELY) makes up for Montana being more than twice as accurate. Or consider this: Take a 0 yd TD pass away from Tarkenton and his rating is WORSE than a guy who's 15/20 for -15 yds (which the PRS treats as 0 yds.) Compared to that, the PRS ignoring rushing is almost a trivial complaint (but only by comparison, hence I'll address it further down in the thread.)


*It is PURELY COINCIDENTAL that the PRS wrongly places the greatest value on Tim Tebows weakest quality while completely ignoring arguably his best ones. My own criticism of both flaws predates Tim Tebows very existence, and is itself the product of the same criticisms I first read about in The Hidden Game of Footballs chapter on "The Glory, the Blame and the Ratings." I STRONGLY advise anyone interested in a sound understanding of both the PRS and how the QB works in the NFL to read that chapter; reading the whole book would greatly improve most peoples grasp of NFL football in general.

Joel
12-11-2011, 10:59 AM
Part II: Building MY Better System

Were the PRS' only problems mercy on losers and treating stars like mortals, I could live with it. I would still attack its treating "West Coast" and "best" as synonymous, but there's an even greater problem: The PRS ignores EVERYTHING a QB does on the ground. That includes sacks, despite the NFL counting them in team "passing yards;" Tebow was sacked for 15 yards last week, which would ironically brings his YPA down to about the PRS cap, but the PRS only uses the 202 yards he gained throwing, without sack yardage. As the NFLs official explanation of the formula notes, "It is important to remember that the system is used to rate passers, not quarterbacks."

The worst thing about that flaw is not how much less relevant it makes the PRS, but how easily it can be remedied. The PRS quickly becomes a QBRS by including rushing stats thus:

Positive rushes=completions, rushing/sack yards= passing yards, rushing TDs= passing TDs, lost fumbles=Ints and EVERYTHING=an attempt. The slightly more complex formula looks like this:

25(20(Comp+Pos. Rush)+Yds+80TD-100(Int+Fum Lost))+12.5/6
6Att

Note "Yds" is now a composite of 3 stats, passing yds, rushing yds and sack yds, and "Att" is a composite of passing attempts, rushing attempts and sacks (which for some reason are counted separetely from QB rushing.)

I don't use this exact formula, because interceptions nearly always occur beyond the line, often FAR beyond and fumbles almost always occur near, often BEHIND, the line. Obviously, a pass intercepted 42 yds downfield and returned 5 isn't as big as recovering a fumble at the the line. According to Virgil Carter and Robert Machols groundbreaking article "Operations Research in Football," the interception is worth -1 expected points and the fumble -4.5, about a FG more: QB fumbles nearly always hurt more than picks and should be penalized accordingly.

I've played with the precise penalty over the years; initially, I doubled it, but that was too much. It is properly

net expected pts (Fum)*interception penalty
net expected pts (Int)

but that ratio's not simple to find. It requires knowing average field position before and after turnovers; it can be found by subtracting average return (easily found at NFL.com and countless other places) from average distance downfield turnovers occur, but the second number's not simply gotten. The only way I know is going through each play of each game noting the original line of scrimmage and where defenders recovered the ball on the 1% of plays that were turnovers.

Having a life, I won't do that; instead, I guesstimate interceptions travel ~31.3 yards before being picked off and returned 15.2 yards (the average 2011 return.) Assuming the average fumble return nets 0 yds (which I doubt, but I also doubt it's >6.6 yds, or 0.5 expected points) the ratio of fumble to interception expected points would be 1.5, so I'll use that: Interceptions cost 100 and QB fumbles cost 150. Final formula:

25(20(Comp+Pos. Rush)+Yds+80TD-100(Int+1.5*Fum Lost))+12.5/6
6Att

A far BETTER system would be to simply award 1 point for every 11.11... yards gained, 7 for each TD, -4.5 for each fumble and average expected points for each Int, then divide the whole thing by attempts. Completions are irrelevant except in their effect on field position and/or score, both of which are already counted. That point was already noted in Bob Carroll, Pete Palmer and John Thorns The Hidden Game of FootballUnfortunately, the ratings produced wouldn't look much like the numbers people are used to seeing, so I don't expect it.

The attached (zipped) Excel file has all required stats for this years 49 starters, as well as the NFL value; QBs are in descending QBR order, with PRS value and ranking listed at the far right of the table. Those so inclined can tweak the bonuses I and/or the League assign, but should do so bearing in mind the principle stated in the previous paragraph: The more you monkey with the numbers, the less familiar the ratings look.

There aren't a lot of BIG changes, but a few, and those few interesting.

Rodgers is still #1 and Leinart #2, though the latters single half of play put him firmly in the "non-qualifier" camp with 13 attempts (Rodgers does well under my total metric or the NFLs WC metric; he's a pretty good QB by ANY metric.) Brady and Brees switch. Orlovsky drops from 5th to 11th. Romo is still 6th; Schaub climbs from 7th to 5th. Eli Manning drops from 8th to 10th andRoethlisberger surprisingly drops from 9th to 14th, for the same reason: 4 fumbles. Smith drops from 10th to 13th. Stafford goes from 11th to 8th (only about 20% of QBs have no fumbles, but he's among them.) Tebow goes from 12th to 9th, for reasons most people know. Moore sinks like a stone from 13th to 22nd (watch those fumbles!) Fitzpatrick drops from 14th to 15th. Cutler drops from 15th to 24th (fumbles again.) Ryan's still 16th. Rivers drops from 17th to 23rd (4 fumbles.) Jason Campbell climbs from 18th to 12th on the strength of 2 rushing TDs and <20 sack yds. McNabb drops from 19th to 21st. Hasselback climbs from 20th to 17th (relatively few sacks and 4YPC.) The big shocker (or maybe not) is Cam Newton, who shoots from 21st to 7th because of 13 rushing TDs and nearly 500 rushing yards. Henne and Dalton crack the top 20 also because the former has a little more than and the latter a little less than 100 rushing yds, as well as a rushing TD apiece.

Joel
01-09-2012, 08:40 AM
By the way, just to round things out, the NFLs cap of 12.5 YPA on passing means (though I haven't looked to confirm this) they're low balling Tebow against Pitt like they did against Minnesota: Without the cap, his PR would be 136.2, but the NFL will say it's 125.6.

Also incidentally, due to a recently recognized actuarial error on my part I've corrected the value of fumbles in my QBR to 14/9*Ints rather than 12/8*Ints. With that in mind, Tebows QBR against Pitt is 137.3, just over a point higher than his PR (by virtue of 10 runs for 50 yards and a TD, with 0 sacks or fumbles.) It's VERY unusual for QBR to be higher than PR, but a 5 yard average and a TD with no sacks or fumbles will do it.

iLands
01-19-2012, 12:32 PM
Please keep working on this Joel. It's fantastic.

Joel
01-19-2012, 02:10 PM
Please keep working on this Joel. It's fantastic.
Thanks, man, glad you liked it, though I think Michael Neft deserves credit for thinking of it first (I did think it up on my own, but a decade or more after he did.)

For the record, I'm reverting fumbles to 1.5*Ints because the precise ratio depends on the precise value of interceptions, which I must estimate in the first place. Thus 1.5 is about as precise as the other value, but the math is simpler. It would help a great deal if I could get an average value for how far balls travel beyond the LoS before fumbles and interceptions, but I still don't feel like digging through every play of 1000+ games for something that only happens about 1% of the time.

I have been meaning to calculating regular season stats for all the Leagues starters now that 2011 is in the books, I just haven't done it yet. Maybe I can do that over the next couple hours and get back to ya'll. ;)

In a perfect world I'd just drop completions altogether, assign 1 point for each yard, 77.77... for each TD and a calculated value for each fumble and interception, then divide by attempts and multiply by a scalar to give something analogous (but NOT equal) to a percentage. Unfortunately, while we can get the average return distance for fumbles and picks, without knowing how far downfield they happened those numbers don't help.

Could be interesting to see what we get if we just leave it as is but drop completions; may take a look at that, too, though it seems given that a different scalar would be necessary.

Joel
01-19-2012, 07:44 PM
As before, I only included QBs who started at least one game, but this time I also separated qualifying passers* (those with 224+ attempts, or 14/game) from non-qualifiers for the same reason the NFL does: So LaDainian Tomlinson doesn't win the "Passing" Title going 5/6 for 7 yards and 5 TDs. You can re-sort by any stat you like if you hold down the mouse button, roll over the whole document, click the "Data" tab and then "Sort."

There are few BIG surprises; going by QBR instead of PR drops Moore and Kolb a LOT (usually indicating fumbles and/or sacks; in Moores case it was fumbles, in Kolbs it was BOTH.) Newton and Tebow both do better but, also as usual, mainly because QBR is lower than PR for EVERYONE, but less so for them. Unlike all other qualifiers, Newtons QBR is actually (significantly) higher than his PR. Tebow would've done better, but his 6 fumbles drop him just below average (19th out of 34, if anyone's curious.) The biggest surprise is Vick only moves up 2 spots, and Big Ben goes DOWN 4. The usual suspects are to blame: Fumbles and, in Bens case, also sacks.

I went ahead and re-calculated QBRs and PRs WITHOUT Completions (which was much simpler,) treating Rodgers season stats as 100. He kinda blew the curve though. :tongue:

I found 6.5*(Yds+80*TDs-100*Ints)/Att gives Rodgers a PR of 99 (getting him to 100 even takes a funkier looking scalar.)

7.5*(All purpose Yds+80*All purpose TDs-100*Ints-150*Fumbles)/Att gives him a QBR of 99.3 (again, I wanted a simpler scalar than 7.55, which is what an even 100 takes.)

Brees is next with an 81.3 QBR and 76.7 PR. :shocked: Rodgers had a SICK year. Everything in the tables after PR can be ignored; the rest is just where each QB ranks by completion percentage, his QB and Passer Ratings WITHOUT counting completion percentage and where he ranks by each of those.

*I listed Kyle Orton among qualifiers. The NFL does NOT, because he requires both his Denver AND KC numbers to reach 224 attempts. In the table, I included each stat as the sum of his Denver plus KC numbers, in that order, so you can easily pull them out and separate them into a "Broncos rating" and "Chiefs rating" if you like. For the record, his rating in Denver was around 75 and his rating in KC around 81.

Convalesce
01-26-2012, 03:43 AM
This is absolutely amazing.

I'm both impressed and stunned.

Good stuff, dude. That was awesome.

Joel
01-27-2012, 10:58 AM
Thanks again; one of these days I may break down and did through 1000 or so games to find the line of scrimmage before and after fumbles/interceptions so I can get an average, since that's the only way I know to do it.

I must note Michael Neft came up with a system closer to The Hidden Game of Footballs back in 1993, and it also incorporates rushing stats. I didn't find out about it until digging around online earlier this season to see if anyone (other than me) had modified the PRS to include rushing prior to The Cold Hard Football Facts doing so, but I ever get a firm read on the precise value of picks and fumbles the result will look a lot more like his:
http://www.profootballresearchers.org/Coffin_Corner/15-01-506.pdf

The main difference, apart from altering his TD/TO values (I think 80 pts/TD is fine, since that's about how 7 points at 11.11... yds/pt works out,) would be that using the best single season rating to "set the curve" from 100.

Ottokar Prohaska
02-03-2012, 09:43 AM
Any way you slice it, the top QBs last year were Rodgers, Brees and Brady. Recalculating things shuffles around where people sit.
Tebow had some superior gaes that broke the curve, but some games where he was not doing that much other than leading his team to a win 2 completions for 7 attempts usually does not win you a game.
I hope that he continues to progress and he sits at least in the middle next year.
A top flight QB does not = a championship but significant weakness at QB does lead to bottom feeder status.

Joel
02-04-2012, 10:15 AM
One of the observations the authors of THGoF made when they first retooled the PRS is that any system that radically changed who was the top QB would immediately be suspicious. You want to avoid fitting data to curves, but also avoid implausible results. If I played with the numbers till Kolb were the "best" QB, I'd know I did something wrong. Rodgers, Brees and Brady, lead under nearly every system, and in that order, because, they were the seasons top QBs, and in that order.

The interesting changes happen further down, like Schaub and Stafford switching places because Schaub managed a couple rushing TDs in a little over half a seasons worth of games. In my system,

Tebow (+9) and Newton (+8) outrushed all other QBs, but that still only puts Tebow (barely) in the top twenty, because his sack and fumble numbers for 11 games are simply awful.

Dalton (+5) and Fitzpatrick (+6) ran well and had few sacks (Dalton is -8 yards on the ground; Fitzpatrick is PLUS 67) or fumbles.

Kolb (-10,) Moore (-9,) Roethlisberger (-6,) and Cutler (-5) plummet by eating sacks (only two QBs were sacked more than "elusive" Big Ben) and drop the balling (Kolbs sack numbers are decent; his fumbles aren't.)

My system is only a tweak to the basic PRS; it doesn't change any of the values the NFL assigns, only adds a few it ignores. The NFL PRS requires QBs to hit their throws most of the time and not have picks; if they can pick up some yards and/or TDs along the way, that's a nice bonus (seriously, that's about how the PRS works, hence its problems.) My system merely adds "don't get sacked" and "even an interception beats a fumble."

Incidentally, ya'll are welcome and encouraged to double check my numbers; I just now noticed I had Cassel down as 106/269 (39% completions, worse than Tebow) so I double checked: He's actually 160/269 (miskey.) I've updated the spreadsheet on my harddrive accordingly (and it's reflected in the rankings I referenced in this post,) but obviously the file I uploaded a few weeks ago is wrong. ;)

While I agree a great QB does not guarantee a championship (ask Marino and Kelly,) I disagree that an awful one guarantees cellar dwelling. You can still run and be productive, and many teams have; the problem is that if that's ALL you do, your awful QB will almost always PRECLUDE a championship. An awful QBs critical position means he'll usually find ways to cost his teams big games even if everyone else is a HoFer.

Simple Jaded
04-16-2012, 07:15 AM
Wtf? Ya know, if you want a QB that does better in the QB ratings go get a better QB.......

ShaneFalco
04-16-2012, 06:07 PM
i always wondered why QB runs never factored into it, Vick and Cam would like that. I know its just based on passing but shouldnt an overall qb rating have both?

Joel
04-17-2012, 09:25 PM
i always wondered why QB runs never factored into it, Vick and Cam would like that. I know its just based on passing but shouldnt an overall qb rating have both?
Because, though the League now calls it a "QB" rather than "PASSER" Rating System, the latter is all it is or ever claimed to be: The NFL explicitly states it ONLY rates passing, not any of a QBs many other duties.


It is important to remember that the system is used to rate pass-ers, not quarterbacks. Statistics do not reflect leadership, play-calling, and other intangible factors that go into making a successful professional quarterback.
http://www.nfl.com/help/quarterbackratingformula
Even the disclaimer only references "intangibles," while ignoring running, which is VERY tangible (as anyone ever tackled by a pro MLB can attest.)

It's ironic ya'll bumped this thread now, as I recently took someones suggestion I email one of the stat sites about this, and they in turn suggested I email Football Outsiders (a site THGoF largely inspired,) from which I got a response this morning. After noting so many people spit out "new" PRSes these days that FO no longer bothers taking guest column submissions on the subject, they offered some data on the troublesome issue of net field position change on fumbles and Ints.

Perhaps the most important was confirming what I already suspected: Fumbles on snaps or sacks are rarely advanced much, if at all, and (by implication) fumbles on called runs aren't advanced much more. According to the email, the most likely fumbles to be advanced are (unsurprisingly) after receptions so, in terms of rating ALL offensive skill positions with the same formula, I will treat receiver fumbles as Ints. Ocurring downfield, away from most of the offense but in the midst of the defensive secondary, makes that more logical. The upshot is that equating fumbles to -50 yards (or -4 expected points) avoids much complexity but sacrifices only negligible precision, so I'll do that going forward.

Unfortunately, I still have no solid read on net field position change for interceptions, and thus can't establish a reliable ratio between them and fumbles. Using the NFLs values makes an interception twice as costly as a fumble, obviously and wildly inaccurate. Subtracting 2011s average Int return (14.1 yds) from yds per completion (9.1) yields similar results, since that would mean the average Int moves the ball 5 yards closer to the careless teams end zone. That there were 506 Ints last year but only 49 pick-sixes makes that implausible.

Meanwhile, I found an online PDF of Virgil Carter and Robert Machols groundbreaking "Operations Research on Football" article that introduced the concept of field position-as-expected-points to football: http://www.stanford.edu/class/stats50/handouts/541.full.pdf

By the bye, Jaded: I couldn't possibly care less about how our QB does in passer rating, because the PRS is such an awful way to rate QBs; I want a better rating SYSTEM for all QBs, not better ratings for particular ones (on the contrary, I'm a bit disgusted the PRS makes Brady and a number of others look better than they are.) Pretty sure I said that at the outset; THGoF introduced their modified system (in an era when the PRS was still so novel and poorly understood alternative systems were just as novel) with the phrase "building a better system," not "building a better QB."

Joel
04-19-2012, 11:05 AM
Alright, so after some more discussion with the guys at FO (who pointed out that NFL play-by-plays DO list where balls were intercepted and length of return, in addition to LoS before and after, which is all I really need) I've decided to break down and go through each of last seasons 256 regular season games after all to find LoS before and after each Int and fumble. I should be able to add up the net change for each of them, divide by the total, and get an average yardage value for both. As an added bonus, going through each game to track each turnover means I can factor in the frequency of pick-sixes and fumbles returned for TDs, with an 87.5 yard penalty for each (but without the 50 yard penalty for loss of possession, since the scoring team kicks off afterward.) With that in mind, I've already come across two pick-sixes the replay official reversed, saying the ball was down at the 1, only to have the team with the ball score anyway two plays later; I'm counting those as pick-sixes.

In the end, I'll not only have the average yardage value for fumbles, but one for interceptions that's far more accurate than the -100 penalty the NFL uses.

Ya'll are welcome and encouraged to double check my numbers (you can get the play-by-plays from NFL.com here: http://www.nfl.com/scores/2011/REG2.) Very encouraged, in fact, because through week 2 I'm not finding a huge difference between the cost of fumbles and interceptions, so I may be investing a lot of time for very little return.

I should also note that I'm ignoring fumbles/interceptions when the same team has possession before and after the play (e.g. in week 2, the Chiefs intercepted Stafford only to have Scheffler force and recover a fumble by the defensive back, so the only net effect was that the Lions lost 12 yards and had 1st and 10 again.) I'm also ignoring receiver fumbles, because the factors that makes passer/runner fumbles different from interceptions (i.e. they occur near the line, closer to the offenses end zone but in the amid many potential tacklers who reduce returns) are not present there; receiver fumbles are more like interceptions (though obviously not the QBs fault.)

Joel
04-19-2012, 05:26 PM
After crunching the first four weeks of stats, the value so far is: Fumbles -63.8479532163743, Interceptions -54.0570866141732. If those numbers hold, we're looking at fumbles being worth about 5 expected points and interceptions 4, but that's a preliminary result. Compared to most events, turnovers are fairly infrequent, and the first month of 2011 had a few statistical outliers that will hopefully average out over the season (e.g. 8/32 week 4 Ints were pick-sixes, several rather long; conversely, two fumbles were returned for TDs, and THREE the previous week.)

Again, I'm only counting turnovers on plays from scrimmage (i.e. not muffed kicks,) and none where with initial possession regained it on the same play (i.e. fumbled interceptions/fumble recoveries) or fumbled receptions (which I would count as interceptions if I did include them.) If you're following along at home, I recommend ESPNs play-by-plays because (ironically) the TV networks site doesn't have ads and video highlights that slow my computer to a crawl when I load up 16 game summaries: http://espn.go.com/nfl/schedule/_/year/2011#1 (http://espn.go.com/nfl/schedule/_/year/2011#3)