As I have for the past few years, I will once again be running the Rational Pastime Rating Systems Challenge, a contest where I pit several bracket-picking systems against each other in an ESPN Tournament Challenge pool. Here's a look at how the various systems have performed over the last few years.
The table below plots the thirteen systems that I have tracked each of the last three years, ranked in order of correct picks made. Now, anyone who picks a bracket knows it's not just about getting the most correct picks—it's about getting the right picks at the right time. That said, my hunch is that correct pick percentage is a better predictor of future performance than how each finishes against other ESPN competitors.
| System | Correct (3yr) | Centile (3yr) |
| FiveThirtyEight | 62% | 84.7 |
| Chalk | 62% | 82.1 |
| ESPN National Bracket | 61% | 84.5 |
| Jeff Sagarin | 61% | 72.6 |
| Pomeroy | 61% | 69.2 |
| AP Preseason Poll | 61% | 55.6 |
| Vegas | 60% | 77 |
| Sonny Moore | 60% | 61.9 |
| ESPN Decision Tree | 59% | 58.9 |
| LRMC Bayesian | 58% | 55.4 |
| NCAA RPI | 56% | 38.2 |
| Lunardi RPI | 54% | 34.2 |
| Nolan Power Index | 53% | 50.9 |
Over the last couple of years, however, I have added more systems. The table below plots the nineteen systems that I have tracked over the previous two years.
| System | Correct (2yr) | Centile (2yr) |
| Survival | 68% | 85 |
| ESPN Preseason Poll | 67% | 75 |
| Jeff Sagarin | 65% | 94 |
| AP Preseason Poll | 65% | 71 |
| ESPN Computer | 64% | 97 |
| FiveThirtyEight | 63% | 91 |
| ESPN National Bracket | 63% | 89 |
| Vegas | 63% | 87 |
| AP Postseason Poll | 63% | 87 |
| Chalk | 63% | 87 |
| Pomeroy | 63% | 82 |
| ESPN Decision Tree | 63% | 81 |
| Sonny Moore | 63% | 69 |
| ESPN BPI | 62% | 86 |
| ESPN Postseason Poll | 61% | 64 |
| LRMC Bayesian | 60% | 73 |
| NCAA RPI | 57% | 41 |
| Lunardi RPI | 56% | 35 |
| Nolan Power Index | 53% | 53 |
The FiveThirtyEight model performs well in this sample, but it's not the best. The most correct picks belong to the Harvard Sports Analysis Cooperative's (and now Sports Illustrated's) Survival bracket, which won the 2012 Rating Systems Challenge. However, the model's accuracy did not serve it well last year, picking a lot of early contests but missing later ones. As such, the Survival model has performed at the 85th percentile against ESPN contestants, on average.
The top performer in terms of percentile over the last two years—and which tied for first in 2013—has been ESPN's computer picker, a system that hides behind their Insider paywall. The second best percentile performer was Jeff Sagarin's computer system, followed by last year's co-champ, FiveThirtyEight.
So which systems should you rely on? Well, none in particular, but you should be wary of any system that doesn't outperform chalk. If a rating system can't do better than the Selection Committee's own rankings in the long run, then it has no business ranking basketball teams in the first place. Look at several of the well-performing systems, take injuries into account (FiveThirtyEight already does this) and, most of all, have fun.
An Incomplete List of NCAAB Rating Systems:
No comments:
Post a Comment