30 March 2014

Rating Systems Challenge: Badgers Break Brackets


This post is part of a series tracking the successes and failures of various NCAA Men's Basketball ranking systems and bracket models throughout the 2014 NCAA Tournament. Click here to check out the full series.

Saturday night was total bracket carnage as 2 Wisconsin upset 1 Arizona, demolishing every system's pick for the West region. So if every system picked Arizona to win their region, then the carnage should be evenly distributed, yes? No! Some systems had Arizona going farther than others. Pure Chalk and RPI put Arizona in the finals. BPI, Lunardi RPI, Pomeroy and Sagarin had Zona winning it all. These systems' prospects are now greatly diminished.

29 March 2014

Rating Systems Challenge: 'Cats are Wild as Vegas Takes Lead


This post is part of a series tracking the successes and failures of various NCAA Men's Basketball ranking systems and bracket models throughout the 2014 NCAA Tournament. Click here to check out the full series.

Last night, twenty Final Four picks, eight finalist picks and five National Championship picks died in the span of just a couple of minutes when 4 Louisville and 1 Virginia were sent packing by 8 Kentucky and 4 Michigan State. And while they didn't shake up the top of the Rating Systems Challenge leaderboard too much, those upsets have major implications for most systems going forward.

28 March 2014

Rating Systems Challenge: Vegas, FiveThirtyEight Higher Flyers


This post is part of a series tracking the successes and failures of various NCAA Men's Basketball ranking systems and bracket models throughout the 2014 NCAA Tournament. Click here to check out the full series.

After the first day of the second week of NCAA Tournament Play, the FiveThirtyEight and Vegas systems remain atop the Rating Systems Challenge leaderboard. They were beneficiaries of their competitors' lack of faith in a certain short-legged omnivore. No big deal—there was little chance for a shakeup on the leaderboard last night, anyway. Tonight is different.

27 March 2014

Playing Pepper: Washington Nationals


Excuse me while I take a break from college basketball blogging.

Every year, Daniel Shoptaw, founder of the Baseball Bloggers Alliance, does a roundup of season previews for each of the thirty MLB teams over at his Cardinals Conclave blog, C70 at the Bat. Despite the fact that I've only been blogging about baseball during the postseason (something I hope will change soon), he was nice enough to ask me to offer my comment on the Washington Nationals.

Head on over to Playing Pepper 2014: Washington Nationals to read what I and three other BBA colleagues have to say about baseball in the nation's capital this season.

Rating Systems Challenge: FiveThirtyEight Volunteers Best Bracket


This post is part of a series tracking the successes and failures of various NCAA Men's Basketball ranking systems and bracket models throughout the 2014 NCAA Tournament. Click here to check out the full series.

Heading into the second weekend of NCAA Tournament play, the FiveThirtyEight and Vegas brackets sit atop the leaderboard.* Interestingly enough, the spread between the best brackets and the worst is only three correct picks out of forty-eight. Will the spread get bigger as the systems start running into Sweet Sixteen and Elite Eight match-ups that they simply didn't foresee?

*Though the LRMC Bayesian, USA Today Postseason and Sports Reference brackets have logged the same number of points as the two leaders, they have fewer points remaining, and thus poorer prospects going forward. 

23 March 2014

Rating Systems Challenge: Pomeroy Bracket Flying High


This post is part of a series tracking the successes and failures of various NCAA Men's Basketball ranking systems and bracket models throughout the 2014 NCAA Tournament. Click here to check out the full series.

It was a day of dizzying highs and heartbreaking lows in the Round of 32, one that sent 11 Dayton and 7 Connecticut on to the Sweet Sixteen and Pomeroy to the top of the Rating Systems Challenge leaderboard. Losses by 3 Syracuse and (to a lesser extent) 3 Duke yielded serious bracket implications for the nineteen systems in the later rounds. The potential for more bracket-busting today is high, as the systems predict five individual upsets in total.

22 March 2014

Rating Systems Challenge: Grin and Bear It


This post is part of a series tracking the successes and failures of various NCAA Men's Basketball ranking systems and bracket models throughout the 2014 NCAA Tournament. Click here to check out the full series.

There was quite a bit of movement in the Rating Systems Challenge after the second full slate of NCAA Tournament contests. Today's standings are just as much a result of yesterday's surprising upsets as much as the upsets that didn't happen. When all the dust settled, the Nolan Power Index slipped into a tie for second place with the revered Pomeroy system, while the Survival Model sits atop the leaderboard.

21 March 2014

Rating Systems Challenge: Bison Stampede into Round of 32


This post is part of a series tracking the successes and failures of various NCAA Men's Basketball ranking systems and bracket models throughout the 2014 NCAA Tournament. Click here to check out the full series.

The first full day of NCAA Tournament play was full of surprises. There were five upsets, several near-upsets and more overtime games than there have been in years. It was a surprising day for the Rating Systems Challenge, too, with perennial laggard Nolan Power Index leading after the first day. This is a system that has finished last or next-to-last in the previous two years and has yet to beat Pure Chalk in the three years I've been tracking them. Yet, here they are today at the top of the leaderboard, beating 99.8% of all other ESPN Tournament Challenge brackets.

20 March 2014

Rating Systems Challenge: Systems Overview


This post is part of a series tracking the successes and failures of various NCAA Men's Basketball ranking systems and bracket models throughout the 2014 NCAA Tournament. Click here to check out the full series.

The 2014 NCAA Men's Basketball tournament has begun, and we've already seen our first upset. When 11 Dayton took down 6 Ohio State, four systems (LRMC Bayesian, Sonny Moore, Sports Reference and Sagarin) lost a Sweet Sixteen pick. Fortunately for them, nobody had OSU in the Final Four. Listed below are the teams that our nineteen systems picked to reach the Final Four, the finals, and their National Champions.


1 Arizona is the dominant favorite out of the West—no system picks any other school to emerge from that region. 1 Florida is nearly as dominant out of the South, with only 2 Kansas threatening the Gators' Final Four hopes. The remaining brackets appear far more competitive: the systems pick four different schools to win both the East and the Midwest.


1 Florida is by far the school in which the systems place their confidence, followed by 4 Louisville and 1 Arizona. 8 Kentucky makes the list by virtue of their #1 ranking in both preseason polls.

Keep up with Rational Pastime as we make our way through the NCAA Tournament, tracking the successes and failures of the individual systems.

Notes on the systems: 1) I include the preseason polls because they, in fact, tend to do a better job of picking winners in March than the postseason polls; 2) Pure Chalk is the seeding order picked by the NCAA Selection Committee; 3) while I usually include both the ESPN Computer picker and ESPN's Decision Tree, they both picked the same bracket this year, so I'm aggregating them; 4) Vegas picks are based on the first round match-up and Final Four lines as presented by BetVega.com at 11 AM EDT on the first day of the Tourney; 5) chalk is the tiebreaker in the event of any ties.

19 March 2014

Rating Systems Challenge: Multiyear Comparison

It's bracket time again!

As I have for the past few years, I will once again be running the Rational Pastime Rating Systems Challenge, a contest where I pit several bracket-picking systems against each other in an ESPN Tournament Challenge pool. Here's a look at how the various systems have performed over the last few years.

The table below plots the thirteen systems that I have tracked each of the last three years, ranked in order of correct picks made. Now, anyone who picks a bracket knows it's not just about getting the most correct picks—it's about getting the right picks at the right time. That said, my hunch is that correct pick percentage is a better predictor of future performance than how each finishes against other ESPN competitors.

System Correct (3yr)   Centile (3yr)
FiveThirtyEight 62% 84.7
Chalk 62% 82.1
ESPN National Bracket     61% 84.5
Jeff Sagarin 61% 72.6
Pomeroy 61% 69.2
AP Preseason Poll 61% 55.6
Vegas 60% 77
Sonny Moore 60% 61.9
ESPN Decision Tree 59% 58.9
LRMC Bayesian 58% 55.4
NCAA RPI 56% 38.2
Lunardi RPI 54% 34.2
Nolan Power Index 53% 50.9

It's a credit to Nate Silver that his FiveThirtyEight model has both the highest pick percentage and finishes better against the general population that any other system I've tracked. In fact, over the last few years, it's the only system that has outperformed chalk (the NCAA's assigned seeding order) on a pick-by-pick basis.


Over the last couple of years, however, I have added more systems. The table below plots the nineteen systems that I have tracked over the previous two years.

System Correct (2yr)   Centile (2yr)
Survival 68% 85
ESPN Preseason Poll 67% 75
Jeff Sagarin 65% 94
AP Preseason Poll 65% 71
ESPN Computer 64% 97
FiveThirtyEight 63% 91
ESPN National Bracket     63% 89
Vegas 63% 87
AP Postseason Poll 63% 87
Chalk 63% 87
Pomeroy 63% 82
ESPN Decision Tree 63% 81
Sonny Moore 63% 69
ESPN BPI 62% 86
ESPN Postseason Poll 61% 64
LRMC Bayesian 60% 73
NCAA RPI 57% 41
Lunardi RPI 56% 35
Nolan Power Index 53% 53

The FiveThirtyEight model performs well in this sample, but it's not the best. The most correct picks belong to the Harvard Sports Analysis Cooperative's (and now Sports Illustrated's) Survival bracket, which won the 2012 Rating Systems Challenge. However, the model's accuracy did not serve it well last year, picking a lot of early contests but missing later ones. As such, the Survival model has performed at the 85th percentile against ESPN contestants, on average.

The top performer in terms of percentile over the last two years—and which tied for first in 2013—has been ESPN's computer picker, a system that hides behind their Insider paywall. The second best percentile performer was Jeff Sagarin's computer system, followed by last year's co-champ, FiveThirtyEight.

So which systems should you rely on? Well, none in particular, but you should be wary of any system that doesn't outperform chalk. If a rating system can't do better than the Selection Committee's own rankings in the long run, then it has no business ranking basketball teams in the first place. Look at several of the well-performing systems, take injuries into account (FiveThirtyEight already does this) and, most of all, have fun.

An Incomplete List of NCAAB Rating Systems: