11 April 2016

Rating Systems Challenge: A Victory for ESPN BPI, a Review and a Mea Culpa

This post is part of a series about the 2016 NCAA Tournament.


First things first: there was a serious error in all my previous posts. I was working off of old data for Nolan Power Index. In fact, I was working off last year's data. Funnily enough, last year's NPI ratings outperformed every one of this year's systems in the 2016 Rating Systems Challenge. Unfortunately, that's not how this works. As usual, the real NPI ratings finished poorly, dead last. This means the winner was actually ESPN's BPI. This also means that not a single system this year picked the correct National Champion.


04 April 2016

Rating Systems Challenge: A Rational Championship on the Line

This post is part of a series about the 2016 NCAA Tournament.


Villanova's triumph over Oklahoma in the Final Four closed the book on Sagarin and Pinnacle's Vegas ratings. The last two systems with a chance to clinch the top spot in the 2016 Rating Systems Challenge are ESPN's Basketball Power Index and Warren Nolan's Nolan Power Index. BPI picked North Carolina to win it all. NPI is the only system that picked the Wildcats. Tonight's National Championship game will decide this year's best bracket.


01 April 2016

It Begins: 2016 Edition

This weekend marks the opening of the 2016 Major League Baseball regular season. Rational Pastime will be following it game-by-game, updating RPScore ratings along the way and checking in occasionally to see what insights the data yield.


As in the past, this year's RPScore ratings are seeded with WAR projections from FanGraphs and park data from ESPN. The season projections and spreads above are the result of running that data 1,024 times through my season simulator. At this point in the season, these projections are rather broad. For instance, the 20.6-win 90% confidence interval is nearly twice as large as the 12 win spread between the best and worst American League teams' median projections. In another instance, last year's disappointing Washington Nationals are just as likely to win 80 games as they are 100.*

*Please head on over to Cards Conclave to read my analysis of the Nationals' offseason and expectations for 2016.

But first, let's take a look at how we get there from here.

28 March 2016

Rating Systems Challenge: The Final Four (Systems)

This post is part of a series about the 2016 NCAA Tournament.


Though the house may not always win, it has excelled through the Elite Eight round of the 2016 Men's NCAA Basketball Tournament. Tied with Jeff Sagarin's ratings, Vegas has outperformed the field this so far spring. These leads are tenuous. Depending on how the Final Four and National Championship play out, one of four different brackets may come away the winner of this year's Rating Systems Challenge.


26 March 2016

Rating Systems Challenge: No System Pulls Away after Sweet Sixteen

This post is part of a series about the 2016 NCAA Tournament.


There was more than a little movement on the Rating Systems Challenge leaderboard through the Sweet Sixteen. Vegas, Sagarin and the USA Today Postseason Poll now lead the field. The preseason polls that led the field through the first weekend have tumbled down to sub-chalk levels. Nevertheless, the field is still wide open, with only a single Elite Eight pick separating the best system from the worst.



Let's have another shout out for Warren Nolan's index, nailing the most picks so far. Nolan Power's future is somewhat cloudy going forward, however, being one of the 15 systems to pick the ousted Michigan State Spartans to advance to the Final Four.

24 March 2016

Rating Systems Challenge: Preseason Polls Dominate Opening Rounds

This post is part of a series about the 2016 NCAA Tournament.


One of my favorite parts of the thought experiment I call the Rating Systems Challenge is discovering how well the AP and USA Today voters performed when they submitted their preseason ballots. The answer, more often than not, is "pretty well." Take this year for example. Through two rounds, the preseason polls are ranked first and second in both points thus far and most possible points through the National Championship.

Key: Tot = total points scored so far (10/correct pick in Round of 64, 20/pick in Round of 32, etc...); Best = most possible points the system can win given later round picks and eliminations; Picks = correct picks so far; R64...NC = points scored per round; Rem = possible points remaining based on later round picks and eliminations.

For those of you who are unfamiliar with this project, it's rather simple. I look at NCAA men's basketball rating systems, simulators and betting markets. I take the data they provide and construct an NCAA Tournament bracket based on these data. For polls and rating systems, it's rather simple: the higher ranked school advances over the lower ranked school. With simulators and betting markets, the team with the highest projected likelihood of advancing is the team that advances. In cases that require a tie breaker, I rely on chalk (the better seeded team advances).

This year as last year, there are 21 rating systems in the Rating Systems Challenge. They range from the quotidian (chalk, postseason polls, RPI) to the advanced (Pomeroy, Sagarin) to the complex (FiveThirtyEight and Christopher Long's simulations) to the esoteric (LRMC Bayesian, Nolan Power Index). It is a diverse group of methodologies that picked a diverse group of teams to reach the Final Four and beyond.


17 March 2016

Rating Systems Challenge: Historical Performance

This post is part of a series about the 2016 NCAA Tournament. Follow along here.

Photo credit: Me (Some Rights Reserved)

Over the years, some rating systems have performed better than others in an NCAA Tournament bracket. A minority of systems successfully outperform chalk on a consistent basis. The table below compares each system based on how each performed in the ESPN Tournament Challenge scoring system over the last five years.

*PAC = Points above chalk. For systems that I have tracked for fewer than five years, I regressed them to even performance against a chalk bracket for years with no data.



*In the future, I'll probably retroactively calculate Massey's performance, as that data is easily retrievable.

It's stunning how few systems reliably outperform a strategy that merely involves picking the better seed (Pure Chalk). If you are looking to use a ratings system to pick a bracket in the future, I would suggest picking the ESPN National Bracket's hive mind, as well as Jeff Sagarin's, Nate Silver's and Christopher Long's. The ESPN Insider's game simulator, USA Today Preseason Coaches Poll (with chalk as a tiebreaker), and Sports Reference's adjusted net ratings may also be worth a look.

Oh, and RPI? Please go away.

Ratings System Challenge: Stand By

Hey, folks. If you are wondering where the 2016 Rating System Challenge is, please stand by. I'm running it on a spreadsheet this year rather than dumping it into a ESPN Tourney Challenge group. I shall return shortly with historical data and the various systems' expectations.