Rating Systems Challenge: Preseason Polls Dominate Opening Rounds

This post is part of a series about the 2016 NCAA Tournament.

One of my favorite parts of the thought experiment I call the Rating Systems Challenge is discovering how well the AP and USA Today voters performed when they submitted their preseason ballots. The answer, more often than not, is "pretty well." Take this year for example. Through two rounds, the preseason polls are ranked first and second in both points thus far and most possible points through the National Championship.

Key: Tot = total points scored so far (10/correct pick in Round of 64, 20/pick in Round of 32, etc...); Best = most possible points the system can win given later round picks and eliminations; Picks = correct picks so far; R64...NC = points scored per round; Rem = possible points remaining based on later round picks and eliminations.

For those of you who are unfamiliar with this project, it's rather simple. I look at NCAA men's basketball rating systems, simulators and betting markets. I take the data they provide and construct an NCAA Tournament bracket based on these data. For polls and rating systems, it's rather simple: the higher ranked school advances over the lower ranked school. With simulators and betting markets, the team with the highest projected likelihood of advancing is the team that advances. In cases that require a tie breaker, I rely on chalk (the better seeded team advances).

This year as last year, there are 21 rating systems in the Rating Systems Challenge. They range from the quotidian (chalk, postseason polls, RPI) to the advanced (Pomeroy, Sagarin) to the complex (FiveThirtyEight and Christopher Long's simulations) to the esoteric (LRMC Bayesian, Nolan Power Index). It is a diverse group of methodologies that picked a diverse group of teams to reach the Final Four and beyond.

Rating Systems Challenge: Historical Performance

This post is part of a series about the 2016 NCAA Tournament. Follow along here.

Photo credit: Me (Some Rights Reserved)

Over the years, some rating systems have performed better than others in an NCAA Tournament bracket. A minority of systems successfully outperform chalk on a consistent basis. The table below compares each system based on how each performed in the ESPN Tournament Challenge scoring system over the last five years.

*PAC = Points above chalk. For systems that I have tracked for fewer than five years, I regressed them to even performance against a chalk bracket for years with no data.

*In the future, I'll probably retroactively calculate Massey's performance, as that data is easily retrievable.

It's stunning how few systems reliably outperform a strategy that merely involves picking the better seed (Pure Chalk). If you are looking to use a ratings system to pick a bracket in the future, I would suggest picking the ESPN National Bracket's hive mind, as well as Jeff Sagarin's, Nate Silver's and Christopher Long's. The ESPN Insider's game simulator, USA Today Preseason Coaches Poll (with chalk as a tiebreaker), and Sports Reference's adjusted net ratings may also be worth a look.

Oh, and RPI? Please go away.

Ratings System Challenge: Stand By

Hey, folks. If you are wondering where the 2016 Rating System Challenge is, please stand by. I'm running it on a spreadsheet this year rather than dumping it into a ESPN Tourney Challenge group. I shall return shortly with historical data and the various systems' expectations.