21 March 2011

Rating System Challenge: Ken Pomeroy, Nate Silver and Chalk Looking Strong

Through the first two full rounds, the Pomeroy system is leading the pack. Those of you who picked a Pomeroy-based bracket currently lead 97.9% of all ESPN Tournament Challenge entrants. Nate Silver's FiveThirtyEight bracket (which is partially based on Pomeroy) is in second, trailing by 10 points and 2 percentiles.

Then again, those of you who picked pure chalk look just as smart as Nate Silver at the moment, so let's not read too much into this too quickly.

It's early yet, and it's beginning to look like March 2011 is particularly idiosyncratic. Pomeroy's bracket is only outperforming pure chalk by the equivalent of one win in the field of sixty-four. FiveThirtyEight has performed exactly as well as pure chalk. Every other system is worse, and in some cases much worse.

If we can't say that Pomeroy or any other system has significantly outperformed simple seeding, then we can't say that any of these systems are particularly useful. At least not yet.

RNK ENTRY R64 R32 S16 E8 FF NCG CHAMPION PPR TOTAL PCT
1 Pomeroy 260 200 0 0 0 0 Ohio St 1080 460 98.0
2 FiveThirtyEight 250 200 0 0 0 0 Ohio St 1120 450 96.0
2 Pure Chalk 250 200 0 0 0 0 Ohio St 1120 450 96.0
4 Nolan Power Index 230 200 0 0 0 0 Kansas 1200 430 85.4
4 Jeff Sagarin 230 200 0 0 0 0 Ohio St 1080 430 85.4
4 Sonny Moore 230 200 0 0 0 0 Ohio St 1080 430 85.4
7 ESPN National Bracket 230 200 0 0 0 0 Ohio St 1120 420 78.0
8 Vegas 210 200 0 0 0 0 Ohio St 1120 410 68.9
8 LRMC Bayesian 250 160 0 0 0 0 Ohio St 1080 410 68.9
10 NCAA Official RPI 220 180 0 0 0 0 Kansas 1200 400 58.4
10 Lunardi RPI 220 180 0 0 0 0 Kansas 1040 400 58.4
12 ESPN Decision Tree 230 160 0 0 0 0 Kansas 1160 390 47.8
13 Preseason AP Poll 240 140 0 0 0 0 Duke 880 380 37.7

More analysis after the jump.


Market-based systems, such as the ESPN National Bracket and the Vegas picks are performing quite poorly so far. The least chalky systems—LRMC Bayesian, Lunardi RPI and the Preseason AP Poll—are bringing up the rear. And woe is anyone who relied on ESPN's Decision Tree to pick their bracket so far.

Why is chalk doing so well in a year with remarkable upsets, including Virginia Commonwealth's shocking advance to the Sweet Sixteen? Simple: the upsets that the systems predicted didn't occur, and the upsets that occurred the systems didn't predict.

Image credit: Wikipedia

No comments: