19 April 2010

2009 Playoffs Reveal Prediction Pitfalls

Just like small sample sizes are not deterministic of long-term, large sample outcomes, large sample sizes are not deterministic of short-term, small sample size outcomes.

BP's Secret Sauce, for instance, ranked the Phillies 23rd overall going into the 2009 Playoffs--the lowest ranking of all contenders. Of course, Secret Sauce could not predict that Brad Lidge would start getting lucky again or that Jeff Mathis and his .211 BA would give the Yankees fits in the ALCS.

What this gets to is the matter of "streakiness." Streaks hardly matter at all during the course of a 162 (or 163) game season. But they matter quite a bit when the season is reduced to 3-7 games as it is every round in October.

Stat models can only tell you what the most likely outcome is based on events that have already occurred, and they can do a reasonably good job of predicting the likelihood of unlikely events. However, they cannot reliable predict future values of individual data points, or small collections of data. These predictions we must leave to Nostradamus and Joe Morgan.

No comments: