Finally, and most importantly, it seemed accurate at the time. The three graphs below show the historical relationship between Secret Sauce Score and playoff success:
Methodological note: the first two graphs compare the series-winning success of playoff teams to their Secret Sauce Score, which is the sum of their regular season rankings in starter K-rate (EQSO9), closer win value (WXRL), and team fielding (FRAA); lower is better. The third graph measures the success of the team with home field advantage in postseason series match-ups compared to the difference between the home team and visiting team's Secret Sauce Score. All three are fractional polynomial curves based on logistic regression analysis of the variables.
The formula is based on the statistical analysis by Silver and Dayn Perry, which showed that starter strikeout rate, closer win value and team fielding were the only statistically significant correlates between regular season performance and postseason success. Of course, historical trends do not always hold, as has been the case for Secret Sauce since ~2002. Take a look at the success of the Secret Sauce favorites in head-to-head competition since the playoffs expanded to eight teams in 1995:
*On two occasions, opposing teams had the same Secret Sauce Score
Since 1995, the Secret Sauce test is certainly better than guessing, correctly predicting the winner of any given series 60% of the time. From 1995-2001, the formula is especially successful, defeating random chance at the same rate that the 1998 Yankees defeated their opponents. However, since 2002, Secret Sauce has performed no better than a coin flip at picking winners.
Having discovered this, my first thoughts were of sample size. Silver and Perry used a larger data set than I'm testing against, and this may just be a small-sample fluke. However, a logistic regression analysis of the formula shows that Secret Sauce, while rather reliable from 1995-2001, is statistically insignificant in the 2002-2009 period.
|Table 1.1: Postseason Success vs. SS Score (P-Values)|
|1995-2009||n = 120||0.017||0.030||0.073||0.030|
|1995-2001||n = 56||0.004||0.008||0.024||0.085|
|2002-2009||n = 64||0.633||0.746||0.869||0.182|
The table above indicates the statistical significance of the relationship between Secret Sauce Score and four different variables: Series Wins (the total number of series a team wins in a given postseason), LDS (whether or not the team wins the first round), LCS (wins the second round) and WS (wins the third round). The p-values for the regressions were statistically significant (using a weak 0.10 standard) for the entire date range and for pre-2002 postseasons, but from 2002-onward there is no statistically significant relationship.
Methodological Note: Ordered logit was used to test Series Wins, binary logit for LDS, LCS and WS.
This pattern repeated itself when looking at head-to-head postseason match-ups:
|Table 1.2: Matchup Success vs. SS Score (P-Values)|
|Range||Sample ||Home Win|
|1995-2009||n = 105||0.033|
|1995-2001||n = 49||0.012|
|2002-2009||n = 56||0.594|
Breaking Secret Sauce down into its components only yields more questions. Despite Silver's findings, closer win value since 1995 is almost never correlated with postseason success in any statistically significant way. In fact, by the 2002-2009 period, the only significantly correlated variable is fielding, and then only in some cases!
|Table 2.1: Postseason Success vs. SS Components (P-Values)|
The pattern continues when looking at Secret Sauce from a head-to-head perspective:
|Table 2.2: Matchup Success vs. SS Components (P-Values)|
So what gives? Is this just an statistical-historical anomaly, or is it an indicator that Secret Sauce no longer accurately models postseason success in the current era? Allow me to submit this harebrained theory:
- Since 1995, the gap between the "poor" and rich teams has grown.
- General managers (thanks to the Sabremetric revolution) have gotten smarter.
- The increasing payroll gap results in an increasing amount of talent on the board as the trade deadline approaches.
- Smarter GMs for the richer teams are spending scarce resources on good relievers and power pitching rather than crafty ground-ball hurlers and overrated offensive talent.
- Secret Sauce is failing to incorporate the proper value of these moves because the formula is based on full-season numbers, undervaluing players who were picked up late in the year.
What I'd hope you take away from this is not that Secret Sauce is worthless, but rather that we should remember its recent poor performance when playoff prognosticators employ it to predict the 2010 postseason (don't repeat that last clause aloud). That said, the concept and construction of Secret Sauce is so elegant, and the idea of a Unified Field Theory of Postseason Success so alluring, that one doesn't want to abandon it so quickly. I took one last crack at it by exchanging some of the BPro variables with less proprietary ones (such as UZR, FIP and xFIP), but to no avail.
I'm sure someone will come up with something. I'll keep trying, too.
P.S. For what it's worth, Secret Sauce has the Padres winning it all this year, currently holding a SS Score of 10. This would qualify for the best since the playoffs expanded to 8 teams. The Rays are nipping at San Diego's heels with a score of 14, which would still rank among the best of all time. I don't think I'd be alone in predicting that a Rays-Padres World Series would be one of the more exciting Fall Classics in recent memory—and the least watched Fall Classic of all time.