It's possible there's something to the "pitching wins pennants" hypothesis, but if so, it's hard to see it in the stats. In 2012, Colin Wyers - then the director of research at Baseball Prospectus, now a "mathematical modeler" for the Astros2 - and I looked for evidence that teams with strong no. 1 starters outperformed expectations in the playoffs. We identified the ace of each playoff team from 1995 to 2011, rated each one using a normalized measure of ace-hood, and then checked for any correlation between the strength of each ace and the difference between his team's regular-season and postseason winning percentages. There wasn't one, which suggests that once you know a team's regular-season record, knowing how good its best pitcher is doesn't add any predictive power. Nor could Colin find any evidence of an effect after rerunning the analysis using the entirety of a team's playoff rotation instead of its ace alone.
For this article, I enlisted the aid of Russell Carleton, a Baseball Prospectus analyst whom the Astros haven't (yet) hired, to perform a variation of the analysis Colin and I did in 2012. For each playoff series from 1969 to 2012, Russell calculated the probability (based solely on regular-season records) that each team would win a single game3 and, from there, the probability that each team would prevail in the series.4
Russell found that regular-season records were a significant predictor of postseason success.5 In other words, while playoff results are highly variable, they aren't completely random: In any given five- or seven-game series, the better team can easily lose, but over a large sample of such series, team quality does tend to win out.
The next step was to see whether, for the purpose of predicting postseason results, every .600 (or .580, or .560, or .540) team is alike, or whether having good starting pitching adds an extra advantage. Russell examined the first three pitchers who started for each team in each series.6 He also looked at Game 1 starters only. No indicator of the quality of those starters (strikeout rate, walk rate, home run rate, ground ball rate, linear weights) proved to be a significant predictor of a team's postseason success, after controlling for that club's regular-season record.7
So why doesn't the quality of a team's top three starters or its ace register as significant? For one thing, the differences between teams are compressed in the playoffs, relative to the regular season: Teams with terrible staffs don't make it to October, so the gulf between the best- and worst-pitching playoff teams isn't as stark as we're used to seeing during the season's first six months. Perhaps more importantly, there's more than one way to win baseball games, and even under an expanded playoff format, teams don't get to October without doing something well. A team with an inferior pitching staff often makes up for its weakness on the mound by being better on offense.