A Measure of Success

538's estimate of state probabilities.

Came up with a way to measure 538's success in predicting the election other than a simple boolean for each state. All predictions are weighted by their certainty. A 50/50 estimate would thus count for nothing (it's worth pointing out that those calling the election a tossup were doing this...), whereas a prediction of 100% would yield a weight of 1.0. Correct predictions are positive, incorrect negative (538 got all 50 states right, plus DC). This weight is then multiplied by a state's electoral votes, summed, and then normalized on the range of the worst score (100% certainty on all, but getting them all wrong: -538) to the best (100% certainty, all correct: 538). Doing this gives 538 a score of .9485. Note that I treated states that award their electoral votes in a split manner as all or nothing for simplicity.

Baseline estimate of state probabilities

To provide more insight, a naive model was used to provide a baseline: all swing states estimated at 50/50 and all non swing states at 100%. This gives a score of 0.8977. This means that 538's estimate gets us 49.66% of the way from the baseline to a perfect forecast. Unfortunately, the data necessary to apply this measure to other forecasts wasn't easily available (and by "easily", I mean "with the amount of effort I was willing to put into this blog post"), but I grabbed some from a couple other sites.

PEC's last estimate map before the election.

Princeton Election Consortium's last posted data was in the form of a color map, so I extracted their estimates by examining the colors and they were clearly not that precise. Applying my measure, they got a score of 0.9429, and were 44.19% of the way from the baseline to a perfect forecast.

Simon Jackman's swing state estimate data.

Simon Jackman over at The Huffington Post had this estimate of swing states probabilities the day before the election. I couldn't find the non swing state opens, so filled in 100% for them, which almost certainly inflated his overall score. It worked out to 0.9544, and was 55.38% of the way from the baseline to perfect.

If anyone has other data (or better versions of the data I did use) I'd be happy to include it in this post. Here is the spreadsheet I used to compute the scores.

Bonus: I edited these maps that have been making the rounds since the election that scale state size according to electoral votes and population respectively to have purple shading that reflects the percentage of Obama and Romney votes. They don't really warrant a post of their own, especially since I didn't create the original maps they're based on.

Size proportional to electoral votes (1 square = 1 vote)

Size proportional to population (original image didn't include HI or AK)