A number of people have compared the final forecasts for the election (e.g. this), but I’m more interested in longer term forecasting, so I’m comparing the state-by-state predictions of Intrade and FiveThirtyEight on the dates for which I saved FiveThirtyEight data a month or more before the election.
Here is a table of states where Intrade disagreed with FiveThirtyEight on one of the first four dates for which I saved FiveThirtyEight data or where they were both wrong on July 24. The numbers are probability of a Democrat winning the state’s electoral votes, with the Intrade forecast first and the FiveThirtyEight forecast second.
| State | 2008-07-24 | 2008-08-22 | 2008-09-14 | 2008-10-01 | |
| CO | 71/68 | 60/53 | 54.5/46 | 67.5/84 | |
| FL | 42/29 | 34.5/28 | 30/14 | 55.2/70 | |
| IN | 38/26 | 34.1/15 | 20/11 | 38/51 | |
| MO | 50/26 | 32.9/13 | 22.1/11 | 42.5/48 | |
| NC | 30/22 | 25/21 | 14/7 | 51/50 | |
| NV | 51.2/49 | 49/45 | 44.9/32 | 55/66 | |
| OH | 65/53 | 50/38 | 40/29 | 53.5/68 | |
| VA | 60.5/50 | 52.3/36 | 42/22 | 59/79 |
On July 24, both sites predicted Florida, Indiana, and North Carolina wrong. FiveThirtyEight got Indiana right on Oct 1 when Intrade was still wrong, but Intrade got North Carolina right on that date (just barely) while FiveThirtyEight rated it a toss-up.
Intrade got Nevada right on July 24 (just barely) while FiveThirtyEight got it wrong (just barely).
For Virginia, Intrade was right in July and August while FiveThirtyEight was undecided and then wrong.
FiveThirtyEight got Colorado wrong on September 14, but Intrade didn’t.
FiveThirtyEight got Ohio wrong on August 22, while Intrade got it right.
Intrade rated Missouri a toss-up on July 24, while FiveThirtyEight got it right.
On September 14, FiveThirtyEight was fooled by McCain’s post convention bounce by a larger margin than Intrade, but by Oct 1 FiveThirtyEight was more confident about correcting those errors.
For states that were not closely contested, there were numerous examples where Intrade prices where closer to 50 than FiveThirtyEight. It’s likely that this represents long-shot bias on Intrade.
In sum, Intrade made slightly better forecasts for the closely contested states through at least mid September, but after that FiveThirtyEight was at least as good and more decisive. Except for Intrade’s Missouri forecast on July 24, the errors seem largely due to underestimating the effects of economic problems – errors which were also widespread in most forecasts for other things affected by the recession.
In the senate races, I didn’t save FiveThirtyEight forecasts from before November 1. It looks like both Intrade and FiveThirtyEight made similar errors on the Alaska and Minnesota races.
[Update on 2009-01-13: contrary to initial reports, they apparently got the Alaska and Minnesota races right, although there’s still some doubt about Minnesota.]
On the other hand, Intrade had been fairly consistently (but not confidently) saying since early July that California’s Proposition 8 (banning same-sex marriage) would be defeated. Pollsters as a group did a somewhat better job there by issuing conflicting reports.
This emphasizes that we need to give priority to reducing the market long shot bias, such as by paying interest on deposits.
I was looking at Betfair’s site at http://www.betfairpredicts.com and noticed they did a decent job vs. 538 also — it looks like they called Indiana correctly when 538 did not prior to election returns. It appears they also got north carolina ahead of the pack. Missouri seems to have been a toss-up late.
Isn’t a lot of this moot because elections are not independent events? Elections which occur at the same time are correlated for a number of reasons, which would show up as bias (either for or against prediction markets), even though none may exist.
Pingback: 2008 US presidential elections | Midas Oracle .ORG
Not moot, but the correlations imply that the data tell us less than what a casual reading might lead one to expect.
Interesting blog entry here. I do not see an email address, but I am a PhD candidate who has been following this 538 versus Intrade debate closely and I was hoping it would be possible for you to share the full set of results that you saved for 538 and Intrade?!? It was be very useful in a project that I am doing on this topic. Thank You!!!