Pollster Power

Pollster Power
  •  emoticon

Here’s a fun Canadian story for you.

During the 1990s, when my home province of British Columbia was ruled by a particularly activist NDP administration, there was briefly a law on the books that made it illegal for newspapers to publish polling data during election season. Obviously such a rule was brazenly unconstitutional (and was predictably turfed by the courts in short order), but the fact that such legislation could even be passed in the first place highlights just how controversial polls can be in modern democracies. The law may be gone, but the mentality that inspired it certainly isn’t — and not only in B.C., either.

Now, at the time, the NDP argued that poll data had a corrupting influence on the democratic process, and claimed publishing it served no purpose other than to manipulate and restrain the mental freedom of voters. What does a newspaper poll do, after all, but bossily inform you of your two “most electable” partisan choices — ie; whatever two parties or candidates are polling highest — while simultaneously demanding you disregard all others.

If a guy is polling at 10% on day one, for instance, then what are you programmed to conclude? “He’s a useless fringe candidate that should be ignored for the duration of the race.” If another guy’s at 56% on that same day, however, then what? “Oh, he’s a shoo-in for sure!’” Such is the narrative structure polls impose. It’s about as self-fulfilling as prophecies come.

I bring all this up because I’ve noticed similar anti-poll arguments beginning to sprout in some conservative Republicans circles in the final days of the current presidential race. Specifically, there seems to be something of a growing conservative backlash against New York Times uber-pollmeister Nate Silver, and his staunchly defended assertion that Mitt Romney, as of now, has but a mere 27% chance of becoming president next week.

Mr. Silver’s blog has been described as the Moneyball of American politics for the sheer enormity of its number-crunching, and certainly few political sites are (or ever have been) anywhere near as aggressive in their ambition to provide the most openly self-confident electoral forecasts based on such a sweepingly comprehensive compilation of American polling data.

I won’t pretend to understand the precise science of how exactly Nate reaches his predictions (even after reading his 2,500 word summary), but in a broad sense they’re the result of aggregation, “weighing,” and simulation. Nate gathers as many credible polls as he can, ranks them based on various standards of reliability, and then uses their data to create computer models where all the various best and worst case scenarios for the candidates are allowed to unfold. The mean outcomes then become his “most likely” predictions — for example, at the moment it’s most likely Obama will win a second term.

In 2008, Nate’s site accurately predicted the presidential preferences of 49 of the 50 states and Nate became something of a celebrity in the aftermath, with long magazine profiles and book deals and all the rest of it. Many journalists and many more political junkies now regard him as nearly infallible, and this aura owes much to his uniquely compelling use of such intimidatingly high-science data to create such unapologetically clear predictions. He’s the ultimate public service geek.

Or at least that’s one way of looking at him. If you’re a conservative pundit, however, it’s increasingly likely that you regard his convoluted science as merely an opaque cover for some lazily biased core assumptions, and his blunt forecasts a cocky attempt to shape the future rather than predict it.

Nate straddles a “very thin line between forecasting and cheerleading,” wrote Josh Jordan in National Review last week, noting that the blogger had not only openly professed his support for Obama in the past, but seems to suspiciously weigh his polls in a way that consistently errs on the side of making the President’s re-election look inevitable. Joe Scarborough,  meanwhile, was even less kind, branding Silver a “joke” and an “ideologue.”

“Nobody” in the Obama campaign thinks Obama has a 73% chance of victory, he said, so why should Nate? I don’t care how tight your computer modeling is, elections aren’t the kind of things you can just conclusively call weeks in advance!

And those are just two critics. As more righties continue to pile on with accusations of propaganda disguised as rigor, the whole thing has descended into a full-on public spat. In the cheeky words of the Media Matters people, it’s a classic case of “Data Vs. Gut.” Silver may have the figures but the conservatives have, well, skepticism.

But what if the gut people aren’t just thinking with their duodenums?

There’s a persistent bias in our culture that says the guy with the numbers is always right, and for all the obvious reasons, this is usually a fairly reasonable conclusion. But in the realm of electoral prediction, which is  based around anticipating specific rates of mass public participation in a future event whose outcome will be determined by millions of constantly-evolving individual perceptions, well, it’s hard to think of a science less exact.

Humans crave firm answers even when none exist, though, and it’s this more than anything else that I think best describes the Nate Silver phenomena. In a tight presidential race defined by nervous ambiguity, he offers blazing clarity. To liberals increasingly fretful about Mittmentum, he offers unambiguous comfort. To voters in general, he offers closure that democratic governance is ultimately a safte, stable, sane science.

These salving traits don’t mean he’s necessarily going to be wrong about 2012, of course, and it certainly doesn’t mean his ability to mould the election narrative to a pre-determined conclusion is dangerous enough to deserve NDP-style suppression. But it does say something — an uneasy something — about his personal power, and the power of people like him in modern democracies.

And for that, a little criticism, no matter how grasping or partisan, can hardly be a bad thing.

29 Comments; - Discuss on Facebook - Discuss on the Forums (17)



^ 29 Comments...

  1. SES

    "Humans crave firm answers even when none exist, though, and it’s this more than anything else that I think best describes the Nate Silver phenomena."

    Yes, one of the problems is that people go to 538 and see "firm answers" rather than the sort of vague clouds that are actually being predicted. A 23% chance of becoming President of the United States is still a pretty huge chance, really. If you had a 23% chance of dying tomorrow, you'd be pretty afraid.

  2. @undefined

    "But it does say something — an uneasy something — about his personal power, and the power of people like him in modern democracies."

    The story of the 2012 election, in my own mind at least, has been about the explosion of polling data available. Every single day, you can find contradictory polls from almost every state of any importance. PPP has Obama up by 4 in Ohio; Rasmussen has him down by 3; Quinnipac thinks gay marriage will pass in Minnesota; Mason-Dixon expects it to fail; SurveyUSA is confident that the Democrats will pick up the Nevada Senate seat; Gallup still gives it to the Republicans…

    This has various rather dull causes (24-hour news cycle, media consolidation, relatively cheap internet polling, etc. etc. etc.), but the net effect is to create a huge cloud of polling data which, even to a relative expert, is essentially incomprehensible.

    In the absence of someone like Nate Silver aggregating this information into something coherent and useful, it would be total chaos, not unlike the modern news environment generally. Anyone who wanted to cite a poll proving *anything* would find it almost trivially easy to do so.

    Well, okay, you wouldn't find a poll showing, like, an impending Romney victory in Hawaii. But if you wait around a few days, you could easily cherry-pick a poll and build a story around it: "Romney's momentum has spread to Hawaii, where he's closed the gap with Obama by 8 points in overnight tracking!" sounds very exciting and plausible, and you simply won't mention the 10-point margin of error or the fact that you're comparing a liberal pollster to a conservative pollster in order to maximize the apparent "momentum".

    In any event: Nate Silver and others like Larry Sabato cut through that. They contextualize this humongous, contradictory cloud of information. They refine it into data we can actually use to have an informed conversation about politics.

    Instead of:
    Monday: "ROMNEY'S UP BY 5!"
    Tuesday: "OBAMA'S UP BY 3!"
    Wednesday: "ROMNEY'S UP BY 2!"

    We can talk about:
    Monday: "Today's polling average suggests that Mitt Romney is gaining momentum, but that the race is still essentially tied."
    Tuesday: "Romney's momentum appears to have collapsed somewhat, and the race is now a dead heat."
    Wednesday: "Romney is pulling slightly ahead, but it's by no means a definite victory."

    It makes for less exciting news coverage, but I think the latter constitutes a healthier, more worthwhile national conversation than energetically rushing back and forth with our hair on fire. (OBAMA'S UP! ROMNEY'S UP! EVERYBODY IS UP! UP UP UPPY UP!)

    Does that give Nate Silver too much power?

    There may be something worth criticizing in our seeming inability to discuss anything that doesn't come with a poll number attached, but , ehh.

    I think that's a problem Silver is making better, not worse.

  3. Taylor

    I don't realize why predicting 49 out of 50 would make someone famous…anybody with a brain could have predicted 46/50 last election.

  4. billytheskink

    UNIVAC did a pretty good job predicting the 1952 election. I'm curious as to what it would predict for this one.

  5. Taylor

    Must say, your Romney is spectacular, J.J.

  6. JonasB

    I prefer to get my polling from prognosticating invertebrates. Having said that, I think that the constant poll reporting is more of an annoyance than an actual hazard, and a natural result of the news cycle and whatnot. If polls were banned, then media outlets would just shift to tons of "experts" and "analysts" instead. Finding a way to shift the conversation away from numbers and more towards actual policy differences would be the best option, but I'm stumped how it could happen.

  7. EBounding

    A 73% chance of getting re-elected isn't all that great though. I believe that's the same chance you have to NOT get pregnant.

  8. SES

    According to 538, Romney has basically the same chance of winning that he has of winning two coin tosses in a row. Like he did before the last debate.

  9. Colin Minich

    Ah polls…or the schizolls as I call them. I recognize their importance but also see the utterly schizophrenic nature behind them, the tiny sample sizes in some of them, and how they jump back and forth as fickle as the autumn skies.

    Concerning Nate, I think it's very bold to declare him a one-term celebrity as the Esquire writer states. But here is a young man that has considerably put his effort into trying for some accuracy behind polling. Then again polls are like flavors and people will always have their preferences whether it be Ipsos, Gallup, Rasmussen, Pew, etc. I would rather be paying attention to the events now, conducted by both Romney and Obama, to impact my vote and what I think will be the vote of others. The attitudes seen during Hurricane Sandy is a clear case of this, both helping for the sake of professionalism but IMO Obama taking the step of leadership and deflecting the rather scathing and illogical words of former FEMA chief Mike Brown.

  10. drs

    It's not like Nate is the only poll aggregator out there, he's just the one with the (apparently) fanciest model. There's also electoral-vote.com and the Princeton Election Consortium (not coming up for me right now) and 4 or 5 others I don't follow. They all predicted Obama victory for months (and now), in fact Silver was generally the *least* bullish about Obama, in terms of electoral votes won. (The others don't claim a probability of winning.) Probably because they add up all the states leaning toward Obama, while Nate assumes some weakly Obama states will end up for Romney (and vice versa), just as a coin biased to heads nonetheless sometimes comes up tails.

    I'm reading Kahneman's _Thinking Fast and Slow_, and the desire for coherence and story points the other way here, I think; it biases us to listen to pundits who can 'explain' sports outcomes or stock moves or poll moves, when in fact no one is an expert at predicting the future, and well-used statistics is generally our least-bad guide to doing so. Conservative demonizing of Nate Silver doesn't change what the polls say, it just adds another item to the growing tally of conservative science-denial.

  11. drs

    Of course, there's a difference between predicting the future and making the future; polls and their aggregators don't count in the effect of Republican suppression and intimidation of voters.

  12. Jake_Ackers

    I wonder if this will be another Carter v Reagan. Where the polls misrepresented the voting turnout and the sort. Especially since there was a strong independent in 1980. Plus polling is better now. However, they keep predicting the same turnout as in 2008.

    Plus one key point is why do polls under 1200-1400 people polled go for Obama and those over that for Romney? Something is completely flawed here. But yah it does look like Obama will since simply because Romney isn't winning Ohio. It's not like Romney is winning WI, MI or PA to compensate for Ohio.

  13. EBounding

    Exactly. If just a few counties in key battleground states turn out their vote, it could change the election. At this point it doesn't really matter what the majority of people think about the candidates or their policies. Unless one of them commits an unbelievable gaffe, it boils down to the 5% or so of undecided independents.

  14. William

    Actually Silver does account for some of this. There is apparently more or less agreement by political scientiests on the effect that voter ID laws (it is a couple points towards the Republicans depending on what kind of law/requirements are in place) and Silver modifies polls from affected states towards Republicans by this amount before entering them in the model.

  15. R. Scanlon

    Predicting 49 of 50 states is not very impressive an accomplishment. Most states predictably lean Republican or Democrat, leaving about 5-7 actual toss-ups. So most people should be able to predict at least 45 by default. Reading even the most rudimentary state polling data indicates how each state is leaning, and barring some big event that shifts the numbers, they tend to stay fairly constant. So for most people that should push the number up close to 50.

  16. SES

    The model never said "each state shall go this way and no other, thus saith the Lord thy God." It is not a booming voice from the sky, unambiguous, or a firm answer. It is explicitly and purposefully not a definitive this/that prediction on anything (in fact, Nate Silver's recent book spends a lot of time railing about those kinds of predictions and how often they totally suck). In 2008, the most rudimentary state polling data tended to give a tiny advantage to McCain.

  17. Eddie

    I think rather than debating whether Nate Silver’s model was good because he predicted 49/50 states, it’s more useful to look at his popular vote predictions on a state-by-state level, and use that to determine “bias”.

    I’m using two sources of data; the first is 538′s “final” projections, which can be found on te sidebar here: http://web.archive.org/web/20081113114818/http://

    The second is the actual results, found here: http://www.infoplease.com/us/government/president

    Comparing the projected value against the actual value should point out if there is a clear “bias”; if the model works well, then the differences between the two should be close to zero. Thus, if there is a “democratic leaning” his model will “overstate” the share of the popular vote that Obaa would recieve (i.e projecting that Obama would gain a greater % of votes than he would eventually receive, and McCain would likewise be projected to gain a smaller % of votes than he would actually receive).

    My results (hand-typed, so I admit potential error) are here: https://docs.google.com/spreadsheet/ccc?key=0AjU-

    You might want to download it to play with it yourself.

    Observations:

    On the whole, the model slightly understated Obama’s chances, by about 0.5% of the vote (i.e on any given projection, Nate’s model gave Obama a 0.5% less share than he would actually receive). In other words, he presumed McCain would get more votes, comparatively, than he did. He overstated Obama’s chances in 20, and understated it in 30. He overstated McCain in 28 states, and understated him in 19 (the rest he got bang-on).

    There are a few big outliers (Vermont, Hawaii, and D.C.) that Nate drastically underestimated Obama’s vote share. Everything else falls within +/- 5%. For McCain, the model overestimated his projected share of the vote by ~6.6%, and underestimated it in Indiana by about ~6.1%.

    If you remove those 5, the variance shrinks considerably; Silver projects that Obama gains about 0.11% more of the vote on average than he does, and McCain gains about 0.03% more of the popular vote.

    Does this say anything about the 2012 election? Nope! But if the 2008 election is any indication (and assuming the model hasn’t changed), then his 2012 predictions are unlikely to be too far off the mark.

  18. Gastel

    I'm puzzled by the point of this entry. Does the last line suggest that as a society we should criticize people who use mathematical models and science to make conclusions? It is easy to criticize when the conclusion does not support your viewpoint, but that doesn't make the criticism valid.

    I truly am confused by the conclusion of this article, "But it does say something — an uneasy something — about his personal power, and the power of people like him in modern democracies." Are we really scared and uneasy about the power of people with answers? I despair if that is the case.

  19. R. Scanlon

    1. No prediction is as firm as your strawman description makes it look. The point is that making such a prediction is not difficult. One doesn't need a complicated mathematical model to look at polling trends in each state and make one's own electoral map based on those trends, and end up with the right answer. At the very least, anyone who does this exercise should end up with something very close to the right answer.

    2. When did McCain have this polling advantage? As I recall, after around mid-September, state polling was consistently favourable to Obama. By the end of October, the way the electoral college vote would go was rather clear to whoever was paying attention.

  20. Jake_Ackers

    What I got from it was that people allow the outcome to be shaped by what they think the result is going to be. Self fulling prophecy. They vote for this person because they think he will win and not because they favor that person. Plus a ton of people just vote for voting sake. Look at how some of these incumbents get reelected. Especially the elderly. Youth vote based on popularity more so than anything. Old people just vote because they feel if they pressed the button they did something important. Voting informed is important. Most people vote for the incumbent without knowing anytime about them.

  21. SES

    By the end of October, polling was much more muddled. There were more polls that didn't show Obama in the lead than those that did.

    In any case, the 538 model only gave McCain a 55% chance of winning. The anti-numbers crowd doesn't seem to care about anything other than the color of the state on the map, but that means that it assigned Barack Obama a 45% chance of winning. It wasn't proven wrong when he won.

  22. drs

    “In any case, the 538 model only gave McCain a 55% chance of winning.”

    Even though it was projecting Obama winning the populat vote by 6%, and was correct in almost every state? That doesn’t seem right.

    Another aggregator, with I think simpler math (especially then) predicted Obama landslide. http://www.electoral-vote.com/evp2008/Pres/Maps/O

    “Predicting 49 of 50 states is not very impressive an accomplishment.”

    You’re right in a sense; especially right before the election, it’s just a matter of reading the state polls and adding up the numbers.

    But that’s kind of the point; all this punditry is so much hot air compared to the data, of which we have a lot now. If you want a best guess at the future, don’t ask some pundit what he (usually he) thinks will happen, look at the polls. Doesn’t make for hot copy and big salaries, though. (Unless you’re first-in like Nate.)

    Like I said, when it comes to predicting the future, simple math tends to beat biased humans who simply lack the data to become true experts.

  23. drs

    “They vote for this person because they think he will win and not because they favor that person”

    If that were the point, I think it’d be complete bollocks. I’d bet a lot that no significant number of people will be voting for Obama over Romney because “he’s the winner”. (I’d say no people, but there’s usually *someone* to make any proposition true.)

    A lot more plausible would be turnout (“he’s going to lose, why bother voting”; however this applies to any ‘safe’ state) and donations. But Romney raised over a billion dollars, so not exactly suffering there either. (Oddly, he hasn’t spent his own money. Real committment there,)

  24. William

    538's projections less than a week out from election day 2008 thanks to the internet archive: http://web.archive.org/web/20081002090513/http://… Obama had a 85.4% chance.

  25. Guest

    Interesting comment drs.
    Romney spent quite a bit of his own money first time around. He consciously chose not to do so this time around (save for a $150,000 joint donation with Ann).
    How much has Obama given from his personal money to his campaign? Anyone know? I think you will find it is similar to Romney – and yet I'm not hearing any complaints about that.

  26. Colin Minich

    Well if this is pretty damn funny, the mighty Rasmussen shows a complete tie now as Romney's been dipping in the poll aggregates. I wonder what will be said now that it's no longer Silver proclaiming a decline. Also, check out Sam Wang…he's pretty good too if not seeming a bit optimistic.
    http://election.princeton.edu/

  27. Psudo

    Obama's actual campaign has dramatically outspent Romney's, $632 million to $389 million (60% more for Obama). Neither of those figures are anywhere near "over a billion dollars."

    If you include the political parties' spending, Super PACs, and other outside money, then they are awfully near a billion each: $930 million for Obama and $997 million for Romney (7% more for Romney). Still, I don't think a whole set of various folks' money can be accurately summarized as "Romney raised over a billion dollars."

    Source: http://www.opensecrets.org/pres12/

  28. drs

    “How much has Obama given from his personal money to his campaign? Anyone know? I think you will find it is similar to Romney – and yet I’m not hearing any complaints about that.”

    The Obamas have around $5 million. Romney has about $250 million. Bit of a difference there.

    Colin: I seem to recall Silver saying Rasmussen tends to converge to other polls right before an election.

  29. Monte

    Wouldn't being only about 1% off actually be a very close estimate?
    When it comes down to it, you can never expect 100% accuracy in any kind of poll; that's why polls have a margin for error. What I would say determines how accurate a poll is, is how close the results fall within a poll's margin of error. And while Silver was way off in a few states (though most states were less than 2 points off), the data you pulled together seems to indicate that he was still highly accurate since the ultimate result of the election fell within 1 point of his estimate. I also feel that his results are too close to say he has a bias in his data; that's just the slight inaccuracies that come with every poll.

Archives





  • Recent Posts

  • Cartoon Archives