On political polls and panopticons

Just how badly did political pollsters and election modelers perform in 2020?

Even though we have real-world consequences of the election to discuss, that question has commandeered an annoyingly large share of our intellectual bandwidth in the last two weeks. Ben Hunt of Epsilon Theory made a solid case this week that the popular applications of statistical modeling to politics are bunk. Hunt’s ruthless critique of FiveThirtyEight’s Nate Silver also offered insight into why polls themselves seem so far off the mark in recent election cycles. The answer lies in the interplay of polling and the so-called Panopticon Effect.

Broadly speaking, the Panopticon Effect refers to the phenomenon in which the act of observing subjects alters their decisions and behavior. Hunt posits that relentless media coverage of electoral horse races and prediction models have induced polling respondents to act strategically. In other words, the knowledge that they will affect the output of election models shapes respondents’ answers or even their willingness to pick up the phone when polling firms call. The models’ very existence degrades their own inputs.

Hunt also points to a major flaw in the FiveThirtyEight project that I’ve long felt doesn’t receive enough attention: We can’t verify the predictive power of Silver’s model for elections; therefore it is of limited usefulness. Each presidential election is a one-shot deal. Silver can argue all he wants that Donald Trump’s win in 2016 would have happened about 30% of the time that the election was held, but how do we test that claim? (I’ll plead ignorance regarding Hunt’s argument that the thin margins in vote totals mean Silver’s model objectively failed in the last two elections – I don’t have the analytical savvy to say one way or the other.)

I’d love to say that I see an easy way to untie some of these knots in both polling and election modeling. Whatever my own concerns with Silver’s electoral model, I do believe that the FiveThirtyEight approach to a wide range of subjects represents a clear net positive for journalism. On the other hand, if Silver and ABC News continue hyping his model without communicating its limits and contextual factors, the credibility of FiveThirtyEight as an enterprise deserves a huge hit.

Meanwhile, at the very least, policymaking needs an efficient way to capture public opinion. We’re left with meager alternatives if we can’t put any credence in the direct responses of people – and we should note that the act of polling itself triggers the Panopticon Effect before FiveThirtyEight-type modeling even comes into play. Perhaps there are behavioral-based approaches that might be less vulnerable to manipulation?

Best of luck to the data scientists out there.