Looking back at the recent Scottish referendum and the preceding campaign on both sides, it was interesting to see how the overall picture depicted by opinion polls really changed during the course of time and how the final result took most by surprise.
Use the tabs below at the top of the graph to cycle through how the story unfolded (I’ll be updating/improving the data visualisation in due course).
Data sources: What Scotland Thinks , Financial Times
Even before the results were announced, the pollsters appeared to be having trouble calling the result not least as there was no previous referendum to compare it to, as a panel of them including Martin Boom (ICM) and Damian Lyons Lowe (Survation) discussed on the Daily Politics show on the eve before the vote (skip ahead to 09.34).
Some experts had successfully managed to predict both the result and how the polls would fluctuate over time :
The result was as pollster Andrew Cooper, David Cameron’s former director of strategy, had privately predicted to Alistair Darling. Hired by Better Together in May, Cooper forecast that “the polls are going to narrow two weeks out and that you might even see a couple where you are behind”. Cooper told Darling that the gap would then widen again, leaving the final result a couple of points off the last poll and in favour of a No. The hearts of the undecideds said Yes and the heads said No, Cooper reasoned. The poll movements would result from those who went with their heart, deciding earlier than the people who went with their head.
However, the margin by which the pro-union side won took many by surprise:
The polls may have had the right winner and they may have captured evidence of a narrowing gap over the past few months but, ultimately, an error of about seven points is sub par and that’s generous within accepted margins
Steve Fisher, an associate professor in political sociology at the University of Oxford, tried to explain how the opinion polls were so far off :
Although the difference was within the margin of error for a single poll, all six final polls were so close together that they all over-estimated the Yes vote as previous referendums and other factors suggested they would. YouGov re-interviewed some of their previous poll respondents after they voted and found that there was both higher turnout among the No supporters and a small shift of Yes support to No at the last moment. Therefore, some differential turnout and some late swing to No would help account for the earlier overestimation. The YouGov prediction after the polls shut was Yes 46 No 54; very close to the actual result.
The most accurate of polls during elections are exit polls. However, on this occasion, there was none .
Exit polls are the best form of voting-related data we can ever get our hands on. They’re collected by large numbers of researchers standing outside polling stations asking tens of thousands of people how they voted – as well as collecting a little demographic information such as age, gender, race or social class.
Exit polls are considered more accurate than opinion polls as they reflect the voter’s actual vote after it has been cast rather than their intended vote before the act has taken place. It seems unusual, as none was commissioned, with the most likely reason being, again according to James Ball from the Guardian:
that the broadcasters were afraid of the consequences of a poll being wrong. If the BBC spends four hours discussing a poll that’s 51-49 in favour of Yes and when the final votes come in, the result is No, deputy heads may roll amid the backlash.
And from Buzzfeed :
In the absence of an exit poll, this private telephone survey, conducted by Conservative peer and politics obsessive Lord Ashcroft, is probably the nearest we’ll get to knowing why individual Scots voted to stay part of the UK. The research was conducted after all votes had been cast, late on Thursday night.
Lord Ashcroft’s poll may be the most accurate of all (in fact it got the exact vote correct 55% to 45%). However, in terms of demographic information (income, gender, etc.), I actually found some of Survation’s poll data to be more detailed. I’ve been playing around with it and think that they’ve provided a wealth of data that can be turned into graphs/visualisations that should provide some interesting insights. One of the possible outcomes would probably look similar to this graph, produced by Matt Hernderson using Mathmatica.
The above graph shows disposable income by region compared to the percentage of No vote using data from the 2011 census. This would imply that the regions that voted pro-union were also the most affluent. A similar conclusion was reached by Steven Fisher as well as the Financial Times , although I would say that Matt’s graph is a little more straightforward to understand at face value.
However, there are plenty of caveats to this kind of analysis. Many would point out that correlation does not equal causation. Among them being Chris Hanretty, who describes here :
It would be hypocritical to now use a sample size of 32 to reach apparently definitive conclusions about the independence referendum. Which brings me to my last point; data. In my view, the independence referendum has given us election nerds very little to chew on. The opinion polling was sporadic and at times unreliable; there was no exit poll; the results were announced by council area. All of which means that it’s pointless to even begin to analyse what happened, at least from my perspective. I have around eighty variables for each of the 32 council areas (at the last count!): age, gender, ethnicity, place of birth, economic activity, household composition, etc. However, what is the point of even beginning to run models? The spatial unit of analysis (council area) is far too large to deliver meaningful results. I suspect there’d be a relationship between some form of deprivation and the Yes vote but at what scale? And who can say how the “deprived” voted? You can say that a council area like Glasgow has pockets of deprivation. You can also say that Glasgow voted for Yes. But you can’t say that “the deprived voted Yes”, certainly not from the data we have.
All good points! Nonetheless, I will still have a go at delving into the Survation data to see if I can come with any data visualisation gems in the next post, which will just have to be taken with a massive pinch of salt.
Sign up to my mailing list below so you’ll be the first to know when it comes out.
