By David Jenkin

You’ve mentioned that algorithms can be like a sledgehammer at times – please elaborate on that point. What did they fail to grasp with the election predictions?

In terms of the election predictions, from what I've read, there was only one poll in the United States that called how the individual states would vote. BrandsEye say they got it right, and not to take away from what they did, but from what I heard, they said "Trump is ahead in 11 battleground states” which is a version of prediction, but not to the detail I was discussing, ie. Nate Silver calling all 50 states in the 2012 election. 

I think people often confuse Social Data with Big Data. Social Data is certainly a component of Big Data, but there is a lot more to it than that. 

Another aspect to it is the amount the candidates themselves relied on Big Data to determine where they campaign, and where to air TV commercials. I've read that the Clinton campaign spent nearly four times the Trump campaign on tech and data analytics, and they still lost.

On the data side, there were some pretty standard and simple small data problems:

  • A misunderstanding / calculation of margin of error;
  • A smaller sample size due to most polling taking place face to face or on phones and people increasingly making themselves unavailable to pollsters;
  • An underestimation of the number of people who had never voted who were inspired by Trump's message;
  • Non-sampling error in the form of people not willing to admit their support for Trump in a poll; and
  • The dumbing down of complex statistics to simplified narrative, or a visualisation. 

Another thing to consider is the way people assess probabilities and risks. We were comfortable with Trump only having a 30% chance of winning, but would you take your family on holiday to Iceland if one of their bigger volcanoes had a 30% chance of erupting while you were there?

Could more complex algorithms ever improve on that?

Is it possible for Big Data to predict the future with such complex, dynamic systems? I'm not sure. In time, perhaps.  So much of what we do is getting recorded now, it does look like there will be a time when much more of our thinking, feeling and doing will be available for analysis, thus enabling more complex algorithms to have better predictive power. 

What needs to come together to create meaningful, intelligent data that can be truly reliable? Do you think the current technologies and systems out there could have done better with the elections predictions if applied differently?

I'm not sure if the current technologies and systems could have done better. Some argue that because one or two companies got it right, that the answer is yes, but it makes me think of two sayings: "given enough time, all predictions come true" and "even a broken clock is correct two times a day". Nate Silver made his name in previous elections, and even he got it wrong this time. I do worry that the people doing the polling (generally more intelligent, hopefully, more open-minded people) were so horrified by the thought of a Trump presidency, that they let that bias their assumptions.

What I always try to say is the human element is as important as the data inputs or systems used to analyse. Data is human-made, and reflects our prejudices, biases and judgments.

What kind of repercussions do you think this failure might have – I imagine many companies out there are doing some serious introspection with regards to their models?

I think with everything in life, the system aims toward some type of balance. Or to put it another way, regresses to the mean. The pendulum swung heavily in favour of Big Data and algorithms from 2010 to Brexit/Trump, and now it is swinging back. I think on the business side, executives are starting to realise that Big Data analysis is expensive and hard, and often not as insightful as they expected. I think a sense of realism is what is needed. Some things might never be able to be predicted, but that doesn't mean we need to throw out the model entirely. If anything, we should keep using data to keep asking better and better questions, and slowly getting to know our world better.

For more information, visit www.eighty20.co.za. Alternatively, connect with them on Facebook or on Twitter.