The Collective Intelligence Behind Big Data

The Collective Intelligence Behind The Big Data

A few weeks ago, a brilliant conference took place at Nesta, the UK Agency for Innovation,  in London.  In it, writer and expert Geoff Mulgan, the Chief Executive of Nesta, presented a masterclass on what he calls the Big Mind.  The masterclass was inspired in his recent book, and in it, Mulgan explores a very interesting way of bringing two different approaches with common goals together: Collective Intelligence and Big Data. Or what’s basically the same: how to bring humanity to IT-based data analytics. And the challenge ahead is huge…

So, what is big data and collective intelligence? Basically, “Big data is a term that describes the large volume of data – both structured and unstructured – that inundates a business on a day-to-day basis. But it’s not the amount of data that’s important. It’s what organizations do with the data that matters. Big data can be analyzed for insights that lead to better decisions and strategic business moves,” according to leaders analytics SAS.

On the other hand, and taking a definition given by the Financial Times, Collective Intelligence “refers to harnessing the power of a large number of people to solve a difficult problem as a group. The idea is that a group of people can solve problems efficiently and offer greater insight and a better answer than any one individual could provide. Collective intelligence can also be a valuable marketing tool.”

The key point of the conference and thus Mulgan’s book, is not just how both could work together but actually how they are meant to join up and tackle the great challenges of our future. The recently deployed Big Data trend needs the human brain to not just quantify data, but also to understand it and, more importantly, predict it.

A Long-Running Relationship

Even though Mulgan is one of the pioneers that publicly acknowledges a brand new path in economics and society itself, by joining up both concepts, he is not the first one to write about the topic.

There are others that have explored the concepts and its possible positive outcomes. Expert Émile Servan-Schreiber already set out the ground about collective intelligence and big data back in 2015 with a more than recommended article about the topic.

He emphasised the  profound need for mutual collaboration between the two concepts. While Big Data would do the hard work of researching and drawing out data-based-patterns, the humans would just do what they do the best: interpreting the data, and thus giving the chance to those patterns to be wrong.

He explained it as it follows:

“In fact, whether or not a big-data search party is likely to discover something useful really depends on the kinds of data that are at hand. Computers are really good at processing data that are well structured: digital, clean, explicit and unambiguous. But when the data are unstructured – analogue, noisy, implicit or ambiguous – human brains are better at making sense of them.”

That is exactly the moment when bigger issues come up and thus bigger minds have to be put at work in solving the problems. Whereas a single human brain, or a modest personal computer, may deal with small data sets of the preferred kind, the “bigger” the data is, the more computing power has to be brought to bear. In the case of structured data, bigger computers will come in handy, but in the case of unstructured data – the kind computers can’t properly deal with – there’s also a hard limit on how much computing power a single human brain can deliver. So the best way to make sense of big unstructured data sets is to tap into the collective intelligence of a multitude of brains.

So the best way to make sense of big unstructured data sets is to tap into the collective intelligence of a multitude of brains

The even bigger intelligence: The AI

On the other hand,Tech expert Amandine Mouillet, wrote an interesting article of what she called An an unsuspected pairing, in which she adds the new AI mainstream to the Big Data-Collective Intelligence ecuation.

One of the main points she highlights refers to “As Big Data is neither limited by time nor location, it has vast potential. When paired with artificial intelligence, processing thousands of data from different sources, apprehending the future through predictive analysis and detecting recurring schemas becomes a reality. Predictive analysis is improving day by day thanks to Machine Learning algorithms, which are perfected on a continual basis. Today, however, Big Data analysis does not yet meet all our requirements and still has a long way to go before being a true match for the human brain.

Collective intelligence, then, has an important role to play as it possess human sensibilities and can interpret a situation according to a number of existing factors, draw up an action plan and also take final decisions.”

Putting the AI on play, it seems to be just a matter of time that human brains would be not necessary any more yet that time is still to come.

For now, Geoff Mulgan’s forecast is much more human-embracing and instead of seeing a probable dispute, he predicts a necessary collaboration.

In fact, “this bigger mind—human and machine capabilities working together—can help solve the great challenges of our time. It also forces us to rethink how many existing institutions are organised, from businesses to universities, governments to cities.”