Data has a habit of being staggering, mind blowing even. Writing for Forbes in January 2017, Gil Press discussed how more and more firms are tuning into the value of Big Data, but still struggle to derive genuine, actionable worth from it. Press’ article cited a report from NewVantage Partners, which described how – while 85% of businesses have initiated moves towards a data driven culture in the workplace – only 37% report that they have been successful.
Perhaps the issue is that data has now transcended conventional human understanding to such an extent that it is now accelerating over the hill and into the sunset. We cannot chase after this data on our own; we need tools and systems that can corral this surfeit of information, and make it more manageable.
But these tools exist. The same article predicted that, by the end of 2017, one third of Fortune 500 firms will be investing twice as much in information-based services and products than in the whole of the rest of their portfolio. This translates to a raft of tools, services and applications designed to make data easier for us to handle.
If the lack of tools is not to blame, what is? The NVP’s Big Data Executive Survey 2017 gives us another explanation: that many people, across all strata of an organization, remain stubbornly resistant to data.
[Too] Big Data
It is this rejection of Big Data, on a personal level, which is fuelling something of a rebellion in the field of Business Intelligence. Increasingly, moves are being made away from the Big, in favor of the Small, as figures in business and beyond, search for something easier to get a handle on.
This concept of Small Data is not particularly new, but it is gathering traction in a world which is increasingly swamped by enormous configurations of data. In 2016, the University of Pennsylvania’s Knowledge@Wharton business analysis journal ran an interview with Martin Lindstrom – an author and Small Data advocate – during which a very convincing argument was put forward which supported a move towards small data.
Lindstrom – who produced the book Small Data: The Tiny Clues That Uncover Huge Trends last year – described how he thought that Big Data was killing innovation and creativity. Over the course of the interview, Lindstrom discusses how cold, clinical knowledge, taken from a myriad of different data points, is no substitute for that creative spark – that flash of inspiration.
The point that Lindstrom is making is certainly valid. Humans are not computers; we are made of flesh and blood and this should not be forgotten. We are hardwired to find meaning, to find narrative, to find the ‘why,’ as opposed to simply the ‘what.’ By blindly pursuing Big Data initiatives, we run the risk of sacrificing this facet of our humanity; something which could arrest our forward motion and sense of innovation.
A Step Backwards?
But are we getting a little ahead of ourselves here? Are the statements we are making a little bold? While it is true that data volumes are increasing at an almost exponential rate, and that this has a serious tendency to swamp and overwhelm, this does not negate the value of Big Data altogether.
As we have already discussed, we have the tools at our disposal to deal with Big Data. We don’t need to dispatch a work experience intern to the back of the office with a pencil, notepad and a scientific calculator – we can use computer systems and applications to wrestle the facts and figures into submission.
Last year, Lindstrom warned that Big Data is all about correlation, while Small Data is all about causation and, as some of us may remember from high school science experiments, these are entirely separate concepts. However, a scientist will not dismiss correlation; they simply find further evidence to back it up. So it is with data interpretation.
It all comes down to adopting the right approach. If we make ourselves slaves to Big Data, suspending all decision-making processes until the next data source is found and mapped, and then the next, and then the next… we are heading only for creative inertia. If we turn our backs on insight and analysis and rely on sheer innovative spirit – sure, we might, unearth the next Snapchat – but we are more likely to end up with a pile of half-baked, poorly-thought-through, failed ideas.
Instead, we simply need balance. Support innovation with data; let creativity be the driver while insight provides the foundation.
Perhaps it is unfair to call Small Data a step backwards, but it is certainly premature to turn our backs on the advantages Big Data insight provides. The two concepts are not mutually exclusive; in fact, they may be more closely linked than you ever previously considered.
Small Data as Big Data Endgame
“Small data connects people with timely, meaningful insights (derived from big data and/or “local” sources), organized and packaged – often visually – to be accessible, understandable, and actionable for everyday tasks”.
This is how the Small Data Group defined Small Data back in 2013, and this definition has become widely accepted among those who seek ‘real’ data, and ‘real’ insights which can be actioned on a day to day level.
But doesn’t this look a little familiar? For me, this is very similar to the desired outcomes of Big Data advocates; an enhanced understanding of the problem or concept at hand, achieved in real terms.
This is proof that the two concepts are not mutually exclusive. Small Data advocates are not seeking to tear down the information structures that we have built over the last five or so years, just as Big Data advocates are not looking to replace human creativity with cold, quantifiable, demonstrable facts.
Instead, Small Data is the endgame of Big Data. We have tools and systems in place to process and handle Big Data, and to break it down into manageable chunks. This is how we can focus on real insight without stifling innovation and creativity, and without losing sight of the bigger picture.