Big data cant be ignored. Big data or information of extreme size, diversity and complexity is everywhere. This disruptive phenomenon is destined to help organizations drive innovation by gaining new and faster insight into their customers. It is one of the main issues which identified for having an impact in the world of business intelligence in 2013. Just ask Gartner. Companies are struggling to keep up with the deluge. However the problems arising with big data aren’t as a result of its usage, but of companies’ inability to make use of all of the data they are collecting.
Much of this data is stored both in their warehouses and in cloud-based services. The thing is do companies know actually what they have stored? Do they know how to actually access it. Undoubtedly there are undiscovered insights into market opportunities, that might never see the light of day, or even predictions of impending problems ahead. This the dilemma of Dark Data or ‘unstructured data’. It is naturally different for each organization, but it is essentially data that is not being used to get a 360 degree view of a strategic objective. It can take the form of underused data include social media, website and mobile data.
Gartner analyst Andrew White offers this insight:
“Dark data is the cute name given to all that data an organization gathers that is not part of their day to day operations. It is old stuff, stuff that turned up in the mail that you kept, ‘just in case’. It is data that you didn’t erase, because “it might come in handy some time”. There is, in most cases, more dark data than there is light data (I guess the data your organizations does use every day). Dark data is a storage vendors’ dream. All those archiving specialists and traditional Information Lifecycle vendors who, despite their name, do NOT look after the birth and life of data, but only at the death (and after life) of data, are jumping for joy. And the BI vendors are just behind it since you will need their tools to help mine that dark vault.”
Imagine all that data you have stored as being in a raw state. You will need the right toolkit to drill down deeper,otherwise you will barely scratch the surface of the knowledge hiding within the dataset. Wired magazine perhaps said it best: “Freeing up dark data could represent one of the biggest boons to research in decades, fueling advances in genetics, neuroscience, and biotech.”
Andrew White cautions against getting to worked up about the dark data question: “Unless you, the business user, have an idea of what you want to ask of this dark data, there is no point worrying about it.” Given today’s available technology, its down to the appointed data analyst to know those questions to ask in the first place which implies working closely with team members that drive strategy.
The key questions Firms should be asking is, what are the strategies and skills required to tackle the dark data deluge? ‘Are we doing enough?’ Do we have a system which allows us to further analyse our data.? Get to grips with these questions, use dark data as a source for decision making and it will change your ability to outperform competitors.
Infographic credit: Infosys
Hayden Richards is Contributor of IntelligentHQ. He specialises in finance, trading, investment, and technology, with expertise in both buy-side, sell-side. Contributing and advising various global corporations, Hayden is a thought leader, researching on global regulatory subjects, digital, social media strategies and new trends for Businesses, Capital Markets and Financial Services.
Aside from the articles, interviews and content he writes for IntelligentHQ, Hayden is also a content curator for capital markets, analytic platforms and business industry emerging trends. An avid new media explorer Hayden is driven by a passion for business development, innovation, social business, Tech Trading, payments and eCommerce. A native Trinidadian, Hayden is also a veteran, having served with the Royal Air Force Reserves for the past 10 years.
Follow Hayden on Twitter @HaydenARichards, linkedin.com/haydenhrichards and http://www.scoop.it/u/hayden-richards
