How real is the value proposition of Big Data for most businesses

Onboard data or Ingest… Store and Organise it…. Extract Insights or Reporting which all equal value creation….Not to mention all this Data Science and Artificial Intelligence stuff?….Easy right?

Maybe not. With more and more businesses frustrated with their Big Data plays and many Big Data platforms failing to realise value uplift that the stakeholders expected. Something is certainly smelly in the world of Big Data.

Maybe it’s not as easy as we all thought? We all hear about the amazing use cases and satisfied customers, though many industry reports suggest they are in the minority. This series of articles is designed to discuss they pitfalls of Big Data and suggest some key areas to keep in mind to help minimise or mitigate issues early.

Lets breakdown the three distinctive phases, Ingest, Store, Consume. In this first part of three, I’ll discuss Ingest or onboarding data.

Ingest: this area is so often misunderstood and unfortunately the start of some of the basic issues with the effective establishment of a Big Data platform. Many businesses believe they have the data, they have lots of data, but what is realised quickly when starting a Big Data play is that, there are gaps, not only in breadth of data, but depth, accessibility and finally quality (Completeness, Uniqueness, Timeliness, Validity, Accuracy, Consistency), lets save the quality discussion for another time.

Why are there gaps? There are many factors in how this occurs, one of the biggest culprits is that years of projects have created very narrow datasets, designed to address very specific issues and or application outcomes. That has lead to the myriad of ETL (Extract Transform and Load) jobs or Data integrations which are largely point solutions. As such they rarely have had the foresight to bring in as much data and certainly the breadth of data needed for a big data outcome.

To be fair to those who came before (including myself), there were many drivers for this behaviour, time to market or completion deadlines — comments like “We need that project deployed and completed by …”, are common. Project management translation, do whatever it takes to narrow the scope to the absolute minimum viable deliverable. On of the first victims of that thinking, narrow data breadth. Ever heard, “Why do we need all that data?”, “Show me the reporting requirements” or “Do you realise how complicated and costly that is?” and my personal favourite, “Don’t worry we can always do another data extract at a later date”.

Many of these points in the frame of a pre Big Data utility world, made sense, Data storage was expensive, Compute was expensive and the ability to provision these capabilities quickly and efficiently was extremely difficult to overcome. Cost did drive many of these behaviours and in some ways even today still does, at least perceptually.

How might we overcome these pitfalls?

Firstly, understand the value of your data and convert senior stakeholders into true evangelists. These programs of “Data Transformation” struggle to succeed without strong executive support. Make the case “real”, tie it back to measurements, yep, you need to commit, don’t just work the “Data Hype Cycle”.

Initially it may be lower storage costs, lower environment costs or reduced testing costs, in my experience keep the measurements grounded or in accounting terms “tangible”. People gravitate to the “intangibles”, they sound better, “greater access to data”, “faster provisioning of data” or “better analytics” and though they all have merit, in the early days of a successful Data transformation, you need to ground it in “tangible”, measurable dollars.

With that credibility, the intangibles become more palatable over time and less likely to be challenged (at least early on), and successful deployments will convert those intangibles into “measurables”. Bear that in mind, when thinking about the value created by the new platform and how you want to measure it.

A related step is understand you have as much work in change management (roles, structures, skills or people and process) as you do in technology change. Typically we find that companies pursue the Big Data dream and then find they run out of steam, when they haven’t faced into the fact that people need to be re-skilled, re-tooled and processes that may have worked yesterday, are no longer relevant in their new data paradigm. In fact trying to shoe horn old techniques into the new Big Data play typically just don’t work. Rebuild that Target Operating Model, be bold!!

Ask for as much data as you can possibly get!!!! Work that Big Data storage cost advantage. Yes, costs in AWS can blow out, so develop the cost management discipline or evolve Data Operations to proactively manage environments in realtime, spin them up and shut them down when they are no longer required. These are extremely powerful capabilities, which deliver tangible benefits, but if you move ahead in a new Operating model unchecked, you start to dilute benefits in cost reduction and the very credibility of the new platform you have built, it can actually get more expensive and slower to deploy, not faster. But whatever you do don’t compromise on the depth and breadth of data, use cases are most likely not even in your pipeline yet that will drive revenue and insight into your business tomorrow. That data will drive value, take the leap.

Find experts, don’t wing it. Whether that be a firm that has the experience and skills in deploying these new capabilities or whether you need to find people on the market and insource the skills (typically a mixture of both is a great long term play). Find people with Data skills and probably more importantly, the ability to think differently. What do I mean? Big Data has brought with it some fundamental changes to the way we interact with our data, as such new techniques are not only required, but also a new mindset. Flooding people with vast amounts of data, when they have only ever previously been exposed to curated data, is daunting and in many ways creating the environment for failure. If you or your deployment partners don’t understand and implement a way to help champion that change within your organisation and imbue the new skills required (remember that change piece is hard, but absolutely required), you will not realise the true benefit of the new Big Data paradigm. The advantage many of the really successful Big Data players today is that they were built in the new Big Data paradigm (in fact many invented key components of the technology we take for granted today) and have applied the new thinking and technology to drive, the creation of maximum value from their data assets (think Netflix, Uber, Google, Amazon — all data driven companies).

Finally a stretch target, its a tough one, but try nevertheless, as the pay off in my experience is very rewarding, not only in long term cost reductions but also in the establishment of fundamental trust in your data. Cleanse your data as much as possible at source, build integrity checks like postcode check or mobile phone number formatting checks into your web or mobile applications (start with the small and relatively simple changes). They all help alleviate so many downstream issues for the analytics, reporting and ultimately business users, who consume that data. Build that mindset into the development of new technologies in your company “Design with Data in mind”, “Deploy knowing someone will use it”.

Doing that very same fixing post load in your Big Data platform is not only expensive in the longer term, but also degrades trust in the data, bear in mind that no one knows the client, customer or B2B data better than the people on the frontline. So spend the time to enhance their experience to ensure you don’t create longer term issues with the data downstream.

In the next article I’ll discuss the key challenges with “Store” and how the shift from a world of structured data to unstructured data, has confused many and again led to sub-optimal outcomes.

About the author:

Related posts:

About 1.5 years ago we released DraftJS Plugins 1.0. It provides an architecture to combine various rich text editing features based on your needs with only a handful lines of code. Today we are…

Carpooling or ride-sharing has become a household name among the office goers. You may have seen a few advertisements and posters while travelling here and there. If you are one of those rare human…

There is so much misunderstanding about sacred sex especially when people first come to hear about Tantra You might agree that orgasm is like no other experience. It is pure, unadulterated pleasure —…