April 2013

Undetected Hidden Biases in Data Lead to Poor Decisions

Workplace with computer with young businessman standing near by

The discovery of big data has reached nearly epiphany scale. Such hype around a new idea makes many people blind for the traps hidden behind the great potential.

Harvard Business Review has lately published many insights regarding big data in their blog network. The latest take on this hot topic by Kate Crawford discusses the hidden biases in big data. Crawford describes examples where big data originated from consumer mobile devices with social media features contains hidden biases caused by social and ethnographical factors. In such cases the generalization of results evidently leads to misinterpretations even if the data first seems comprehensive.

Hidden biases are not only an issue with big social media data but the same principles apply also to carefully collected corporate data. No matter how lavish your data is, it never makes the detailed understanding of its nature and shape redundant. In fact, quite the opposite. The more excessive your data is, the more accurately tailored your supportive tools need to be.

“As soon as the data was deeply understood, the model producing important predictions was created accordingly and the adaptive tool started to pay off…”

Our team worked with an industrial company that had recently deployed nice mathematical tools to refine their abundant data into demand predictions. These predictive analytics formed the foundation for operational management and decisions. The only issue was that the forecasts were far from accurate which led to poor managerial decisions affecting operational efficiency in over 30 countries.

The data was complete and the tools were based on rigor statistical methods so how could the derived predictions be wrong? Situation was frustrating as those predictions were the only meaningful information regarding successful decision-making.

There was a hidden bias in fundamental assumptions regarding distribution of the data. False assumptions made predictions inaccurate which directly led to uninformed decisions. As soon as the data was deeply understood, the model producing important predictions was created accordingly and the adaptive tool started to pay off. Decisions were supported by accurate analyses.

Specialized business analysts are able to tell you how reliable your data is for different predictive analyses. Once the biases in data have been detected, multi-method approaches are often needed to interpret and analyze the data accurately.

Don’t waste the excessive potential that lies in your business data. Take actions to first understand it thoroughly and build your predictive analytical tools on that. It will pay off.

Gartner Taps Predictive Analytics as Next Big Business Intelligence Trend

Focus on documents and pen on the table. Blurred people on background.
** Note: Shallow depth of field

Analyzes on firm’s internal processes and business environment are the driving engine in modern companies. This is moneywise evident in the large-scale investments that the big firms are still making to build new business intelligence tools. If the current growth continues, Gartner estimates that the total value of business intelligence business over triples by 2020 (current size estimated at 57 billion USD).

“Gartner proposes that in near future simulation based models are required to make more sophisticated predictions…”

The fastest growing subclass in business intelligence is data discovery or more generally large data analysis tools. These tools give an opportunity to describe and understand what the data implies. The current trend is that these tools are developing towards more visualized and illustrative ways to present the datasets.

Predicting the future means looking forwards – This is also the future of business intelligence according to Gartner. The article identifies that the next breakthrough in business intelligence is predictive analytics, which will increase the accuracy of predictions. Currently used methods such as extrapolation enables predictions to certain accuracy, but Gartner proposes that in near future simulation based models are required to make more sophisticated predictions. Gartner foresees that having an efficient way to utilize big data in predictions and then building decision-making on them is a critical component in building foundations for firm’s sustained competitive advantage in the future.

It’s hard not to agree.

Read the full article HERE.