The main inhibitor, it is worth repeating, is not Big Data itself. It is the fear of it. The top decision makers in our society, the CEOs and the Government economists, are unsure how this big data can be used.
- The markets in America reflect a lot of fear. We are bombarded with information that we can not make sense out of it. We can't decide whether this Big Data information is good or is bad.
- As a result every CEO, government decision maker handling money only invests short term. Some CEO's buy back shares to invest the cash in government bonds at 2% interest rate. Companies sit on lots of cash, too afraid to spend it.
- Banks have tons of money, they sit in the vaults without making loans. The banks want to make loans, but say someone who qualified for a loan in 2007, does no longer qualify. The companies who qualify, they do not need any loans as they have tons of cash already.
As seen at Strata 2014, people are talking about Predictive Analysis, creating the popular confusion that now, with lots of data, and fancy statistical and analysis, we can predict the future from past data. This is not true! See what Benoit Mandelstam , the creator of fractals, clearly states.
So what can we do to make Big Data useful for our top decision makers who have the power to create jobs and economic growth? How can we win over Larry Fink and his clients to make longer term investment using Big Data input? How do we transform their skepticism in belief?
The partial answer to these questions I saw in Matei Zaharia from Databricks keynote at Strata 2014 conference two weeks ago.
Here is the story. Commodity clusters like those Hadoop based has made up to 20x cheaper to store datasets
By 2020 we will have 44 ZB (one zettabyte equals one billion terabytes), but the cost of storing this enormous data will be about the same as in 2013. What we can do with all this data? We can do two things:
- Define speed as "How quickly we can go from data to decisions?"
- Define sophistication as "Can you run the best algorithms on the data?" These are NOT off-the-shelf-algorithms.
Bullet #1 drew my attention. The purpose of any analysis of big data is not to deliver solutions. The purpose of of all big data is to convince the top decision makers to take decisions.
The future can not be predicted, but human can make informed decisions with minimized risks. Risks can never be eliminated. The technology is just an enabler for humans.
To use some metaphors, the world information is reported by the media. Wall Street Journal and New York Times, The Economist, etc are extremely reputable that the information is real, and inaccuracies are rare. The National Enquirer is another publication that no one or - very few people, perhaps from tabloid media business - will use to make investment decisions.
Even when reading Wall Street Journal, the decision makers have different criteria. George Soros, for example, in one interview said that each morning, he reads the papers to discover an error humans always do. We are not perfect. But such mistakes can make hundreds of millions for Soros, an investor who bets on stocks and currencies going down, not up.
Big Data is information that each one of us assesses whether it is reliable and usable in different ways.
The best tools in Big Data then, are the ones that generate decisions. Lutz Finger in his article STRATA 2014 - Data Demystified quotes James Burke:
Information is causing change… if it is not causing change, it is no information.If this is true, then we may have some sort of metrics on how to judge any big data tool. Something based on how popular and effective the tool is for decision makers.
Media, newspapers, books, movies have circulation numbers, Nielsen ratings, revenues generated.
Why not Big Data? Sure we heard the women sleep more than man, or customers in a bar ordering Corona beer spend more money.
But this somewhat useful information is trivia in comparison to Blackrock. If Larry Fink can use Big Data to revitalize the economy, and his success will propagate virally worldwide this will be by far the greatest breakthrough Big Data achieves.