Big data seems to be one of the hottest topics in the press and blogosphere. Today, I read about IDC study of the growth numbers and it is pretty amazing. IDC expects there to be 40% compound annual growth rate, from $3.2 billion in 2010 to $17 billion in 2015. The assumption that researchers have is that the data sets are becoming too large to be able to manage with traditional database management tools.
This is all good news for the software business and the tools market in Big Data market as it forces software vendors to reimagine their existing solution portfolio to take Big Data more seriously and I bet there will be new startup players that will come with some innovative solutions and stir the markets for the big players. In our innovation and business modeling workshops at TELLUS International, almost every case has touched Big Data in one way or the other.
The only think I hope that the market does not do what it has always done is to create too much irrelevant noise about the topic (which it already is). I have built tens of analytical solutions and I will tell you that unclean data is still unclean and some traditional things will still matter when building any type of analytical solution. With new fad, we tend to make things more simple than they really are, so if you are in the Big Data market, make sure you ask the right questions from the vendor before you go forward with your purchase.
The post in eWeek today lists following trends for Big Data for 2014:
- SQL Holds biggest promise for big data – SQL development using Hadoop enables business analysts to use their skills in SQL tools of choice for big data projects.
- Yet SQL poses challenges – SQL requires data structure, and centrally structuring data causes delays and requires manual administration. SQL also limits the type of analysis.
- Authentication is top of data security issue.
- Data errors will occupy organizations in 2013.
- Emerge of operational Hadoop – it is estimated that there is a dramatic increase in product deployments of Hadoop across industries.
- More Data Warehouses will deploy enterprise data hubs – data hubs offload ETL processing and data from enterprise data warehouses to Hadoop, acting as a central enterprise hub that is 10 times cheaper.
- New data-centric apps will become mandatory – big data will emerge as competitive weapon in 2013. More companies will pinpoint individual consumers’ preferences for profitable up-sell and cross-sell opportunities, better mitigate risk, and reduce production and overhead costs.
- Data becomes the center of the data center – organizations will transition from developers driving the big data initiatives. IT will be tasked with defining the data infrastructure required to support diverse applications and focus on the infrastructure required to deploy, process and protect and organization’s core assets..
- Search emerges as the Unstructured Query Language – SQL initiatives for Hadoop in 2013, and 2014 will be the year that the unstructured query language comes into full focus.
- Hadoop will gain in stature – Hadoop will continue to displace other IT spending, disrupting enterprise data warehouse and enterprise storage.
- Hadoop still needs help to be ready for prime time – more organizations will realize that Apache Hadoop alone isn’t enterprise ready.
If we review these 10 trends, it is easy to agree on many of them and what I think is most important, Big Data initiatives will increase in 2014 and more organizations specifically in vertical domains serving retail customers will benefit of the results.
I do think that the complexity of Big Data is downplayed and this will hurt the market with failed efforts going forward. I do remember vividly many large failed enterprise data warehouse projects that were never able to recoup the investments and some organizations became fearful of even going into data warehousing projects. On the other spectrum you have business intelligence solution providers saying that they are doing Big Data without any merits of having the needed scalability or intelligence in the product to do so.
It is fascinating to see how this market where I have been for more than 20 years is evolving and creating some renewed enthusiasm. I am all for that and this is also a great opportunity for college and university students to put some focus on data analytics and statistical analysis courses as the market will be in great demand for people that understand data. That is already happening as reported on the markets.
