Can Cloud Bridge the Big Data Talent Gap?

Big Data and cloud are both outpacing the rest of the software industry in growth rate according to recent IDC findings, and according to one analyst, as this trend persists, cloud-enabled big data software may be the tonic to help salve the skills gap that gets talked about so much in the big data arena.

First things first, the results are in – big data and cloud are growing.  While the rest of the software market puttered through 2012, cloud computing and big data applications grew at nearly twice the rate of their software peers, according to IDC’s Worldwide Semiannual Software Tracker. But while the worldwide software market grew at a sluggish 3.6% year over year rate for a total market size of $342 billion, applications orbiting the cloud and big data universes grew in the 6-7% rate.

“We found that those that were moving stronger and faster to the cloud were doing the best,” Henry Morris, Senior VP of Worldwide Software and Services at IDC told Datanami, noting that these applications primarily centered on customer applications such as CRM, applications for marketing and sales, or collaboration and social business type applications.

The same customer-centric theme emerged in the numbers reflecting the markets related to the use of information, said Morris, noting that the management and leveraging of information for competitive advantage is fueling growth in the big data and analytics arena.

Morris notes that he sees these growth trends continuing and one of the reasons is due to cloud’s ability to solve big data’s problems – particularly where expertise is concerned.

“One of the big barriers for companies to get into analytics is the availability of talent – the resources to do this type of work,” said Morris. However, he notes, cloud software offerings are starting to fill in the gaps on this challenge. “We’re seeing more analytical services available in the cloud that is providing these companies with access to more people and more expertise.”

The challenge is one that has been talked about in big data circles for some time. While major enterprises like Citibank or JP Morgan can hire their own data scientists, MapReduce programmers, and etc., a lot of the smaller companies are simply being outmuscled for the talent.

A recent study commissioned by Actuate revealed that “not enough staff with expertise” was among the major inhibitors preventing even large corporations from implementing big data solutions.  Another survey by analytics company, Pentaho, agreed, noting that the top three challenges in deploying big data implementations are resource availability (45%), skills (43%), and complexity of big data (38%).

Morris suggests that cloud software is starting to provide answers in this arena, providing the analytics know-how by expertise through proxy. “The cloud becomes a concentrator of talent in my view, and that opens up availability to more companies,” he says noting that the shift to close the talent gap through cloud offerings is underway with companies offering advanced analytic capabilities like SAS and Omniture (absorbed into Adobe’s Marketing Cloud in 2012) offering their respective analytics packages in the cloud.

“I think the cloud and the growth of analytics are beginning to come together because companies are forced to ask where they get the talent to do this type of work – the talent to pull the information together, the talent to build the analytical models, the talent to figure out what analytical techniques are appropriate to which problem,” he explained. “All of those point to this convergence of analytics, big data, and also the cloud.”

The direction of the VC winds seem to support this notion, and Morris noted that the investment trends may be shifting up the big data technology stack, gradually away from the infrastructure and data organization and management, and towards the tools, such as analytics & discovery and decision support applications.

“We’ve been seeing the VC money going towards the lower part of the stack, but it was interesting to see [at the IDC Directions 2013 conference in March], several VCs presenting around where money is being invested, and they were saying that it’s a little late to be investing in the lower part of the stack, that really started 3 or 4 years ago, but that there is more investment happening up the stack,” where software and analysis tools drive value for data.

This begins to come into focus as you look at how the people in big data are distributed relative to their place in the big data technology stack, explains Morris, noting that there are a lot of people dealing with data management, a relatively small number of people building models, and a large number of people who are downstream from those models who have to make decisions.

IDC’s estimates have the overall big data segment at $8.1 billion in 2012, with nearly 25% being software.  In 2016, IDC projects that number to grow to a sizeable $23.8 billion, with big data software maintaining 25% of that figure.

Though, it’s important to note that it’s not all cloud. Morris notes that companies are emerging to provide tools and techniques available to people in the line of business in the traditional way. How much of this software ends up being made as cloud-offerings is yet to be determined, but the trends already interplaying between big data and cloud suggest that it has the potential to be significant.

Related Items:

Survey: Solving Big Data Jobs Gap Requires Cutting Edge Technology 

Budding Data Scientists Lack Access to Big Datasets 

Cloudera Releases Impala Into the Wild