Follow Datanami:
September 16, 2013

Salaries Account for Nearly Half of Big Data Spending

Isaac Lopez

While the rest of the economy has been performing quite sluggishly, those involved in big data and data science haven’t been feeling the pinch. One research group says that where big data is concerned, companies are overspending on their talent.

Global spending on big data by organizations will exceed $31 billion in 2013, says ABI Research citing a new market forecast. While the forecast includes money spend on professional and technology services, hardware and software, a huge chunk of that money is going straight into salaries.

“We estimate that in big data initiative salaries account for about half of the current spending, with the other half allocated to vendors’ products and services,” said Senior Analyst Aapo Markkanen. “What we’re now seeing is quite significant overspending on salaries, as organizations turn to data scientists and other specialists in order to leverage big data in the first place. Similarly, a good share of the money is spent on the associated professional services, which have sprung up to assist firms that are data-rich but skills-poor.”

The dynamic is no surprise for anyone familiar with the space. Many companies have plans to increase the size of their data analytics teams, only to find a market place thin on the talent, with the best skilled going to the highest bidder. Counter to what is happening in the rest of the economy, in big data there are plenty of jobs, but few people to fill them.

In one recent example, Chicago-based lender, Enova International has doubled the amount of analyst on their staff to 25 over the last three years. The company, which uses big data analytics to determine in less than 10 minutes whether to approve a loan, would like to double that number by next year, but aren’t finding the talent domestically. The results, writes Becky Yerak with the Chicago Tribune, is importing of the talent – about half of Enova’s data analysts have visas or green cards.

While importing the talent is one strategy being employed, another ploy involves raiding talent from sister industries that require similar skill sets. Yerak writes that while historically, Enova typically hired people with degrees in statistics, computer science, and industrial engineering, it has employed a strategy to broaden the talent pool by including people with backgrounds in astrophysics and computational chemistry.

While companies are out in the arena trying to keep and grow their talent pools, ABI says that narrowing the skills gap represents a lucrative revenue opportunity for the vendors selling solutions in the big data space. Certainly, this fact hasn’t been lost on the leading Hadoop distro vendors in the space, who have worked hard and fast to build more on-ramps into the complicated framework. From efforts to bring the popular SQL query language to Hadoop (such as Apache Hive and Cloudera’s Impala), to even integrating basic search, there is work underway to make big data much more accessible to a broader pool of working talent.

Charles Zedlewski with Cloudera said it well earlier this summer in a discussion about adding Apache Solr search into Hadoop. “If you think about all the MapReduce developers there are in the world today, there is probably a few hundred thousand. If you think of all the people that are SQL business analysts, there’s probably a million. But if you think about all the people who know how to use search, Google has more than a billion users by itself.”

Of course, the idea that any Joe Schmoe sitting in front of a monitor with a search engine can do the work of a data scientist is a slap in the face to many of the data scientists out there. They may have to suffer the indignation in stride as companies including Cloudera, Hortonworks, Platfora, and Alteryx and many more move toward bringing the big data fire down from Mount Olympus and into the hands of the mortals.

In the meantime, the big data gravy train rolls for those with the skills to capitalize, with no real end in sight. The big data space will exceed $31 billion in 2013, and reach up to $114 billion by 2018 says ABI Research. If the trends continue to hold, the skills gap looks to be one that won’t be bridged any time soon.

Related items:

 

Data Science Back to School 

IBM Ships Hadoop Appliance for the Big-Skills Challenged 

Data Science Has Got Talent as Facebook Launches Competition  

Datanami